The Elastic Stack Study Notes

The Elastic Stack Study Notes & guide explores the Elastic Stack (ELK), an open-source suite that includes Elasticsearch, Logstash, Kibana, and Beats, used primarily for data ingestion, storage, analysis, and visualization. It is highly valuable for data analysts, security engineers, and operations teams to manage real-time logs and metrics.

The ELK Stack is a robust platform for managing and analyzing large-scale, real-time data. It starts with Elasticsearch for storage and search capabilities, Logstash or Beats for data ingestion, and Kibana for visualization. Security engineers focus on integrating log data from devices and using KQL for investigation.

Data analysts load and analyze datasets with customized index templates. The guide offers detailed installation instructions for multiple OS and methods (Docker, Linux, Windows), describes architecture including nodes and clusters, and contrasts data ingestion methods (Beats vs. Logstash). Advanced Kibana features include dashboards, Canvas, maps, and alerting. Finally, it provides cyber investigation use-cases like brute-force detection and phishing analysis using KQL queries and visual tools.

Table of Contents:

• I am a data analyst, how should I start?
• I am a security engineer, how should Istart?

• Elastic Search
• Purposes of Using Elastic Search
• Elastic Search Index
• Elastic Search Node
• Elastic Search Clusters
• Elastic Search Installation and
• configuration
• Elastic Search Configuration
• Verifying Installation
• Executing Search Queries in Elastic
• Search

• With Elastic Agent
• With Log Stash
• Installing and ConfiguringLogstash
• With Beats
• Types of Beats
• Installation and Configuration
• Beats Vs Logstash: Which one to usefor log collection and ingestion?
• Example Ingesting Fortinet FirewallLogs

• Installing and Configuring Kibana
• Kibana Components
• Discover Tab
• Fields
• Tables
• KQL (Kibana Query Language)
• Reserved Characters in KQL
• WildCards in KQL
• Searching The Logs with KQL