This book is aimed at introducing the building of your own ELK Stack data pipeline using the open source technologies stack of Elasticsearch, Logstash, and Kibana. This book is also aimed at covering the core concepts of each of the components of the stack and quickly using them to build your own log analytics solutions. The book is divided into ten chapters. The first chapter helps you install all the components of the stack so that you can quickly build your first data pipeline in the second chapter. Chapter 3 to Chapter 7 introduce you to the capabilities of each of the components of the stack in detail. The eighth chapter builds a full data pipeline using ELK. The ninth chapter introduces you to some of the use cases of the ELK Stack in practice. Finally, the tenth chapter helps you know about some of the tools that can work with ELK Stack to enhance its capabilities.
Chapter 1, Introduction to ELK Stack, introduces ELK Stack, and what problems it solves for you. It explains the role of each component in the stack, and also gets you up and running with ELK Stack and with installations of all its components—Elasticsearch, Logstash, and Kibana.
Chapter 2, Building Your First Data Pipeline with ELK, helps you build a basic ELK Stack pipeline using a CSV formatted input, and explores the basic configurations to get your ELK Stack up and running to analyze data quickly.
Chapter 3, Collect, Parse, and Transform Data with Logstash, covers the key features of Logstash, and explains how Logstash helps integrate with a variety of input and output sources. This chapter also aims to explain various Logstash input, filter, and output plugins, which help collect, parse, transform, and ship data using Logstash.
Chapter 4, Creating Custom Logstash Plugins, explains how we can create our own custom Logstash plugins catering to our various needs that are not satisfied using the already available plugins. It explains the lifecycle of a logstash plugin and how various types of input, filter, and output plugins can be developed and published.
Chapter 5, Why Do We Need Elasticsearch in ELK?, explains the role of Elasticsearch in ELK Stack, and explains the key features and basics of Elasticsearch, such as index, documents, shards, clusters, and so on. It also covers various indexing and searching APIs and Query DSLs available in Elasticsearch.
Chapter 6, Finding Insights with Kibana, explains how to use Kibana to search, view, and interact in real time with data that is stored in the Elasticsearch indices. It explores the various search options that are available, and how we can use the Discover page of the Kibana interface.
Chapter 7, Kibana – Visualization and Dashboard, explains in detail about the various visualizations and dashboards that are available in Kibana with various examples. It also explains the Settings page, which helps configure the index patterns, scripted fields, and so on.
Chapter 8, Putting It All Together, shows the combination of all three components to build a fully-fledged data pipeline using ELK Stack, mentioning the role of each of the components as explained in previous chapters.
Chapter 9, ELK Stack in Production, explains some of the important points to keep in mind while using ELK Stack in production. It also explains the various use cases, and case studies that make use of ELK Stack in various use cases across the industry.
Chapter 10, Expanding Horizons with ELK, explains various tools, which combined with ELK, enhances the capabilities of the stack.