As already mentioned in Logstash plugin management, Logstash plugins are self-contained RubyGems.
As extensive knowledge of Ruby is not expected from the readers, we will take a look at some simple illustrations of how a plugin works, and how we can design one simple plugin on our own. We will also cover some details of how the plugins are bundled to gems. More information is available at https://www.elastic.co/guide/en/logstash/current/contributing-to-logstash.html.
Let's look at the structure of a drop filter
plugin, which is used to drop certain events on certain conditions (https://github.com/logstash-plugins/logstash-filter-drop):
# encoding: utf-8 require "logstash/filters/base" require "logstash/namespace" # Drop filter. # # Drops everything that gets to this filter. # # This is best used in combination with conditionals, for example: # [source,ruby] # filter { # if [loglevel] == "debug" { # drop { } # } # } # # The above will only pass events to the drop filter if the loglevel field is # `debug`. This will cause all events matching to be dropped. class LogStash::Filters::Drop < LogStash::Filters::Base config_name "drop" # Drop all the events within a pre-configured percentage. # # This is useful if you just need a percentage but not the whole. # # Example, to only drop around 40% of the events that have the field loglevel wiht value "debug". # # filter { # if [loglevel] == "debug" { # drop { # percentage => 40 # } # } # } config :percentage, :validate => :number, :default => 100 public def register # nothing to do. end public def filter(event) event.cancel if (@percentage == 100 || rand < (@percentage / 100.0)) end # def filter end # class LogStash::Filters::Drop
Now, let's try to break it down and look at each component of a plugin.
The first requirement actually loads the logstash/namespace.rb
file, which defines the modules namespaces for the input, filter, output, and codec plugins.
require "logstash/namespace"
Then, since this is a filter
plugin, we will add dependency for the filter:
require "logstash/filters/base"
Similarly, for input, we can add /logstash/inputs/base
, and for output
/logstash/outputs/base
.
Next, for each plugin, we need to declare a class for it, and it should include the required Base
class for the filter
plugin as well:
class LogStash::Filters::Drop < LogStash::Filters::Base
So, as we have a drop
filter, we will declare a class by its name.
Next, we need to specify the name of the plugin that will be used in the Logstash configuration. We do this by declaring config_name
:
config_name "drop"
So, it will be used like this:
filter { drop { } }
We can define as many configuration options as we need for the plugin with this setting. It allows us to set the name of the option, its data type and default value, and specify if it is required:
config :percentage, :validate => :number, :default => 100
The following are the configurations:
: validate
: It allows us to enforce the data type for the option. The possible values can be :string
, :number
, :array
, :hash
, :boolean
, and so on.For the drop
filter, we have a specified validation for the percentage
option to be of type : number
.
: default
: It allows us to specify the default value for the option.For the drop
filter, we have specified the value 100
as the default for the option named percentage
.
: required
: It takes a boolean value as either true
or false
and specifies whether the option is required or not.Every plugin type (input, filter, output, and codec) has certain methods that they need to implement to initialize instance variables and to execute actual operations inside the plugin.
Plugin type |
Methods |
---|---|
Input plugin |
|
Filter plugin |
|
Output plugin |
|
Codec plugin |
|
For the input plugin, the register
and run
(queue) methods need to be implemented.
The register
method is used to initialize the instance variables if any.
The run
method converts the stream of incoming messages to events that can be further transformed:
public def run(queue) #Code which converts messages to event here. end # def run
For the filter plugin, the register
and filter
(event) methods need to be implemented:
public def register # nothing to do. end
The register
method is used to initialize instance variables if any. For drop
filter, we don't need to use any instance variables, so we will keep it empty.
public def filter(event) event.cancel if (@percentage == 100 || rand < (@percentage / 100.0)) end # def filter
The filter
method does the actual work of filtering the events. Inside the filter
method, we can use the config
parameters set using an '@'
prefix, and we have event properties available using event hashmap
.
For example, we can get the message as event["message"]
.
Also, certain operations, such as event.cancel
, are also available.
For example, in the drop
filter, we will use event.cancel
to cancel the event matching this filter.
For the output plugin, the register
and receive
methods need to be implemented.
The register
method is used to initialize the instance variables, if any.
The receive
method processes the events before sending them to the output destination, depending on the type of plugin.
public def receive(event) end # def event
The codec plugin is used with input and output plugins to decode an input event or encoding an outgoing event.
For the codec plugin, register
, encode
or decode
methods need to be implemented.
The register
method is used to initialize instance variables, if any.
The encode
method is used to encode an event to another format.
An example is the json
codec plugin, which transforms the events to json
format:
public def encode(event) @on_event.call(event, event.to_json) end
The decode
method decodes the incoming data to an event. This method needs a yield
statement to return decoded events to a pipeline.
For example, in the
spool
codec plugin, to send the messages to some buffer:
public def decode(data) data.each do |event| yield event end end
Now, we have seen the structure of a plugin, which gives us a head start on developing one of our own.
In this section, we will demonstrate building a simple filter plugin using the knowledge of the structure of a plugin that we acquired in the previous section.
In this illustration, we will assume that we have a sequence of numbers coming in a stream, and we want to denote them with certain currencies based on a name, which we will pass as a parameter to the plugin. Let's see what our simple currency
filter plugin looks like:
# Adds a Currency Symbol to price field # #filter { # currency{ # name => "$" # } #} require "logstash/filters/base" require "logstash/namespace" class LogStash::Filters::Currency < LogStash::Filters::Base config_name "currency" config :name, :validate => :string, :default => "$" public def register #do nothing end public def filter(event) if @name msg = @name + event["message"] event["message"] = msg end end end
Let's take a look at how the preceding filter is structured.
First, we have added the dependency for the required classes:
require "logstash/filters/base" require "logstash/namespace"
Then, we have defined a class for the filter:
class LogStash::Filters::Currency < LogStash::Filters::Base
Next, we named the filter using config_name
:
config_name "currency"
Now, we will specify the configuration option needed for this filter as we need the name of the currency to be specified so we can add it to the message. We will define it as follows:
config :name, :validate => :string, :default => "$"
Then, as we don't need to set any instance variables, we have provided an empty register
method for the filter:
public def register #do nothing end
Next, we will implement the filter
method for the filter
plugin, which will take an event and apply the logic for currency
:
public def filter(event) if @name msg = @name + event["message"] event["message"] = msg end end
Here, we will first check the value of the name
filter and if it is present, we will add the value in front of the message; otherwise, the filter will be ignored.
Now, filter
can be used as follows:
filter { currency{ name => "$" } }
Let's say if your input is 200
after using this filter, each incoming event's output from the Logstash filter plugin will look like this:
{ "@timestamp" => "2015-06-21T14:21:54.123Z", "message" => "$200", }
Now, when we have successfully created a plugin, save it as currency.rb
in the following folder structure:
logstash-filter-currency └───lib | └───logstash | └───filters | └───currency.rb Gemfile logstash-filter-currency.gemspec
Now, to create the RubyGem for the folder, we will require a gemfile and a gemspec file present in the logstash-filter-currency
top folder.
Let's add some specifications to our gemspec file:
Gem::Specification.new do |s| s.name = 'logstash-filter-currency' s.version = '0.1.0' s.licenses = ['Apache License (2.0)'] s.summary = "This plugin adds a currency name before message." s.description = "This plugin is used to add core logstash available plugin, to define a new functionality of adding currency symbols for certain messages" s.authors = ["Saurabh Chhajed"] s.email = '[email protected]' s.homepage = "http://saurzcode.in" s.require_paths = ["lib"] # Files s.files = ["lib/logstash/filters/currency.rb"] # Special flag to let us know this is actually a logstash plugin s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" } # Gem dependencies s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 2.0.0' s.add_development_dependency 'logstash-devutils' end
Save this logstash-filter-currency.gemspec
file under the root plugin folder as shown in the folder structure.
It requires Ruby gem bundlers to build gems based on these files, which can be easily installed on the Ruby console using:
$ gem install bundler
More information on using bundler can be found at http://bundler.io/.
Now, we can build the gem using:
$gem build logstash-filter-currency.gemspec
That's it! This should have created a gem named logstash-filter-currency-0.1.0.gem
in the same folder.
It can be installed to the existing Logstash installation easily:
$ bin/plugin install /path/to/ logstash-filter-currency-0.1.0.gem
If successful, you should see the plugin listed in:
$bin/plugin list
We can quickly test the plugin using the logstash -e
flag option:
bin/logstash -e 'input { stdin{} } filter { currency { name => "$" } } output {stdout { codec => rubydebug }}'
For the filter
plugin, any number that we write will be appended by the $
currency name:
200 { "message" => "$200" "@version" => "1", "@timestamp" => "2015-06-27T19:17:20.230Z", "host" => "saurzcode" }
We can see $
being added to the number 200
that we entered as standard input.
Now, we have successfully created our first Logstash filter plugin and tested it successfully.
Similarly, plugins of input and output types can be created and deployed.