Logstash Plugin: Hello World

In this Logstash plugin series, we will developing simple Logstash plugins to power up your ELK stack with minimal and highly-maintainable code. However before we code, you may want to explore few already available filter plugins to begin with.

Pre-requiste:

Basic understanding of Logstash.
Basic understanding of Ruby.
Understanding ELK as a whole is added advantage but not required.

Here is the getting-started with for writing a Filter Plugin for Logstash: elastic.co/guide/en/logstash/current/filter... However we will explore this in greater detail and explore different aspects of Logstash Plugins.

In this documentation, we will explore writing the basic logstash:

Lets the run the command to generate a template for writing a filter type of logstash plugin.

/Users/Smit/Documents/Dev/ELK/logstash-7.15.2/bin/logstash-plugin generate --type filter --name hello_world --path /Users/Smit/Documents/Dev/ELK/logstash-plugin

Lets drill into each of the above components:

/Users/Smit/Documents/Dev/ELK/logstash-7.15.2/bin/logstash-plugin : Plugin executable location often found in the logstash folder

--type filter : As we are going through different filter type, this is set as filter, but it could have outputs like input, output, and filter

--name hello_world : The name of the plugin and the reference also when we are going add it to our logstash configuration.

--path ... : This argument states where you want to save the plugin, it could be some place in your computer or in the logstash folder itself.

Once generated the output of the above command will look similar to below:

Screen Shot.png

Next, edit the hello_word.rb file, with below content

# encoding: utf-8
require "logstash/filters/base"

# This  filter will replace the contents of the default
# message field with whatever you specify in the configuration.
#
# It is only intended to be used as an .
class LogStash::Filters::HelloWorld < LogStash::Filters::Base

  # Setting the config_name here is required. This is how you
  # configure this filter from your Logstash config.
  #
  # filter {
  #    {
  #     message => "My message..."
  #   }
  # }
  #
  config_name "hello_world"

  # Replace the message with this value.
  config :message, :validate => :string, :default => "Hello World!"


  public
  def register
    # Add instance variables
  end # def register

  public
  def filter(event)

    if @message.downcase =~ /#{"hello|world"}/
      # Replace the event message with our message as configured in the
      # config file.
      event.set("type", "hello-world")
      event.set("message", @message)
    else
      message = event.get("message").to_s
      if message && message.downcase =~ /#{"hello|world"}/
        event.set("type", "hello-world")
        event.set("message", message)
      end
    end

    # filter_matched should go in the last line of our successful code
    filter_matched(event)

  end # def filter
end # class LogStash::Filters::HelloWorld

In the above example, if the message contains hello or world it will add a new attribute in the logstash event called type with value of hello-world

Next, let's edit logstash-filter-hello_world.gemspec especially the ones marked as TODO as they will need to be resolved before building the plugin. For now, I have removed TODO for simplicity.

  s.summary       = 'Write a short summary, because Rubygems requires one.'
  s.description   = 'Write a longer description or delete this line.'
  s.homepage      = 'https://shahsmit.hashnode.dev'

Next, lets run the command to build the gem:

gem build logstash-filter-hello_world.gemspec

Output of the above may look like

 Successfully built RubyGem
  Name: logstash-filter-hello_world
  Version: 0.1.0
  File: logstash-filter-hello_world-0.1.0.gem

Next step, install the built plugin into Logstash using the below command:

/Users/Smit/Documents/Dev/ELK/logstash-7.15.2/bin/logstash-plugin install /Users/Smit/Documents/Dev/ELK/logstash-plugin/logstash-filter-hello_world/logstash-filter-hello_world-0.1.0.gem

After installing we will run the logstash,

/Users/Smit/Documents/Dev/ELK/logstash-7.15.2/bin/logstash -e 'input { stdin{} } filter { hello_world { message => "%{message}" } } output {stdout { codec => rubydebug }}'

In the above command, hello_world plugin has been added:

message => "%{message}" this snippet takes the message from input and assigns to the variable message

Once the logstash is started, you can type in any message, for example:

Input 1: Type in hello is this amazing and it will output:

{
    "@timestamp" => 2022-03-29T10:33:25.330Z,
       "message" => "hello is this amazing",
          "type" => "hello-world",
      "@version" => "1",
          "host" => "Smits-MacBook-Pro.local"
}

Input 2: test 123 and it will output:

{
    "@timestamp" => 2022-03-29T10:33:45.743Z,
       "message" => "test123",
      "@version" => "1",
          "host" => "Smits-MacBook-Pro.local"
}

Upon comparing input 1 and 2, the attribute type is missing in input-2 output as the input value does not contain any words such as hello or world.

That is all for this tutorial, and very first installment on building custom logstash plugins.

Feel free to share your thoughts and feedbacks on comments or on twitter at @smit_shah_95

Logstash Plugin: Hello World

101 on Logstash Plugins