Logstash x Kafka Integration Part 2

In the previous post, we explored the in-build functionality of Kafka Inputs in Logstash bundle i.e. those are already available when you install Logstash. In this post, we will install plugins that are not available in the default bundle.

Pre-requiste:

Ensure Kafka is up and running, for me I am running it on docker for ease.
Download latest Logstash (I am running 8.4.3)
Also go through Part 1, to be are of Kafka input.

Avro Codec

Avro codec is a plugin applicable to only Kafka input, where we can parse incoming message that is in Avro format. Similar to the JSON codec we have seen in Part 1.

Firstly lets look at the configuration, here we are using a avro_schema_registry instead of inbuilt avro provided by Logstash.

input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topics => ["hash-node-topic"]
    codec => avro_schema_registry {
          endpoint => "http://localhost:8081"
    }
    value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
  }
}

filter { }

output {
   stdout {  }
}

In order to install this plugin, you can simply run,

logstash-plugin install logstash-codec-avro_schema_registry

More info on the above custom-plugin, you can refer to the documentation.

Once installed, and updated configuration. You can publish the message,

For this demo, I have published below message in Avro:

{
    "id": -6827561954658259000
}

the above in interpreted and output in logstash as below:

{
            "id" => -6827561954658259000,
    "@timestamp" => 2022-10-15T05:09:42.228576Z,
      "@version" => "1"
}

As you can see, the above is very Similar to the avro output we saw in Part 1.

The above was a result of few other references, so I cannot end this tutorial without the speical mention of those.

Github: github.com/revpoint/logstash-codec-avro_sch.. Stackoverflow: stackoverflow.com/questions/74069511/unable.. Github Issue Tracker: github.com/shah-smit/observability-spring-d..

Feel free to share your thoughts and feedbacks on comments or on twitter at @smit_shah_95

Logstash x Kafka Integration Part 2

Extending the part 1, to explore open-source plugins available

Avro Codec