How to configure Kafka Burrow for an SASL Kafka Cluster
Configuring LinkedIn Kafka Burrow on SASL Kafka Cluster to check consumers Lag, topics and consumers.
Burrow: Kafka Consumer Lag Monitoring Tool
In the modern application development era, all the organizations are trending towards Event-Driven Micro Services architecture. Apache Kafka is one of the widely adopted distributed event streaming platform between micro-services for its scalability, performance, fault-tolerant, durability, reliability and many more features.
As these organizations tend to use Apache Kafka Cluster, quite a few applications evolve rapidly using multiple hundreds of Kafka Topics, and many more producer and consumer applications. It becomes quite hard to manage hundreds of topics in cluster, information about consumers, offsets, consumer lag etc. so there have been a few open source monitoring tools on kafka cluster like Yahoo's Kafka Manager, LinkedIn Burrow, Landoop Kafka Tools etc.
This blog will focus on configuring LinkedIn Kafka Burrow on SASL Kafka Cluster to check consumers Lag, topics and consumers.
Sample Grafana Dashboard after integrating with burrow
Configuring Burrow using SASL Connection:
As we already know SASL Authentication for Kafka Cluster can be done in below three different ways:
As of now current open source LinkedIn Burrow supports only SASL PLAIN Authentication configuration. To add support for all SASL mechanisms, I have forked the base burrow repository and added support for SASL_SCRAM_256 and SASL_SCRAM_512 at this GitHub repo. Docker version of this repo is available at docker hub (vishwavangari/burrow).
Spin up local Kafka Cluster which accepts clients based out of SASL authentication as in my previous article.
Below burrow.toml configuration file is used in building up burrow docker image, so we'll need to pass in the required cluster parameters while spinning up burrow docker instance.
Using vishwavangari/burrow and joway/burrow-dashboard docker images, passing in required configuration to spin up burrow for SASL Kafka Cluster using below docker-compose file. Thanks joway for setting up Burrow Dashboard to visualize consumers, consumer lags, topics etc.
If we use SHA-256 SCRAM algorithm for Kafka Cluster users, then we can specify burrow env variable SHA_ALGORITHM in docker-compose file as _sha256.
_Note: If the connection between burrow docker container and local kafka cluster fails, then start docker-compose file by adding in network_mode as host
We could leverage these burrow APIs and then integrate with any visualization, alerting frameworks like Grafana, Splunk dashboards to visualize and alert support teams by email/message, in case of any cluster disaster or if consumers are lagging way behind or if something abnormal happens by configuring few rules.
We could hit up http://localhost:8095/ for Burrow dashboard:
Burrow, a Kafka monitoring tool is widely used and integrated with other visualization, alerting frameworks like Influx, Grafana, Splunk. If you want to read more about burrow refer this wiki and about telegraf plugin for burrow at this link.
You might also like
Meet GrocBOT, an App Designed to Reduce Food Waste and Save Money
How do grocers balance demand and inventory to avoid food waste and lost revenue? GrocBOT improves forecasting and estimates demand while providing consumers with an easy-to-use chatbot.Download solution brief
Meet iREC, a Predictive Product Recommendation Engine
How do you deliver 1:1 personalized promotions with ease? iREC is a predictive product recommendation engine that allows retailers to deliver the right coupon to the right customer at the right time.Download solution brief
Reimagining Retail for Real-Time Responsiveness
Personalization isn’t possible without a real-time and historical 360° view of each customer—what they’ve bought in the past, what they’re considering now, previous interactions with customer support, their product reviews, and more.Download white paper
Engineering Last-Mile Delivery
Online grocery sales grew 54% in 2020 to reach $95.82 billion. We recommend 3 proven strategies for grocery retailers to help modernize their infrastructure for last-mile delivery.Download white paper
How to build your own Clubhouse - Part 2
How to Build your own Clubhouse
How AI Can Enhance Your Product and Customer Experience
A deep dive into implementing AI-based analytics to help transform your product experience and build strong brand loyalty.Read blog
AWS re:Invent in Review - Part 3
Let's go over the all the major announcements from the Week-3 of the AWS re:Invent 2020.Read blog