How to configure Kafka Burrow for an SASL Kafka Cluster
Configuring LinkedIn Kafka Burrow on SASL Kafka Cluster to check consumers Lag, topics and consumers.
Burrow: Kafka Consumer Lag Monitoring Tool
In the modern application development era, all the organizations are trending towards Event-Driven Micro Services architecture. Apache Kafka is one of the widely adopted distributed event streaming platform between micro-services for its scalability, performance, fault-tolerant, durability, reliability and many more features.
As these organizations tend to use Apache Kafka Cluster, quite a few applications evolve rapidly using multiple hundreds of Kafka Topics, and many more producer and consumer applications. It becomes quite hard to manage hundreds of topics in cluster, information about consumers, offsets, consumer lag etc. so there have been a few open source monitoring tools on kafka cluster like Yahoo's Kafka Manager, LinkedIn Burrow, Landoop Kafka Tools etc.
This blog will focus on configuring LinkedIn Kafka Burrow on SASL Kafka Cluster to check consumers Lag, topics and consumers.
Sample Grafana Dashboard after integrating with burrow
Configuring Burrow using SASL Connection:
As we already know SASL Authentication for Kafka Cluster can be done in below three different ways:
As of now current open source LinkedIn Burrow supports only SASL PLAIN Authentication configuration. To add support for all SASL mechanisms, I have forked the base burrow repository and added support for SASL_SCRAM_256 and SASL_SCRAM_512 at this GitHub repo. Docker version of this repo is available at docker hub (vishwavangari/burrow).
Spin up local Kafka Cluster which accepts clients based out of SASL authentication as in my previous article.
Below burrow.toml configuration file is used in building up burrow docker image, so we'll need to pass in the required cluster parameters while spinning up burrow docker instance.
Using vishwavangari/burrow and joway/burrow-dashboard docker images, passing in required configuration to spin up burrow for SASL Kafka Cluster using below docker-compose file. Thanks joway for setting up Burrow Dashboard to visualize consumers, consumer lags, topics etc.
If we use SHA-256 SCRAM algorithm for Kafka Cluster users, then we can specify burrow env variable SHA_ALGORITHM in docker-compose file as _sha256.
_Note: If the connection between burrow docker container and local kafka cluster fails, then start docker-compose file by adding in network_mode as host
We could leverage these burrow APIs and then integrate with any visualization, alerting frameworks like Grafana, Splunk dashboards to visualize and alert support teams by email/message, in case of any cluster disaster or if consumers are lagging way behind or if something abnormal happens by configuring few rules.
We could hit up http://localhost:8095/ for Burrow dashboard:
Burrow, a Kafka monitoring tool is widely used and integrated with other visualization, alerting frameworks like Influx, Grafana, Splunk. If you want to read more about burrow refer this wiki and about telegraf plugin for burrow at this link.
You might also like
How AI and Personalized Marketing are Transforming Retail Sales
How AI/ML, CDP, personalization, and BI are revolutionizing retail, fashion, and beauty. Dive into brand examples from Sephora, ThredUp, and H&M.Read article
19 Cloud Computing Statistics You Need to Know in 2023
By 2025, over 100 zettabytes of data will be stored in the cloud—50% of all global data storage.Read article
5 Ways to Transform Grocery Retail with an AI-Driven Data Strategy
Explore 5 AI-driven data strategies for grocery retail. Learn how to solve challenges like workforce management, pricing, and disconnected CX.Read article
Copilot and the Future of AI-Assisted Coding: Insights from a Software Engineer
GitHub's Copilot is an IDE-integrated tool that streamlines coding by offering real-time text completion suggestions, predicting what developers might type next.Read article
Data Mesh and Event-Driven Architecture: Unleashing Healthcare Data Potential
Harness the power of data mesh and event-driven architecture. Streamline healthcare data management, drive innovation, and unlock lasting growth.Read article
Unlocking Hidden Revenue: Optimizing Data Systems for a Leading Car Retailer
Explore how a leading care retailer teamed up with us to optimize workflows, boost security, and uncover hidden revenue.Read article
Rapid Migration of a Legacy Java Monolith for a Major Retail Brand
Find out how we helped a grocery giant tackle a two-year standstill and the loss of key team members to conquer a critical deadline.Read article
How an HC & Tech Services Provider Optimized Scalability, Agility, and Security
Discover how a human capital and tech services provider unlocked the power of cloud infrastructure to optimize scalability, agility, and security.Read article
Egen Cuts Sifter's Infrastructure Costs by Over 30%
Learn how Sifter, an online grocery platform, reduced infrastructure costs by over 30%, allowing the startup to focus on growth and profitability.Read article