This article was a bit harder to write than it would seem on the surface. Amid all the hustle and bustle of the current development cycle, it was hard to find a particular focus on what this effort was all about. Then it finally dawned on me. There wasn’t one. …

After my last blog post about Logstash, Elasticsearch, and Kibana, I wanted to investigate something else I kept coming across during my Logstash research: Elastic Beats. Beats initially appeared to me to be a way to send data to Elasticsearch, the same as Logstash, leading me to wonder how Beats…

Whether you’re integrating multiple microservices, looking to improve app reliability, or building a new streaming app, you might need a message queue (MQ) or message broker platform. These types of software pass messages from producing apps or services to consuming apps or services. Two of the most popular platforms for…

Kafka Streams is a client library providing organizations with a particularly efficient framework for processing streaming data. It offers a streamlined method for creating applications and microservices that must process data in real-time to be effective. Using the Streams API within Apache Kafka, the solution fundamentally transforms input Kafka topics…

Author, Paul Brebner In part 1 of this blog series we explored the background to Kafka® and meta-data management, including the use of ZooKeeperTM and the recent move to KRaft to replace it. We revealed the results of our first experiment with Kafka ZooKeeper vs. KRaft, to measure the impact…

Paul Brebner, November 2022 Recently Instaclustr announced the general availability of the Open Source (Apache 2 license) Karapace Apache Kafka® Schema Registry add-on service for our Managed Kafka service. Now, you don’t often hear the words “Kafka” and “Schema” in the same sentence, unless the sentence is something like “Kafka…

In the previous blog, we demonstrated that the process of sending messages via Avro and Karapace from Kafka producers to consumers works seamlessly, although what exactly is going on under Karapace’s exoskeleton is perhaps a bit opaque (e.g. the communication between producers and consumers and Karapace isn’t visible at this…

1. Apache Kafka® Abandons the Zoo(Keeper) For several years Kafka has been modified to remove the dependency on ZooKeeper for meta-data management, using a new protocol called KRaft. More details on the Kafka Raft protocol are here. “KRaft” stands for “Kafka Raft”, and is a combination of Kafka topics and the Raft consensus algorithm. …

If you’re interested in viewing the metrics from your Instaclustr clusters along with the applications that use those clusters in one place, there are a number of ways you can do so. I touched on one option of using Instaclustr’s Prometheus endpoint, Prometheus, and Grafana in a previous blog post…