In this article, we’ll see how to make use of Kafka client library(REST Api’s) to interact with Kafka cluster. Let’s start with Part-1 of the 3 part series

Kafka Client API — Kafka logo taken from official Kafka website

Part-1: We’ll see 3 Java Api’s
- AdminClient API
- Producer API
- Consumer API

Part-2: Kafka Streams(In-progress — coming soon)
- Streams API

Part-3: Kafka Connect (In-progress — coming soon)
- Connect API

Setup Kafka Cluster in local/Docker:

Setup Kafka Cluster by creating a topic — “numbers”

Pre-requisites:

- Kafka Cluster (configured and running in local/docker)
- IntelliJ Community (or any IDE of your choice)
- Maven/Gradle (build tool)
- Java…


In this article, we’ll see the steps to install and run Kafka cluster on Docker (Docker toolbox) or Local machine.

Apache Kafka: Quick start
Kafka Connect: Quick Start

Note: Here we are setting up a Kafka cluster with 3 brokers & one Zookeeper instance.

Setup Kafka on local machine (windows):

Download Kafka from the below link:

Once kafka is downloaded on the local machine, extract Kafka on to the directory and create couple of directories to save logs for Zookeeper and Kafka broker’s as below.

Kafka CLI commands to start Kafka cluster in local/docker (windows)
Windows commands path: ../kafka_2.12v/bin/windows

  1. To…

In this article, we’ll see how to document Spring Boot application (using Kotlin) implemented in my last article.

Gadget gallery App: Kotlin + Sprint Boot +H2 database

OpenAPI 3.0 is an open-source format for describing and documenting API’s formerly known as Swagger specification.

Step-1: Let’s start by adding OpenAPI 3.0 maven dependency to our pom.xml

<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
<version>1.5.4</version>
</dependency>

Step-2: Enable openapi documentation by adding @OpenAPIDefinition on the main class as below.

package com.shasr.gadgetgallery

import io.swagger.v3.oas.annotations.OpenAPIDefinition
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication

@OpenAPIDefinition
@SpringBootApplication
class GadgetgalleryApplication

fun main(args: Array<String>) {
runApplication<GadgetgalleryApplication>(*args)
}

With steps 1 and 2 we configured OpenAPI 3.0…


In this Part-1 article, we’ll implement a simple Gadget gallery(CRUD API’s to Create, Retrieve, Update and Delete Gadget details) REST API’s with Kotlin, SpringBoot 2.x, JPA and H2 in-memory database.

Technologies/Tools:

IDE: IntelliJ (STS/Eclipse)
Kotlin: 1.4.x
Build tool: Maven
Spring Boot: 2.4.x
Database: H2 (in-memory database)
Rest Client/Postman

Gadget gallery API’s — let’s create a sprint boot project from spring initializer which exposes 6 API’s to perform CRUD operations as below.

  • Fetch all existing Gadgets from database
  • Post a new Gadget details to Gadget table
  • Fetch Gadget data based on gadgetId
  • Update Gadget data based on gadgetId
  • Remove/Delete Gadget data…

In this article, we’ll see Kafka Connect components, why it is needed and how it helps to integrate Kafka with different systems.

Note: new to Kafka ?? check “Apache Kafka: Quick start

Kafka Connect

Topics Covered:

  • Kafka Connect
  • Connector
  • Task
  • Worker
  • Convertor
  • Transform

What is Kafka Connect?


Apache Kafka

In this article, we’ll look into Kafka fundamentals which helps Kafka beginners to get insights of Kafka components.

1. Producer
2. Consumer
3. Consumer Group
4. Topic
5. Partition
6. Offset
7. Segment
8. Replication Factor
9. Leader and Follower
10. ISR-In Sync Replica
11. Acknowledgement (Acks: 0, 1, all)
12. Zookeeper
13. Broker / Node / Bootstrap Server
14. Controller

Apache Kafka is a scalable, fault tolerant and distributed streaming platform with key capabilities as below.

- Publishes and subscribes to streams of records/events.
- Stores streams of records/events in a fault-tolerant durable way. …


Sidecar Pattern

In microservice architecture, it’s very common to have multiple services/apps often require common functionalities like logging, configuration, monitoring & networking services. These functionalities can be implemented and run as a separate service within the same container or in a separate container.

Implementing Core logic and supporting functionality within the same application:

When they are implemented in the same application, they are tightly linked & run within the same process by making efficient use of the shared resources. …


In this article, we’ll look how into how producers work internally and how the partitions are selected by producer while sending messages

Kafka Series:

- Apache Kafka: Quick Start
- Kafka Connect: Quick Start
- Setup Kafka Cluster in Local/Docker

Here we have 3 Kafka brokers running at ports : 29091, 29092, 29093

Sample Topic: kafka-prod

Producer:

It’s an application which acts as a source of the data stream. In other publish/subscribe systems, the same components are known as publishers or writers.

Producers sends a message/record to a specific topic (as specified while creating a Producer).

By default, the producer…

Shashir

Middleware Engineer, Java/Kotlin/Spring/Kafka

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store