Apache Kafka Complete Deep Guide (Installation + KRaft + Commands + Spring Boot)

Apache Kafka is a distributed event streaming platform used for building real-time data pipelines, asynchronous communication, and event-driven microservices. Kafka works as a high-throughput message broker where producers publish events to topics and consumers process those events.

Kafka Architecture Overview

• Producer sends messages to Kafka topics
• Topic stores ordered messages
• Partition provides scalability
• Broker stores and serves messages
• Consumer reads messages from topics
• Controller manages Kafka metadata

Kafka Installation Requirements

• Java 17 or higher recommended
• Kafka binary distribution
• Minimum 4 GB RAM recommended
• Available port 9092

Verify Java Installation

java -version

Kafka requires Java runtime because Kafka broker and command-line tools are written in Java.

Kafka Directory Structure

kafka/
 ├── bin/
 │    └── windows/
 ├── config/
 │    └── server.properties
 ├── libs/
 └── logs/

KRaft Mode Introduction

Modern Kafka versions support KRaft mode where Kafka manages metadata using its own Raft-based controller instead of ZooKeeper. KRaft stores cluster metadata inside the metadata log directory.

Generate Kafka Cluster ID

.\bin\windows\kafka-storage.bat random-uuid

Kafka generates a unique cluster identifier. Example output:

PwZd7ux4TxmP6fpgxXT4OA

Format Kafka Storage

.\bin\windows\kafka-storage.bat format -t <CLUSTER_ID> -c .\config\server.properties

This initializes Kafka metadata storage. It should be executed only when creating a new Kafka cluster.

Start Kafka Broker

.\bin\windows\kafka-server-start.bat .\config\server.properties

Keep this terminal open because Kafka broker runs as a foreground process.

Verify Kafka Broker

.\bin\windows\kafka-broker-api-versions.bat --bootstrap-server localhost:9092

Create Kafka Topic

.\bin\windows\kafka-topics.bat --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

List Kafka Topics

.\bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092

Kafka Producer and Consumer Commands

Start Kafka Console Producer

.\bin\windows\kafka-console-producer.bat --topic test-topic --bootstrap-server localhost:9092

After running the command, type messages in the console. Every message will be published to the Kafka topic.

Start Kafka Console Consumer

.\bin\windows\kafka-console-consumer.bat --topic test-topic --bootstrap-server localhost:9092 --from-beginning

The --from-beginning option reads all existing messages from the earliest offset.

Check Topic Details

.\bin\windows\kafka-topics.bat --describe --topic test-topic --bootstrap-server localhost:9092

• Shows partition information
• Shows leader broker
• Shows replication details
• Shows partition offsets

Check All Kafka Topics For URLs

Sometimes Kafka messages contain API URLs, callback URLs, webhook URLs, or service endpoints. You can scan all topics for http or https values.

for ($topic in .\bin\windows\kafka-topics.bat --list --bootstrap-server localhost:9092) { .\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic $topic --from-beginning | Select-String 'http|https' }

Kafka KRaft AccessDeniedException Issue

A common local Kafka startup failure happens when Kafka cannot access or delete files inside the KRaft metadata directory.

java.nio.file.AccessDeniedException:
C:\kafka\kafka\tmp\kraft-combined-logs\__cluster_metadata-0\checkpoint.deleted

Root Cause

• Kafka was stopped forcefully
• Previous Java Kafka process was still running
• Windows locked metadata files
• Antivirus blocked file deletion
• KRaft metadata recovery failed

Why Kafka Did Not Accept Connections

Kafka broker startup failed before opening port 9092. Because the broker was unavailable, topic commands failed with Request METADATA failed errors.

Request METADATA failed on brokers [localhost:9092]

Local Development Fix

• Stop Kafka process
• Kill old Java process if required
• Delete kraft-combined-logs directory
• Generate new cluster UUID
• Format Kafka storage again
• Restart Kafka broker

taskkill /F /IM java.exe

rmdir /S /Q C:\kafka\kafka\tmp\kraft-combined-logs

Production Environment Warning

Deleting kraft-combined-logs is NOT a production recovery solution. This directory contains Kafka cluster metadata. Removing it can destroy topic and cluster information.

Production Recovery Approach

• Do not delete Kafka metadata files
• Check broker logs first
• Check file permissions
• Check disk availability
• Check controller quorum health
• Restore from backup if metadata corruption exists

Spring Boot Kafka Integration

Spring Boot provides Spring Kafka support for building producers and consumers easily. It integrates Kafka clients with dependency injection, configuration management, and message listeners.

Add Maven Dependency

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Kafka Application Configuration

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      group-id: user-service-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

Kafka Producer Example

@Service
public class KafkaProducer {

 private final KafkaTemplate<String,String> kafkaTemplate;

 public KafkaProducer(KafkaTemplate<String,String> kafkaTemplate){
     this.kafkaTemplate=kafkaTemplate;
 }

 public void sendMessage(String message){
     kafkaTemplate.send("test-topic", message);
 }
}

Kafka Consumer Example

@Service
public class KafkaConsumer {

 @KafkaListener(topics="test-topic", groupId="user-service-group")
 public void consume(String message){
     System.out.println(message);
 }
}

Sending JSON Messages With Spring Boot Kafka

In real applications, Kafka messages are usually JSON objects instead of simple strings. Spring Kafka supports automatic JSON serialization and deserialization.

Create Event Object

public class UserEvent {

 private Long id;
 private String name;
 private String email;

 // getters and setters
}

Producer JSON Configuration

spring:
 kafka:
  producer:
   value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

Send JSON Event

kafkaTemplate.send("user-topic", new UserEvent(1,"John","john@test.com"));

Consumer JSON Configuration

spring:
 kafka:
  consumer:
   value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer

Kafka Consumer Groups

A consumer group allows multiple consumers to share message processing. Kafka guarantees that one partition is consumed by only one consumer inside the same group.

• More partitions allow more parallel consumers
• Consumers in same group share workload
• Different groups receive the same messages independently

Kafka Partition Concept

Partitions are the unit of scalability in Kafka. Messages inside a partition are ordered by offset.

Topic: order-topic

Partition-0
 offset 0 -> OrderCreated
 offset 1 -> OrderPaid
 offset 2 -> OrderCompleted

Kafka Offset Management

• Offset represents message position
• Consumer commits offsets after processing
• Committed offsets allow restart recovery
• Incorrect offset handling can cause duplicate processing

Kafka Retry Mechanism

Production applications must handle temporary failures such as database downtime or external API failures. Kafka retry topics help process failed messages again.

• Main topic receives event
• Consumer processing fails
• Message goes to retry topic
• After retry attempts message goes to DLT

Dead Letter Topic (DLT)

A Dead Letter Topic stores messages that cannot be processed successfully after multiple retry attempts.

@RetryableTopic(attempts="3")
@KafkaListener(topics="order-topic")
public void consume(Order order){
 // processing logic
}

Kafka Production Configuration

spring:
 kafka:
  consumer:
   enable-auto-commit: false
   max-poll-records: 100
  producer:
   acks: all
   retries: 5

• Disable auto commit for important workflows
• Use acknowledgements after successful processing
• Enable retries for temporary failures
• Monitor consumer lag

Kafka Docker Setup

Docker is commonly used to run Kafka locally without manual installation.

services:
 kafka:
  image: apache/kafka
  ports:
   - 9092:9092

Kafka Monitoring Commands

.\bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092 --list

Check Consumer Group Details

.\bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --group user-service-group

Common Kafka Production Problems

• Consumer lag increasing
• Broker disk full
• Incorrect replication factor
• Message serialization errors
• Slow consumers
• Partition imbalance

Kafka Interview Questions

• Difference between Kafka topic and partition
• How Kafka guarantees ordering
• What is consumer group
• What happens when broker fails
• Difference between ZooKeeper and KRaft
• How Kafka handles message recovery

Kafka Setup Checklist

• Install Java
• Download Kafka
• Configure broker properties
• Generate cluster ID
• Format KRaft storage
• Start Kafka broker
• Create topics
• Test producer and consumer
• Connect Spring Boot application
• Configure retries and monitoring

Final Summary

• Kafka is a distributed event streaming platform
• KRaft removes ZooKeeper dependency
• Topics store events and partitions provide scalability
• Spring Boot makes Kafka integration simple
• Production Kafka requires monitoring and recovery planning
• Never delete Kafka metadata directories in production