Building Search Analytics Microservice Using Kafka
A large-scale search system does not only return suggestions. It also learns from user behavior. Every search request provides valuable information such as popular keywords, trending topics and user preferences. In this part, we will build an analytics pipeline using Kafka.
Why Use Kafka for Search Analytics?
Search requests must remain fast. We should not slow down the user request by performing analytics calculations synchronously. Kafka allows us to send search events asynchronously and process them separately.
User
|
|
React
|
|
Search Service
|
|
Kafka Topic
(search-events)
|
|
Analytics Service
|
|
PostgreSQL + Redis
Create Analytics Microservice
We will create a separate Spring Boot application responsible only for analytics processing.
analytics-service
src/main/java
com.example.analytics
├── consumer
│ └── SearchEventConsumer.java
│
├── entity
│ └── SearchHistory.java
│
├── repository
│ └── SearchHistoryRepository.java
│
└── service
└── RankingService.javaAnalytics Service Dependencies
- Spring Web
- Spring Data JPA
- Spring Kafka
- PostgreSQL Driver
- Spring Data Redis
- Lombok
Kafka Configuration
spring:
kafka:
bootstrap-servers: localhost:9092
datasource:
url: jdbc:postgresql://localhost:5432/searchdb
username: admin
password: admin123Create Search Event Model
package com.example.analytics.model;
import lombok.Data;
@Data
public class SearchEvent {
private String keyword;
private Long timestamp;
}Create Search History Table
CREATE TABLE search_history(
id BIGSERIAL PRIMARY KEY,
keyword VARCHAR(255),
search_time BIGINT
);Create Entity
package com.example.analytics.entity;
import jakarta.persistence.*;
import lombok.Data;
@Entity
@Data
public class SearchHistory {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String keyword;
private Long searchTime;
}Repository Layer
public interface SearchHistoryRepository
extends JpaRepository<SearchHistory,Long>{
}Create Kafka Consumer
The consumer listens to the search-events topic and stores user search activity.
@Service
public class SearchEventConsumer {
private final SearchHistoryRepository repository;
public SearchEventConsumer(
SearchHistoryRepository repository){
this.repository=repository;
}
@KafkaListener(
topics="search-events",
groupId="analytics-group")
public void consume(String keyword){
SearchHistory history=new SearchHistory();
history.setKeyword(keyword);
history.setSearchTime(System.currentTimeMillis());
repository.save(history);
}
}Create Kafka Producer in Search Service
After every successful search, the Search Service publishes an event.
@Service
public class SearchEventProducer {
private final KafkaTemplate<String,String> kafkaTemplate;
public void send(String keyword){
kafkaTemplate.send(
"search-events",
keyword
);
}
}Complete Event Flow
User searches: spring boot
|
|
Spring Boot Search API
|
|
Return Results Immediately
|
|
Publish Kafka Event
|
|
Analytics Service
|
|
Store Search HistoryTrending Search Using Redis
Redis Sorted Set is useful for ranking because every keyword can have a score. Higher score means more popular search.
@Service
public class TrendingService {
private final RedisTemplate<String,String> redisTemplate;
public void increase(String keyword){
redisTemplate
.opsForZSet()
.incrementScore(
"trending-searches",
keyword,
1
);
}
}Get Top Trending Searches
Set<String> trending =
redisTemplate
.opsForZSet()
.reverseRange(
"trending-searches",
0,
10
);Search Ranking Algorithm
Autocomplete results should not only match text. They should also consider popularity and user behavior.
Final Score =
Keyword Popularity * 0.5
+
Recent Searches * 0.3
+
Trending Score * 0.2Example Ranking
Create Ranking Service
@Service
public class RankingService {
public double calculate(
long popularity,
long history,
long trend){
return
(popularity * 0.5)
+
(history * 0.3)
+
(trend * 0.2);
}
}Testing Kafka Flow
- Start Kafka container
- Start Search Service
- Start Analytics Service
- Search keywords from React
- Verify Kafka event creation
- Check search_history table
- Check Redis trending scores
Completed Features
- Kafka event driven architecture
- Analytics microservice
- Search history tracking
- Trending keyword calculation
- Popularity based ranking
Next Part
In the next part, we will build the React frontend with TypeScript, debounce search, Google-style dropdown suggestions, API integration and AWS S3 deployment.