🇺🇸English

Kafka Engineer

Purpose

Provides Apache Kafka and event streaming expertise specializing in scalable event-driven architectures and real-time data pipelines. Builds fault-tolerant streaming platforms with exactly-once processing, Kafka Connect, and Schema Registry management.

When to Use

• Designing event-driven microservices architectures
• Setting up Kafka Connect pipelines (CDC, S3 Sink)
• Writing stream processing apps (Kafka Streams / ksqlDB)
• Debugging consumer lag, rebalancing storms, or broker performance
• Designing schemas (Avro/Protobuf) with Schema Registry
• Configuring ACLs and mTLS security

2. Decision Framework

Architecture Selection

What is the use case?
│
├─ **Data Integration (ETL)**
│  ├─ DB to DB/Data Lake? → **Kafka Connect** (Zero code)
│  └─ Complex transformations? → **Kafka Streams**
│
├─ **Real-Time Analytics**
│  ├─ SQL-like queries? → **ksqlDB** (Quick aggregation)
│  └─ Complex stateful logic? → **Kafka Streams / Flink**
│
└─ **Microservices Comm**
   ├─ Event Notification? → **Standard Producer/Consumer**
   └─ Event Sourcing? → **State Stores (RocksDB)**

Config Tuning (The "Big 3")

1. Throughput: batch.size, linger.ms, compression.type=lz4.
2. Latency: linger.ms=0, acks=1.
3. Durability: acks=all, min.insync.replicas=2, replication.factor=3.

Red Flags → Escalate tosre-engineer:

• "Unclean leader election" enabled (Data loss risk)
• Zookeeper dependency in new clusters (Use KRaft mode)
• Disk usage > 80% on brokers
• Consumer lag constantly increasing (Capacity mismatch)

3. Core Workflows

Workflow 1: Kafka Connect (CDC)

Goal: Stream changes from PostgreSQL to S3.

Steps:

1. Source Config (postgres-source.json)

   {
     "name": "postgres-source",
     "config": {
       "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
       "database.hostname": "db-host",
       "database.dbname": "mydb",
       "database.user": "kafka",
       "plugin.name": "pgoutput"
     }
   }
   

2. Sink Config (s3-sink.json)

   {
     "name": "s3-sink",
     "config": {
       "connector.class": "io.confluent.connect.s3.S3SinkConnector",
       "s3.bucket.name": "my-datalake",
       "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
       "flush.size": "1000"
     }
   }
   

3. Deploy

   • curl -X POST -d @postgres-source.json http://connect:8083/connectors

Workflow 3: Schema Registry Integration

Goal: Enforce schema compatibility.

Steps:

1. Define Schema (user.avsc)

   {
     "type": "record",
     "name": "User",
     "fields": [
       {"name": "id", "type": "int"},
       {"name": "name", "type": "string"}
     ]
   }
   

2. Producer (Java)

   • Use KafkaAvroSerializer.
   • Registry URL: http://schema-registry:8081.

5. Anti-Patterns & Gotchas

❌ Anti-Pattern 1: Large Messages

What it looks like:

• Sending 10MB images payload in Kafka message.

Why it fails:

• Kafka is optimized for small messages (< 1MB). Large messages block the broker threads.

Correct approach:

• Store image in S3.
• Send Reference URL in Kafka message.

❌ Anti-Pattern 2: Too Many Partitions

What it looks like:

• Creating 10,000 partitions on a small cluster.

Why it fails:

• Slow leader election (Zookeeper overhead).
• High file handle usage.

Correct approach:

• Limit partitions per broker (~4000). Use fewer topics or larger clusters.

❌ Anti-Pattern 3: Blocking Consumer

What it looks like:

• Consumer doing heavy HTTP call (30s) for each message.

Why it fails:

• Rebalance storm (Consumer leaves group due to timeout).

Correct approach:

• Async Processing: Move work to a thread pool.
• Pause/Resume: consumer.pause() if buffer is full.

7. Quality Checklist

Configuration:

• Replication: Factor 3 for production.
• Min.ISR: 2 (Prevents data loss).
• Retention: Configured correctly (Time vs Size).

Observability:

• Lag: Consumer Lag monitored (Burrow/Prometheus).
• Under-replicated: Alert on under-replicated partitions (>0).
• JMX: Metrics exported.

Examples

Example 1: Real-Time Fraud Detection Pipeline

Scenario: A financial services company needs real-time fraud detection using Kafka streaming.

Architecture Implementation:

1. Event Ingestion : Kafka Connect CDC from PostgreSQL transaction database
2. Stream Processing : Kafka Streams application for real-time pattern detection
3. Alert System : Producer to alert topic triggering notifications
4. Storage : S3 sink for historical analysis and compliance

Pipeline Configuration:

Key Logic:

• Detects velocity patterns (5+ transactions in 1 minute)
• Identifies geographic anomalies (impossible travel)
• Flags high-risk merchant categories

Results:

• 99.7% of fraud detected in under 100ms
• False positive rate reduced from 5% to 0.3%
• Compliance audit passed with zero findings

Example 2: E-Commerce Order Processing System

Scenario: Build a resilient order processing system with Kafka for high reliability.

System Design:

1. Order Events : Topic for order lifecycle events
2. Inventory Service : Consumes orders, updates stock
3. Payment Service : Processes payments, publishes results
4. Notification Service : Sends confirmations via email/SMS

Resilience Patterns:

• Dead Letter Queue for failed processing
• Idempotent producers for exactly-once semantics
• Consumer groups with manual offset management
• Retries with exponential backoff

Configuration:

# Producer Configuration
acks: all
retries: 3
enable.idempotence: true

# Consumer Configuration
auto.offset.reset: earliest
enable.auto.commit: false
max.poll.records: 500

Results:

• 99.99% message delivery reliability
• Zero duplicate orders in 6 months
• Peak processing: 10,000 orders/second

Example 3: IoT Telemetry Platform

Scenario: Process millions of IoT device telemetry messages with Kafka.

Platform Architecture:

1. Device Gateway : MQTT to Kafka proxy
2. Data Enrichment : Stream processing adds device metadata
3. Time-Series Storage : S3 sink partitioned by device_id/date
4. Real-Time Alerts : Threshold-based alerting for anomalies

Scalability Configuration:

• 50 partitions for parallel processing
• Compression enabled for cost optimization
• Retention: 7 days hot, 1 year cold in S3
• Schema Registry for data contracts

Performance Metrics:

Best Practices

Topic Design

• Naming Conventions : Use clear, hierarchical topic names (domain.entity.event)
• Partition Strategy : Plan for future growth (3x expected throughput)
• Retention Policies : Match retention to business requirements
• Cleanup Policies : Use delete for time-based, compact for state
• Schema Management : Enforce schemas via Schema Registry

Producer Optimization

• Batching : Increase batch.size and linger.ms for throughput
• Compression : Use LZ4 for balance of speed and size
• Acks Configuration : Use all for reliability, 1 for latency
• Retry Strategy : Implement retries with backoff
• Idempotence : Enable for exactly-once semantics in critical paths

Consumer Best Practices

• Offset Management : Use manual commit for critical processing
• Batch Processing : Increase max.poll.records for efficiency
• Rebalance Handling : Implement graceful shutdown
• Error Handling : Dead letter queues for poison messages
• Monitoring : Track consumer lag and processing time

Security Configuration

• Encryption : TLS for all client-broker communication
• Authentication : SASL/SCRAM or mTLS for production
• Authorization : ACLs with least privilege principle
• Quotas : Implement client quotas to prevent abuse
• Audit Logging : Log all access and configuration changes

Performance Tuning

• Broker Configuration : Optimize for workload type (throughput vs latency)
• JVM Tuning : Heap size and garbage collector selection
• OS Tuning : File descriptor limits, network settings
• Monitoring : Metrics for throughput, latency, and errors
• Capacity Planning : Regular review and scaling assessment

Security:

• Encryption: TLS enabled for Client-Broker and Inter-broker.
• Auth: SASL/SCRAM or mTLS enabled.
• ACLs: Principle of least privilege (Topic read/write).

Weekly Installs

157

Repository

404kidwiz/claud…e-skills

GitHub Stars

43

First Seen

Jan 24, 2026

Security Audits

Gen Agent Trust HubPassSocketPassSnykPass

Installed on

opencode136

codex127

gemini-cli125

github-copilot125

cursor114

kimi-cli103