Kafka Stress – Stress Test Tool for Kafka Clusters, Producers and Consumers Tunning

Documentation License: MIT Build CI Release Twitter: fidelissauro

Installation

Docker

docker pull fidelissauro/kafka-stress:latest

docker run --network host -it fidelissauro/kafka-stress:latest --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumers 6

MacOS amd64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.5/kafka-stress_0.0.5_darwin_amd64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Linux amd64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.5/kafka-stress_0.0.5_linux_amd64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Linux arm64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.5/kafka-stress_0.0.5_linux_arm64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Freebsd amd64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.5/kafka-stress_0.0.5_freebsd_amd64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

Freebsd arm64

wget https://github.com/msfidelis/kafka-stress/releases/download/v0.0.5/kafka-stress_0.0.5_freebsd_arm64 -O kafka-stress 
mv kafka-stress /usr/local/bin 
chmod +x /usr/local/bin/kafka-stress

v0 Usage

Usage of kafka-stress:
  -ack int
    	Required ACKs to produce messages (default 1)
  -batch-size int
    	Batch size for producer mode
  -bootstrap-servers string
    	Kafka Bootstrap Servers Broker Lists (default "0.0.0.0:9092")
  -consumer-group string
    	Consumer group name (default "kafka-stress")
  -consumers int
    	Number of consumers will be used in topic (default 1)
  -create-topic
    	Auto Create Topic?
  -events int
    	Numer of events will be created in topic (default 10000)
  -schema string
    	Schema
  -schema-registry string
    	Schema Registry URL (default "0.0.0.0:8081")
  -size int
    	Message size in bytes (default 62)
  -ssl-enabled
    	SSL Mode
  -test-mode string
    	Test Type; Ex producer;consumer. Default: producer (default "producer")
  -topic string
    	Kafka Stress Topics (default "kafka-stress")
  -zookeeper-servers string
    	Zookeeper Connection String (default "0.0.0.0:2181")

Producer

kafka-stress --bootstrap-servers localhost:29092 --events 30000 --topic kafka-stress

kafka-stress --bootstrap-servers localhost:29092 --events 10000 --topic kafka-stress

Sent 10000 messages to topic kafka-stress with 0 errors
Tests finished in 1.232463918s. Producer mean time 8113.83/s

Customize ACK’s

Use --ack parameter to customize ack’s quorum

kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 20000 --topic tunning-3 --test-mode producer --ack 1

Consumer

kafka-stress –bootstrap-servers 0.0.0.0:9092 –topic kafka-stress –test-mode consumer –consumers 6


```bash
kafka-stress --bootstrap-servers 0.0.0.0:9092 --events 10000 --topic kafka-stress --test-mode consumer  --consumers 6

...
[Consumer 6] Message from consumer group kafka-stress at topic/partition/offset kafka-stress/1/0: 65b27cc0-1053-4fbc-b7a9-a40972fcaca6 = a124e95d-9226-4eb9-9169-411318bb6e4e
[Consumer 5] Message from consumer group kafka-stress at topic/partition/offset kafka-stress/2/0: 35fc905f-27f3-4b9f-81db-af89c1fa4c28 = 94f06092-589c-4030-8e3f-66a5a980123b
...

Customize consumer-group name

Use --ssl to enable ssl authentication.

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --consumer-group custom-consumer-group

Authentication methods

SSL

Use --ssl to enable ssl authentication.

kafka-stress --bootstrap-servers 0.0.0.0:9092 --topic kafka-stress --test-mode consumer --ssl

Roadmap

Improve reports output

  • Add execution time on reports – :check:
  • Add retry mechanism
  • Add message size options
  • Add test header
  • Add SCRAM authentication
  • Add SASL authetication
  • Add TLS authetication
  • Add IAM authentication for Amazon MSK
  • Add Unit Tests
  • Add time based tests
  • Add Schema Registry / AVRO, JSON, PROTO Support

GitHub

https://github.com/msfidelis/kafka-stress