Kafka/distributed commit log service in Go.
Goals of this project:
- Implement Kafka in Go
- Protocol compatible with Kafka so Kafka clients and services work with Jocko
- Make operating simpler
- Distribute a single binary
- Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
- Smarter configuration settings
- Able to use percentages of disk space for retention policies rather than only bytes and time kept
- Handling size configs when you change the number of partitions or add topics
- Learn a lot and have fun
- Partition consensus and distribution
- Create Topics
- Delete Topics
- Consumer group [current task]
- API versioning [more API versions to implement]
- Replication [first draft done – testing heavily now]
Hiatus Writing Book
I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.
- How Jocko’s built-in service discovery and consensus works
- How Jocko’s (and Kafka’s) storage internals work
├── broker broker subsystem ├── cmd commands │ └── jocko command to run a Jocko broker and manage topics ├── commitlog low-level commit log implementation ├── examples examples running/using Jocko │ ├── cluster example booting up a 3-broker Jocko cluster │ └── sarama example producing/consuming with Sarama ├── protocol golang implementation of Kafka's protocol ├── prometheus wrapper around Prometheus' client lib to handle metrics ├── server API subsystem └── testutil test utils └── mock mocks of the various subsystems
$ go get github.com/travisjeffery/jocko
$ cd $GOPATH/src/github.com/travisjeffery/jocko $ make build
(If you see an error about
depnot being found, ensure that
$GOPATH/binis in your
docker build -t travisjeffery/jocko:latest .
See CONTRIBUTING for details on submitting patches and the contribution workflow.
Jocko is under the MIT license, see the LICENSE file for details.