The TinyKV Course
After completing this course, you will have the knowledge to implement a horizontally scalable, highly available, key-value storage service with distributed transaction support. Also, you will have a better understanding of TiKV architecture and implementation.
The whole project is a skeleton code for a key-value server and a scheduler server at the beginning – you need to finish the core logic step by step:
- Standalone KV
- Implement a standalone storage engine.
- Implement raw key-value service handlers.
- Raft KV
- Implement the basic Raft algorithm.
- Build a fault-tolerant KV server on top of Raft.
- Add the support of Raft log garbage collection and snapshot.
- Multi-raft KV
- Implement membership change and leadership change to Raft algorithm.
- Implement conf change and region split on Raft store.
- Implement a basic scheduler.
- Implement the multi-version concurrency control layer.
- Implement handlers of
- Implement handlers of
Similar to the architecture of TiDB + TiKV + PD that separates the storage and computation, TinyKV only focuses on the storage layer of a distributed database system. If you are also interested in the SQL layer, please see TinySQL. Besides that, there is a component called TinyScheduler acting as a center control of the whole TinyKV cluster, which collects information from the heartbeats of TinyKV. After that, the TinyScheduler can generate scheduling tasks and distribute the tasks to the TinyKV instances. All of instances are communicated via RPC.
The whole project is organized into the following directories:
kvcontains the implementation of the key-value store.
raftcontains the implementation of the Raft consensus algorithm.
schedulercontains the implementation of the TinyScheduler, which is responsible for managing TinyKV nodes and generating timestamps.
protocontains the implementation of all communication between nodes and processes uses Protocol Buffers over gRPC. This package contains the protocol definitions used by TinyKV, and the generated Go code that you can use.
logcontains utility to output log based on level.
We provide a reading list for the knowledge of distributed storage system. Though not all of them are highly related with this course, they can help you construct the knowledge system in this field.
Also, you’re encouraged to read the overview of TiKV’s and PD’s design to get a general impression on what you will build:
- TiKV, the design of data storage (English, Chinese).
- PD, the design of scheduling (English, Chinese).
Build TinyKV from Source
git: The source code of TinyKV is hosted on GitHub as a git repository. To work with git repository, please install
go: TinyKV is a Go project. To build TinyKV from source, please install
gowith version greater or equal to 1.13.
Clone the source code to your development machine.
git clone https://github.com/tidb-incubator/tinykv.git
Build TiDB from the source code.
cd tinykv make
It builds the binary of
Run TinyKV with TinySQL
tinysql-serverfollow its document.
- Put the binary of
tinysql-serverinto a single dir.
- Under the binary dir, run the following commands:
mkdir -p data ./tinyscheduler-server ./tinykv-server -path=data ./tinysql-server --store=tikv --path="127.0.0.1:2379"
Now you can connect to the database with an official MySQL client:
mysql -u root -h 127.0.0.1 -P 4000
Any feedback and contribution is greatly appreciated. Please see issues if you want to join in the development.