2 October 2019 / Tools

A Faster parser for Redis's Data Persistence

go-redis-parser

A efficient and safe parser in Golang. Its characteristics are: make full use of the language's goroutine, write at the same time as the file is written. This important thing is that you can analyze without connecting to an online service.

 _____ ____        ____  _____ ____  _  ____        ____  ____  ____  ____  _____ ____
/  __//  _ \      /  __\/  __//  _ \/ \/ ___\      /  __\/  _ \/  __\/ ___\/  __//  __\
| |  _| / \|_____ |  \/||  \  | | \|| ||    \_____ |  \/|| / \||  \/||    \|  \  |  \/|
| |_//| \_/|\____\|    /|  /_ | |_/|| |\___ |\____\|  __/| |-|||    /\___ ||  /_ |    /
\____\\____/      \_/\_\\____\\____/\_/\____/      \_/   \_/ \|\_/\_\\____/\____\\_/\_\

go-redis-parser version: alpha 2019-08-24 15:24:34

Usage of:
  -o string
    	set the output directory for gen-file. (default: current directory. parser.(json|csv) will be created)

  -rdb string
    	<rdb-file-name>. For example: ./dump.rdb

  -type string
    	set the gen-file's type, support type: json、csv. (default: csv)
    	 (default "csv")

Feature

Supports Redis from 2.8 to 5.0, all data types except module. Including:

String
Hash
List
Set
SortedSed
Stream(Redis 5.0 new data type)

In addition to exporting all key/values, it also looks for all types of Bigkeys (like redis-cli --bigkeys).

Installation

go-redis-parser will build a binary file，you can use git or go get to install.

via git

$ git clone https://github.com/8090Lambert/go-redis-parser.git && cd go-redis-parser
$ go install

via go

$ go get github.com/8090Lambert/go-redis-parser

Using

Before using, you should set export PATH=$PATH:$GOPATH/bin

$ go-redis-parser -rdb <dump.rdb> -o <gen-file folder> -type <gen-file type, json or csv, default csv>

Generate File

DataType	Key	Value	Size(bytes)
AuxField	redis-ver	5.0.5	0
AuxField	redis-bits	5.0.5	0
AuxField	ctime	5.0.5	0
AuxField	used-mem	5.0.5	0
AuxField	aof-preamble	5.0.5	0
SelectDB	select	0	0
ResizeDB	resize db	{dbSize: 6, expireSize: 0}	0
String	s	a	1
List	li	a,b	2
Set	set	b,a	2
Stream	stream	{"Entries":{"1569553992318-0"...	41
Sortedset	zset	[{"Field":"a","Score":1},{"Field":"b","Score":2}]	2
Hash	h	[{"field":"a","value":"a"}]	2

BigKeys outputs

# Scanning the rdb file to find biggest keys

-------- summary -------

Sampled 6 keys in the keyspace!
Total key length in bytes is 17

Biggest string found 's' has 1 bytes
Biggest   hash found 'h' has 1 fields
Biggest   list found 'li' has 2 items
Biggest sortedset found 'zset' has 2 members
Biggest    set found 'set' has 2 members
Biggest stream found 'stream' has 3 entries

1 string with 1 bytes
1 hash with 1 fields
1 list with 2 items
1 sortedset with 2 members
1 set with 2 members
1 stream with 3 entries