Grower

Grower is a tool that allows you to write Nginx logs to the Clickhouse columnar database.

Main Features:

  • no dependencies, it works like a regular binary file or in docker
  • ability to write via syslog protocol
  • ability to read logs and write to Clickhouse
  • integrated file rotation
  • configurable buffer size and data reset interval
  • support for replays of unsent data
  • flexible configuration of reading log files
  • type-safe
  • default support native nginx log attributes
  • ability to add your own attributes and types to them
  • configurable multithreading of data processing, both syslog and filelog
  • manual data type management (only for custom columns, maybe there will be for embedded attributes in the future)

TODO:

  • prometheus metrics and dashboard configuration
  • saving corrupted files for manual processing
  • possibility of log native compression
  • native support for more data types
  • native support for complex geographic data types such as GeoIPRegion(ip), GeoIPCity(ip), GeoIPAS(ip)

How to use?

Configuration

is very simple and clear, see sample.yaml configuration file

FileLog server – fast, safe, async, nginx logs parser

With native binary:

go run -race ./cmd/filelog/  \
    --config-file ./sample_test.yaml \
    --bind-address 0.0.0.0:3000 \
    --logs-dir /var/log/nginx \
    --source-log-file access.log \
    --scrape-interval '10s' \
    --backup-files 5 \
    --backup-file-max-age '1m0s' \
    --clickhouse-host 'xxx.xx.xx.xx:9000' \
    --clickhouse-host 'xxx.xx.xx.xx:9001' \
    --clickhouse-user default \
    --clickhouse-database default \
    --clickhouse-password '' \
    --buffer-size 10000 \
    --buffer-flush-interval 5000 \
    --parallelism 5 \
    --debug \
    --auto-create-target-from-scratch \
    --enable-rotating \
    --skip-nginx-reopen \
    --run-rotating-at-startup \
    --rewrite-nginx-local-time

With Docker container:

docker run -d --net=host \
   -v /usr/share/config/:/config/ \
   -e CONFIG_FILE='/config/sample_test.yaml'
   -e BIND_ADDRESS='0.0.0.0:3004' \
   -e SOURCE_LOG_FILE='access.log' \
   -e LOGS_DIR='/var/log/nginx' \
   -e SCRAPE_INTERVAL='1m0s' \
   -e BACKUP_FILES=5 \
   -e BACKUP_FILE_MAX_AGE='5m0s' \
   -e CLICKHOUSE_HOST='xxx.xx.xx.xx:9000,xxx.xx.xx.xx:9001' \
   -e CLICKHOUSE_USER='default' \
   -e CLICKHOUSE_PASSWORD = '' \
   -e CLICKHOUSE_DATABASE='default' \
   -e BUFFER_FLUSH_INTERVAL=2000 \
   -e BUFFER_SIZE=5000 \
   -e PARALLELISM=5 \
   -e RUN_HTTP_SERVER=true \
   -e AUTO_CREATE_TARGET_FROM_SCRATCH \
   -e ENABLE_ROTATING \
   -e SKIP_NGINX_REOPEN \
   -e RUN_ROTATING_AT_STARTUP \
   -e DEBUG=true \
   --name grower-syslog qwx1337/grower-filelog:latest

Local build Docker image:

#!/bin/bash

docker build -t your_image_name:latest -f ./cmd/filelog/Dockerfile .

For more information:

$ go run ./cmd/filelog/main.go --help

Syslog server – system log standard

With native binary:

go run ./cmd/syslog/  \
    --config-file ./sample_test.yaml \
    --bind-address 0.0.0.0:3000 \
    --syslog-unix-socket /tmp/syslog.sock \
    --syslog-udp-address 0.0.0.0:3011 \
    --syslog-tcp-address 0.0.0.0:3012 \
    --listeners 'unix' \
    --listeners 'tcp' \
    --listeners 'udp' \
    --clickhouse-host 'xxx.xx.xx.xx:9000' \
    --clickhouse-host 'xxx.xx.xx.xx:9001' \
    --clickhouse-user default \
    --clickhouse-database default \
    --clickhouse-password '' \
    --buffer-size 5000 \
    --buffer-flush-interval 2000 \
    --parallelism 5 \
    --run-http-server \
    --debug

With Docker container:

docker run -d --net=host \
   -v /usr/share/config/:/config/ \
   -e CONFIG_FILE='/config/sample_test.yaml'
   -e BIND_ADDRESS='0.0.0.0:3004' \
   -e SYSLOG_UNIX_SOCKET='/tmp/syslog.sock' \
   -e SYSLOG_UDP_ADDRESS='0.0.0.0:3011' \
   -e SYSLOG_TCP_ADDRESS='0.0.0.0:3012' \
   -e LISTENERS='unix,tcp,udp' \
   -e CLICKHOUSE_HOST='xxx.xx.xx.xx:9000,xxx.xx.xx.xx:9001' \
   -e CLICKHOUSE_USER='default' \
   -e CLICKHOUSE_PASSWORD = '' \
   -e CLICKHOUSE_DATABASE='default' \
   -e BUFFER_FLUSH_INTERVAL=2000 \
   -e BUFFER_SIZE=5000 \
   -e PARALLELISM=5 \
   -e RUN_HTTP_SERVER=true \
   -e DEBUG=true \
   --name grower-syslog qwx1337/grower-syslog:latest

Local build Docker image:

#!/bin/bash

docker build -t your_image_name:latest -f ./cmd/syslog/Dockerfile .

For more information:

$ go run ./cmd/syslog/main.go --help

GitHub

View Github