Nginx-JSON-Log-Analyzer

What is it

Nginx-JSON-Log-Analyzer is a lightweight (simplistic) JSON format log analyzer, used to analyze Nginx access logs for
myself.

Nginx-JSON-Log-Analyzer is written in Go programming language, needs only a 2 MB executable file to run, currently
supported features are as follows:

  • Filter logs based on the request time
  • Analyze multiple files at the same time
  • Analyze .gz compressed files
  • Support a variety of statistical indicators

Advantages compared to GoAccess

GoAccess is an excellent and powerful real-time web log analyzer, interactive viewer that runs in a terminal in *nix
systems or through your browser. But as far as I know, GoAccess seems does not support counting URI response time by
percentile, Nginx-JSON-Log-Analyzer supports this feature.

If I knew about GoAccess before developing Nginx-JSON-Log-Analyzer, I might choose to use it directly. GoAccess is so
powerful, I love GoAccess.

Advantages compared to ELK

Although ELK is powerful, it is troublesome to install and configure, and it also has certain requirements for machine
performance. Nginx-JSON-Log-Analyzer is more lightweight and easier to use, suitable for some simple log analysis
scenarios.

Quick start

Installation

Just download the binary executable file for the corresponding platform from the
GitHub Release page of Nginx-JSON-Log-Analyzer.

GeoIP2 and GeoLite2

GeoIP2 is a commercial IP geolocation database, need to pay to use
it. GeoLite2 is a free and low-precision version of
GeoIP2, distribute by Attribution-ShareAlike 4.0 International
license, download by logging in to the MaxMind official
website.

When using Nginx-JSON-Log-Analyzer, if you need to resolve the geographic location of the IP (that is, use the -t 4
mode), then you will need to download the GeoIP2 or GeoLite2 City database file, save it to the City.mmdb file in the
default configuration directory ${HOME}/.config/nginx-json-log-analyzer/. The corresponding shell commands are as
follows:

~$ mkdir -p ${HOME}/.config/nginx-json-log-analyzer
~$ tar -xzf GeoLite2-City_20211109.tar.gz
~$ cp GeoLite2-City_20211109/GeoLite2-City.mmdb ${HOME}/.config/nginx-json-log-analyzer/City.mmdb

Configure Nginx

Nginx-JSON-Log-Analyzer can only parse Nginx access logs in JSON format, so you need to add the following log_format
and access_log directives in the Nginx configuration:

log_format json_log escape=json '{"time_iso8601":"$time_iso8601",'
                                '"remote_addr":"$remote_addr",'
                                '"request_time":$request_time,'
                                '"request":"$request",'
                                '"status":$status,'
                                '"body_bytes_sent":$body_bytes_sent,'
                                '"http_user_agent":"$http_user_agent"}';
access_log /path/to/access.json.log json_log;
  • The log_format directive can only appear in the http context;
  • The access_log directive could appear in the http, server, location context, and should use the log_format
    declared above;
  • You can make multiple access_logs at the same time without deleting the original configuration. e.g.

    access_log /path/to/access.log;
    access_log /path/to/access.json.log json_log;
    

Related document: http://nginx.org/en/docs/http/ngx_http_log_module.html

Command line options

show version -v

The -v options show Nginx-JSON-Log-Analyzer’s build version, build time, and Git Commit at build time.

specify the configuration directory -d

The -d option specify the configuration directory that Nginx-JSON-Log-Analyzer required at runtime, the default value
is ${HOME}/.config/nginx-json-log-analyzer/.

specify the analysis type -t

The -t option specify the type of this analysis, the analysis type and corresponding statistical indicators are as
follows:

Supported Analysis Type -t Statistical Indicators Required Fields or Libraries
0 PV and UV $remote_addr
1 Most visited IPs $remote_addr
2 Most visited URIs $request
3 Most visited User-Agents $http_user_agent
4 Most visited user countries and cities $remote_addr, MaxMind GeoIP2 or GeoLite2 City Database
5 Most frequent response status $status, $request
6 Largest URI mean response times $request, $request_time
7 Largest URI percentile response times, e.g. p1(min), p50(median), p95, p100(max) $request, $request_time

limit the analysis start and end time -ta -tb

-ta and -tb options are used to filter logs based on the request time, ta is the abbreviation of time after, tb
is the abbreviation of time before.

-ta and -tb options required the $time_iso8601 field in log_format directive of Nginx configuration.

limit the output lines number -n -n2

-n and -n2 options are used to limit the number of output lines of Nginx-JSON-Log-Analyzer, -n2 option only works
in -t 4 mode.

specify the percentile value -p

The -p option specify the percentile value in the -t 7 mode, the default value is 95.

Usages

Filter logs based on the request time

image

Analyze multiple files at the same time

image

Analyze .gz compressed files

image

Count the most visited IPs

image

Count the most visited URIs

image

Count the most visited User-Agents

image

Count the most visited countries and cities

image

Count the most frequently response status

image

count the largest URI mean response times

image

count the largest URI percentile response times

image

FQA

Q: Will it support real-time analysis in the future?

A: No. If you want this feature, it is recommended to use solutions such as GoAccess, ELK, Grafana + Time Series DBMS.

Q: Is it possible to parse the existing logs before configure Nginx?

A: Not currently. Nginx-JSON-Log-Analyzer can only parse logs in JSON format, may support parsing combined format logs
in the future.

License

GeoLite2 Database License

Nginx-JSON-Log-Analyzer License

Copyright (c) 2021 fantasticmao

GitHub

View Github