Nomad Events Sink
Nomad Events Sink is an events collection agent which uses Nomad Events SDK to fetch events.
Events can help debug the cluster state and alert operators about new deployments, failing allocations, node updates etc. Nomad emits these events in a channel and provides an SDK to access these events.
The missing piece was a tool to ingest these events and analyze/visualize them later. This is where
nomad-events-sink agent comes into picture and helps to ingest these events in multiple configurable Sinks.
The tool is designed to be generic in order to handle many use-cases of processing events. Multiple Sink Providers can be created and events will be handled by them. Common usecases included storing events for long term use (Vector->Loki), alerting on cluster state changes (Slack/Rocketchat webhooks) etc.
How does it work?
- A subscription for each topic is created and new events are watched.
- Whenever a new event comes, it gets added to a
- Multiple background workers (Goroutines) are listening to this channel, processing events in FIFO.
- These workers batch events and push to upstream providers (for eg HTTP/S3 etc).
- A background worker commits the event index state to a file on disk. This is done so that the program is able to pickup events from where it left before shutting down.
This program uses 2 batch strategies in order to avoid an inefficient process of making a request for every incoming event:
idle_timeout: If a batch is in memory for more than
idle_timeoutduration, it is flushed to providers.
events_count: If a batch has more events than
events_counts, it is flushed to providers.
You can choose one of the various deployment options:
To deploy on a Nomad cluster, you can refer to this jobspec.
Grab the latest release from Releases.
$ ./nomad-events-sink.bin --config config.toml
docker pull ghcr.io/mr-karan/nomad-events-sink:latest
Refer to this example for quickly getting started with ingesting Deployment Events to Loki using Vector.
config.sample.toml for a list of configurable values.
All config variables can also be populated as env vairables by prefixing
NOMAD_EVENTS_SINK_ and replacing
Nomad API client reads the following environment variables:
You can read about them in detail here.
- Currently only HTTP Provider is implemented. There’s a full working example of ingesting events to the HTTP server provided by Vector and dumping them to Loki.
Please feel free to open a new issue for bugs, new features (like Sink Providers), feedback etc.