/ Job Scheduler

A lightweight and language-independent job queue system

A lightweight and language-independent job queue system

Fireworq

Fireworq is a lightweight, high-performance job queue system with the following abilities.

  • Portability - It is available from ANY programming language which can talk HTTP. It works with a single binary without external dependencies.
  • Reliability - It is built on top of RDBMS (MySQL), so that jobs won't be lost even if the job queue process dies. You can apply an ordinary replication scheme to the underlying DB for the reliability of the DB itself.
  • Availability - It supports primary/backup nodes. Only one node becomes primary simultaneously and the others become backup nodes. A backup node will automatically be active when the primary node dies.
  • Scalability - It always works with a single dispatcher per queue which can concurrently dispatch jobs to workers via HTTP. Scalability of workers themselves should be maintained by a load balancer in the ordinary way. This means that adding a worker will never harm performance of grabbing jobs from a queue.
  • Flexibility - It supports the following features.
  • Multiple queues - You can define multiple queues and use them in different ways: for example, one for a low priority queue for a limited number of high latency workers and another one for a high priority queue for a large number of low latency workers.
  • Delayed jobs - You can specify a delay for each job, which will make the job dispatched after the delay.
  • Job retrying - You can specify the maximum number of retries for each job.
  • Maintainability - It can be managed on a Web UI. It also provides metrics suitable for monitoring.

GitHub