Ingestor

The ingestor is an HTTP API that reads JSON streams of sensor data and publishes them onto a configured event bus.

Design

Streams of JSON-encoded objects are received via the HTTP handler, here is a sample format:

{"sensor": "speed", "value": 55.5, "timestamp": "2012-04-23T18:25:43.511Z"}
{"sensor": "speed", "value": 60.2, "timestamp": "2012-04-23T18:26:43.511Z"}
{"sensor": "speed", "value": 61.9, "timestamp": "2012-04-23T18:27:43.511Z"}

The reason for accepting JSON streams is that in the event that there is no internet connection for the on-board instruments (which is likely for some portions of the trip), they can be stored on disk. Once the connection is reestablished, these local files can be streamed to the ingestor. For the same reason, each reading contains a timestamp of when it was taken.

Endpoints

The list below describes all HTTP endpoints exposed by the ingestor:

  • POST /ingest – The endpoint where sensor data is accepted.
  • GET /ready – The endpoint where server readiness can be determined.
  • GET /live – The endpoint where server liveness can be determined.

Configuration

The ingestor provides a few configuration values as command-line flags:

  • --event-writer-url – The URL that describes which event bus to send messages to
  • --api-key – The API key to accept from inbound requests.
  • --port – The port for listening to inbound HTTP requests.

Deployment

This repository contains a Kustomize manifest in the manifests directory which can be used to run one or more ingestor instances in a Kubernetes cluster. You can use make kustomize to produce a static manifest that contains everything you need. You will need to override the ConfigMap and Secret resources for your specific setup.

Flow

flowchart LR
    subgraph Lada
        direction LR
        A[Take Reading] --> B{Is internet available?}
        B -->|No| C[Store reading on disk]
        C --> A
    end
    
    subgraph Ingestor
        direction LR
        B -->|Yes| D[Handle JSON stream]
        D --> E[Publish to event bus]
    end

GitHub

View Github