Using golang to produce data to kinesis data stream
What is this
The idea behind this repo was to quickly determine how easy it would be to add a serverless function
to consume an API and add the data to a kinesis data stream.
- Install golang: https://golang.org/doc/install
- Compile the program:
GOOS=linux go build dataConsumer.go
- Zip up the program:
zip function.zip dataConsumer
- Install terraform: https://learn.hashicorp.com/tutorials/terraform/install-cli
- Navigate to infra folder:
- Apply the infrastructure:
terraform apply --auto-approve
Up next would be to either consume in a lambda function and perform some sort of ETL work
prior to storing in to a database.
The other option is to send it to AWS GLUE to perform the ETL to continue learning the data
ingestion then storing in to a database.