Bundle Stream Logo

Bundle Streamer 📦

BundleStreamer is a golang project for building and streaming zip files from a series of web links. For example, if you have 200 files on S3, and you want to download a zip file of them, you can do so in 1 request to this server.

Highlights include:

  • Low memory: the files are streamed out to the client immediately
  • Low CPU: the default server doesn’t compress files, only packages them into a zip, so there’s minimal CPU load (configurable)
  • High concurrency: the two properties above allow a single small server to stream hundreds of large zips simultaneous
  • It includes an HTTP server but can be used as a library (see bundle_streamer.go).

Note

This project is a fork of zip-streamer.

Deploy – Heroku

Deploy

Be sure to enable session affinity if you’re using multiple servers and /create_download_link.

Deploy – Docker

View on Docker Hub

docker pull almogtavor/bundle-streamer
docker run --rm -p '4008:4008' almogtavor/bundle-streamer

HTTP Endpoints

POST /download_zip

This endpoint takes a post and returns a zip file.

It expects a JSON body defining which files to include in the zip. The ZipPath is the path and the filename in the resulting zip file (it should be a relative path).

Example body:

{
  "entries": [
    {"Url":"https://server.com/image1.jpg","ZipPath":"image1.jpg"},
    {"Url":"https://server.com/image2.jpg","ZipPath":"in-a-sub-folder/image2.jpg"}
  ]
}

By default, this route doesn’t compress data. If one would like that to be enabled, the DEFLATE env variable of the server can be configured to DEFLATE=1, and then this option will be enabled.

POST /download_tar

This endpoint works the same as /download_zip, but with tar bundling. Besides, since the .tar the format does not compress data, the DEFLATE env variable won’t affect this route.

POST /create_download_link

This endpoint creates a temporary link that can be used to download a zip via a GET. This is helpful as on a web app it can be painful to POST results and trigger a “Save File” popup with the result. With this, you can create the link in a POST, then open the download link in a new window.

Important:

  • This stores the link in an in-memory cache, so it’s not suitable for deploying to a cluster. However, if using Heroku and requests are coming from a browser, you can use a cluster if you enable session affinity which ensures requests from a given client are routed to the same server.
  • These links are only live for 60 seconds. They are expected to be used immediately and are not long living.
  • The timeout for the streaming session is 900 seconds.

It expects the same body format as /download.

Here is an example response body:

{
  "status":"ok",
  "link_id":"b4ecfdb7-e0fa-4aca-ad87-cb2e4245c8dd"
}

GET /download_link/{link_id}

Call this endpoint with a link_id generated with /create_download_link to download that zip file.

Config

These ENV vars can be used to config the server:

  • PORT – which port the HTTP server binds to. If not set defaults to 4008
  • ZS_URL_PREFIX – if set, requires that the URL of files downloaded start with this prefix. Useful to prevent others from using your server to serve their files.
  • DEFLATE – if set to 1, uses the DEFLATE algorithm in the .zip file. Otherwise, no compression is employed.

Motivation

This project is meant to help the work of package-zipper, which demands a live bundle streamer to work properly.

GitHub

View Github