volume-syncing-operator

Docker container and Kubernetes operator for periodically synchronizing volumes to cloud-native storage, and restoring their state from cloud-native storage.

Features:

  • Supports all storage kinds that are supported by Rclone
  • Rclone configuration using environment variables
  • End-To-End encryption support
  • Periodical synchronization with built-in cron-like scheduler
  • volume-syncing-operator sync-to-remote command to synchronize local files to remote
  • volume-syncing-operator restore command to sync files back from remote to local
  • Support for Kubernetes: initContainer to restore files, and side-car to back up files to remote

Runtime compatibility

🐧 Platform Usage type
Bare metal ✔️ Environment variables
Docker ✔️ Environment variables
Kubernetes ✔️ Environment variables, Helm + operator (mutating kind: Pod by adding initContainer + side-car container)

How it works?

Container and binary are configured with environment variables that are translated into rclone configuration file.

Example usage:

export REMOTE_TYPE=s3
export REMOTE_PROVIDER=Minio
export REMOTE_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export REMOTE_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export REMOTE_ENDPOINT=http://localhost:9000
export REMOTE_ACL=private

volume-syncing-operator sync-to-remote -s ./ -d testbucket/some-directory

Will translate into configuration:

[remote]
type = s3
provider = Minio
access_key_id = AKIAIOSFODNN7EXAMPLE
secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
endpoint = http://localhost:9000
acl = private

And will run:

rclone sync ./ remote:/testbucket/some-directory

Scheduling periodically

volume-syncing-operator has built-in crontab-like scheduler to optionally enable periodical syncing.

volume-syncing-operator --schedule '@every 1m'    # ...
volume-syncing-operator --schedule '0 30 * * * *' # ...

GitHub

View Github