Running with Docker Compose
Start here once you have SSH access and
sudo privileges for a server or VM.
Docker server installation is only supported on Linux distributions
A DIVE Web deployment consists of 2 main services.
- kitware/viame-web - the web server
- kitware/viame-worker - the queue worker
In addition, a database (MongoDB) and a queue service (RabbitMQ) are required.
SSH into the target server and install these system dependencies.
You can skip this section if you used Ansible to configure your server, as it already installed all necessary dependencies.
- Install NVIDIA Driver (version specified in VIAME)
sudo ubuntu-drivers installusually works.
dockerversion 19.03+ guide
docker-composeversion 1.28.0+ guide
- Install nvidia-container-toolkit
Clone this repository and configure options in
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
VIAME server will be running at http://localhost:8010. You should see a page that looks like this. The default username and password is
If you have a server with a public-facing IP address and a domain name that points to it, you should be able to use our production deployment configuration. This is the way we deploy viame.kitware.com.
containrrr/watchtowerupdates the running containers on a schedule using automated image builds from docker hub (above).
linuxserver/duplicatiis included to schedule nightly backups, but must be manually configured.
You should scale the girder web server up to an appropriate number. This stack will automatically load-balance across however many instances you bring up.
1 2 3 4 5 6 7 8
It's possible to split your web server and task runner between multiple nodes. This may be useful if you want to run DIVE Web without a GPU or if you want to save money by keeping your GPU instance stopped when not in use. You could also increase parallel task capacity by running task runners on multiple nodes.
- Make two cloud VM instances, one with NVIDIA drivers and container toolkit, and one without. This is still a special case of scenario 1 from the Provisioning Guide
- Clone the dive repository on both, and set up
.envon both with the same configuration.
- Be sure that
CELERY_BROKER_URLin particular are uncommented and set to the IP or domain name of your web server. This is how the worker will talk to the web server, so the web server must be network accessible from the worker.
1 2 3 4 5
In order to run any jobs (video transcoding, pipelines, training, addon upgrades) the GPU server will need to be running.
After initial deployment, DIVE Server will require an addon upgrade in order to download and scan for VIAME addons. Run this by issuing a
POST /dive_configuration/upgrade_pipelines request from the swagger UI at
- Whether you
forceor not, only those pipelines from addons from the exact urls passed will be enabled on the server.
- An old addon can be disabled by simply omitting its download from the upgrade payload.
forceshould be used to force re-download of all URLs in the payload even if their zipfiles have been cached.
- An upgrade run is always required if the "common" pipelines in the base image change. These are updated for every run, and do not require
- See the job log to verify the exact actions taken by an upgrade job.
- Optional patches are updated occasionally and you can find the latest urls here.
Server branding config
You can configure the brand and messaging that appears in various places in the DIVE Web UI using the config API.
- Open the swagger page at /api/v1
PUT /dive_configuration/brand_datawhere the body is a JSON object from the template below. If you do not want to set a value and use the default, omit the key and value from the config body.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Web Server config
This image contains both the backend and client.
||a mongodb connection string|
||rabbitmq connection string|
||Address for workers to reach web server|
There is additional configuration for the RabbitMQ Management plugin. It only matters if you intend to allow individual users to configure private job runners in standalone mode, and can otherwise be ignored.
||Management API username|
||Management API password|
||Virtual host should match
||Management API Url|
You can also pass girder configuration and celery configuration.
This image contains a celery worker to run VIAME pipelines and transcoding jobs.
Note: Either a broker url or DIVE credentials must be supplied.
||max concurrnet jobs. Lower this if you run training|
|WORKER_GPU_UUID||null||leave empty to use all GPUs. Specify UUID to use specific device|
||rabbitmq connection string. Ignored in standalone mode.|
||kwiver log level|
|DIVE_USERNAME||null||Username to start private queue processor. Providing this enables standalone mode.|
|DIVE_PASSWORD||null||Password for private queue processor. Providing this enables standalone mode.|
||Remote URL to authenticate against|
You can also pass regular celery configuration variables.
Running the GPU Job Runner in standalone mode
Individual users can run a standalone worker to process private jobs from VIAME Web.
- Install VIAME from the github page to
- Activate the install with
- Install VIAME pipeline addons by running
cd bin && download_viame_addons.shfrom the VIAME install directory.
- Enable the private user queue for your jobs by visiting the jobs page
- Run a worker using the docker command below
--volumemount maps to the host installation. You may need to change the source from
/opt/noaa/viamedepending on your install location, but you should not change the destination from
1 2 3 4 5 6 7 8 9