Deploy a Submission Layer

This page contains:

a summary of what is involved when deploying a Five Safes TES Submission Layer
a Sample Deployment Guide
- with recommendations for deploying in Production

Component Summary

Here is a summary of the required components which make up the Submission Layer:

Component	Notes
KeyCloak	The Submission apps require a Keycloak realm containing users and clients authorised to access them
Minio	An Amazon S3 compatible Submission Storage service
RabbitMQ	A message broker for queueing submissions for target TREs
PostgreSQL	A database for the Submission apps to keep local state
Submission API	A REST API for Submission Layer functionality, including the GA4GH TES API
Submission GUI	A Web Frontend for users to log into and interact with the Submission Layer

You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.

For some components, such as KeyCloak or MinIO, you may wish to use your own existing deployments.

User accessible components

A subset of the components will need to be accessible by users, outside of the host environent:

Component	Reason
KeyCloak	So that users can authenticate and admins can manage the Submission KeyCloak Realm
Minio	So that Researchers and TREs can upload and download Submission data (inputs or outputs)
Submission GUI	So that users can log into and interact with the Submission Layer
Submission API	So that Researchers and TREs can interact with the Submission Layer via a REST API

Sample Deployment Guide

Note that this guide is a sample. It can be modified to reflect your own infrastructure and configuration choices.

This guide deploys all the components of the Submission Layer on a single host (e.g. a Virtual Machine) using Docker Compose.

All the components are deployed on suitable Docker networks and can communicate with each other as required.

It also automatically configures the KeyCloak realm, ready for following our other guides.

User accessible components

Components which may need to be accessed from outside the host are forwarded to the host on specific ports

Component	Ports
KeyCloak	HTTP: `8085`
MinIO	API HTTP: `9000`, GUI HTTP: `9001`
Submission GUI	HTTP: `7220`
Submission API	HTTP: `5034`

This sample also includes some additional components useful for diagnostic purposes:

Component	Notes	Ports
Seq	Logs aggregator and web-based viewer	HTTP: `5341`
Adminer	Web-based PostgreSQL client	HTTP: `8087`

Production Recommendations

To use this sample as a starting point for a production deployment, some general recommendations should be considered.

The ports specified above can be used for configuring a reverse proxy for access to the necessary components.

💡

If you are doing reverse proxy for MinIO, the Websockets Support should be enabled for MinIO GUI Proxy host.

Get Started

Prerequisites

Docker and Docker Compose installed. For Linux/Ubuntu VMs, you can follow this guide
Git installed on your machine.

Deployment Steps

To deploy a Submission Layer:

Clone the repository

git clone https://github.com/SwanseaUniversityMedical/5S-TES-deployment.git

Prepare the necessary files

To deploy a Submission Layer, we need the following files and directories in 5S-TES-deployment repository:

DeploymentStack/Submission/docker-compose.yml
DeploymentStack/Submission/config/
ServiceStack/
.env, which needs to be configured with the appropriate environment variables based on the DeploymentStack/Submission/.env.example file.

The docker-compose.yml file in the DeploymentStack/Submission/ directory includes the services for the Submission Layer using relative paths to the compose manifests in the ServiceStack/ directory, where they are defined.

If you need to modify the compose manifests or the docker-compose.yml file, we recommend to move these four components to a separate directory, e.g., /srv/submission, and update the relative paths in the docker-compose.yml file and CONFIG_PATH in the .env file accordingly. When in sync with the remote repository, the local 5S-TES-deployment repository will then only serve as a reference for your own deployment, rather than the source of truth.

The compose manifests used in the Submission Layer are:

ServiceStack/compose-manifests/applications/submission-layer.yml which defines the Submission Layer services, i.e., Submission API, Submission GUI.
ServiceStack/compose-manifests/shared/platform.yml which defines the platform services, i.e., PostgreSQL, Adminer, Serilog, RabbitMQ.
ServiceStack/compose-manifests/shared/auth.yml which defines the authentication services, i.e., Keycloak. This is optional and only needed if you want to use Keycloak for authentication internally within the Submission Layer.
ServiceStack/compose-manifests/storage/minio-submission.yml which defines the Submission Layer’s Minio server.

💡
Comment or remove the line ../../ServiceStack/compose-manifests/shared/auth.yml in the docker-compose.yml file if you don’t want to use it.

Configure the .env file

Create a new .env file in the directory where you store the docker-compose.yml file, and configure the environment variables based on the .env.example file in the DeploymentStack/Submission/ directory or here.

The descriptions of the essential environment variables and the guide to set them are as follows:

Environment Variable	Description
`DEPLOYMENT_VERSION`	The version of the Submission Layer app you are deploying. Find the version here, between `v` and `-containers`. For example, `2.15.1` is the version for the release `v2.15.1-containers`.
`PGLOGIN` and `PGPASSWORD`	The admin credentials for the PostgreSQL database used by the Submission Layer.
`SerilogLevel`	The Serilog logging level for the Submission Layer. Default is `Information`. Change this depending on the logging level you want to set for the Submission Layer. More details here.
`KeycloakHostName`	The hostname of the Keycloak server. For example, `my-keycloak`. This is only used if you are using a local Keycloak service defined in the `ServiceStack/compose-manifests/shared/auth.yml` file.
`SUBMISSION_KEYCLOAK_URL`	The URL of the Keycloak server. For example, `http://keycloak:8080` (if using a local Keycloak server) or `https://my-submission-keycloak.net` (if using a remote Keycloak server).
`MinioOpenidSecret`	The OpenID secret for the Minio client of Dare-Control realm. There is a default value in the realm configuration, but you should regenerate it for production deployments. Do this by navigating to `Dare-Control` realm -> `Clients` -> `Dare-Control-Minio` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`MinioRootUser`	The root user for the Minio server
`MinioRootPass`	The root password for the Minio server
`submissionMinioUrl`	The URL for the Submission Layer’s Minio server, for internal access within Docker network. For example, `http://minioSubmission:9000`
`submissionMinioAdminConsole`	The URL for the Submission Layer’s Minio server’s admin console. For example, `http://localhost:9001`
`SubmissionAPIKeyCloakSecret`	Same as `MinioOpenidSecret`, but this one is for `Dare-Control-API` client.
`SubmissionUIClientSecret`	Same as `MinioOpenidSecret`, but this one is for `Dare-Control-UI` client.
`SubmissionKeyCloakClientUIRedirectURL`	The URL for the Submission Layer’s Keycloak redirect URI.
`SubmissionKeyCloakServer`	The URL for the Submission Layer’s Keycloak server. For example, `keycloak:8080` or `my-keycloak.net`
`SubmissionServerProtocol`	The protocol for the Submission Layer’s Keycloak server. Default is `http`. If you are using HTTPS for Keycloak, set this to `https`.
`URLSettingsFrontEndQueryImageSQL`	The default SQL query image for the Submission Layer’s UI. It is used in the `Simple SQL Query` mode in the Wizard.
`SubmissionTokenRefreshSeconds`	The number of seconds to refresh the Submission Layer’s Keycloak token. Default is `3600`.
`SubmissionAutoTrustKeycloakCert`	If this is set to `true`, the Submission Layer will automatically trust the Keycloak certificate. Default is `false`.
`CONFIG_PATH`	The path for the to the shared configuration files (e.g., realm config, init scripts) for the Submission Layer. Default is `../../../DeploymentStack/Submission/config`.
`DemoMode`	Set to true if you’d like to simulate execution, otherwise default to false.
`KeyCloakDemoMode`	Allows Keycloak to not require https. Default is true.
`sslcookies`	If this is set to `true`, the Submission Layer will use secure SSL cookies. Default is `false` to prevent issues with Keycloak.
`httpsRedirect`	Disable HTTPS redirect (setting this to `false`) if using http only site to prevent silent redirect to non existent https site
`SuppressAntiforgery`	If this is set to `true`, the Submission Layer will suppress antiforgery. Default is `false`.

You can find an example .env file here.

Run docker compose

At the directory where docker-compose.yml and .env files are located, run the following command to start the Submission Layer:

docker compose up -d

Check-in

After the containers are up and running (check with docker ps), if you have configured the ports for the Submission layer components for user accessibility, you can access the Submission Layer UI by navigating to http://<hostname>:7220 in your browser.

If you are using the default Keycloak service defined in the ServiceStack/compose-manifests/shared/auth.yml file, you can access the Keycloak Client portal by navigating to port 8085 of the host machine, and Then, the default to access KeyCloak admin portal are username admin and admin; the default username and password for the Submission Layer are globaladminuser and password123.

💡

If you encounter the error “Invalid parameter: redirect_uri” from Keycloak when trying to login into Submission Layer UI, please make sure the redirectURL environment variables in the .env file, the URL you put on the browser’s address bar, and the Valid redirect URIs registered in Keycloak’s client settings are the same.

Recommended next steps

Once the TRE Agent is deployed, connect it to the Submission Layer by following this guide.
Submit a Hello World submission to the Submission Layer by following this guide.
Submit a simple analysis submission to the Submission Layer by following this guide.
Add a new user in the Dare-Control Keycloak realm with appropriate permissions, then use this user credentials to access the Submission Layer UI.
Update passwords or secrets for the Submission Layer, including the Keycloak client secrets, Minio server credentials, etc.

Debugging

If you encounter any issues, you can check the logs of the containers by running the following command:

docker logs <container-name> -f

Alternatively, you can check the logs of the stack by accessing port 5341 (Seq service) of the host machine.

Requirements Add a new researcher