Deploy a TRE Agent and Data Egress

This page contains:

a summary of what is involved when deploying a Five Safes TES TRE Layer
a Sample Deployment Guide
- with recommendations for deploying in Production

Component Summary

Here is a summary of the required components which make up the TRE Layer:

Component	Notes
KeyCloak	The TRE Agent and Egress apps each require a Keycloak realm containing users and clients authorised to access them
Minio	An Amazon S3 compatible TRE Storage service
RabbitMQ	A message broker used for queueing
PostgreSQL	A database for the TRE Agent and Egress apps to keep local state
Hashicorp Vault	Used for accessing ephemeral credentials as secrets
OpenLDAP	Used to provide ephemeral credentials for Trino (if in use as a datasource)
Camunda Connectors	A REST API allowing other services (e.g. TRE Agent) to integrate with Camunda
Camunda Orchestration	A consolidated service that combines Camunda functionality via Zeebe + Operate + Tasklist
ElasticSearch	Used for Camunda’s local datastore such as workflow state
TRE-Camunda	A service defining the handlers for ephemeral credentials management. Registers the handlers with Camunda, via Zeebe
TRE Agent API	A REST API for TRE Agent functionality
TRE Agent GUI	A Web Frontend for TRE Admins to log into and interact with the TRE Agent
Egress API	A REST API for Egress Portal functionality
Egress GUI	A Web Frontend for Egress Officers to log into and interact with the Egress Portal
TES Backend	A standard GA4GH TES implementation to execute the analysis

You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.

For some components, such as KeyCloak or MinIO, you may wish to use your own existing deployments.

User accessible components

A subset of the components will need to be accessible by users, outside of the TRE Layer environent:

Component	Reason
KeyCloak	So that users can authenticate and admins can manage the TRE Agent and Egress KeyCloak Realms
TRE Agent GUI	So that TRE Admins can log into and interact with the TRE Agent
TRE Agent API	So that TRE Agent GUI browser functionality, and optionally other services, can interact with the TRE Agent REST API
Egress GUI	So that Egress Officers can log into and interact with the Egress Portal
Egress API	Optional. So that other services can integrate with the Egress REST API

Sample Deployment Guide

Note that this guide is a sample. It can be modified to reflect the your own infrastructure and configuration choices.

This guide deploys all the components of the TRE Layer (except the TES Backend) on a single host (e.g. a Virtual Machine) using Docker Compose.

All the components are deployed on suitable Docker networks and can communicate with each other as required.

It also automatically configures the necessary KeyCloak realms, ready for following our other guides.

💡

Because the TES Backend is not part of this deployment, it will need to be deployed separately and configured to communicate with the required components.

We also provide guidance for installing Funnel as a TES Backend.

User accessible components

Components which may need to be accessed from outside the host are forwarded to the host on specific ports

Component	Ports
KeyCloak	HTTP: `8085`
TRE Agent GUI	HTTP: `8989`
TRE Agent API	HTTP: `8072`
Egress GUI	HTTP: `8100`
Egress API	HTTP: `8101`

In this sample, where the TES Backend is deployed elsewhere, the TES Backend will also need access to some components:

Component	Ports
MinIO	API HTTP: `9002`
TRE Agent API	HTTP: `8072`

💡

Remember the TES Backend’s environment will also need network access to project data sources, in order for analysis to run against them.

This sample also includes some additional components useful for diagnostic purposes:

Component	Notes	Ports
Seq	Logs aggregator and web-based viewer	HTTP: `5341`
Adminer	Web-based PostgreSQL client	HTTP: `8087`

Production Recommendations

To use this sample as a starting point for a production deployment, some general recommendations should be considered.

The ports specified above can be used for configuring a reverse proxy for access to the necessary components.

💡

If you are doing reverse proxy for MinIO, the Websockets Support should be enabled for MinIO GUI Proxy host.

TRE Database

A TRE database is the database which the analysis query runs against.

If a TRE database is deployed in cloud managed instances (for example, AWS RDS, Azure SQL DB Managed Instance, etc.), this 5 Safes TES implementation requires that you set the TRE Agent with the admin credentials of the TRE database, in order to create and revoke ephemeral credentials.

This can be done by setting the environment variables TRE_DATA_USER and TRE_DATA_PASSWORD with the credentials of the user, more details here .

admin credentials are the ones used when creating or setting up the TRE DB instance.
If the TRE decides to use PostgreSQL as the database, version 16 or later is required. Because in Postgres, to revoke a user/role, either superuser or admin (in PostgreSQL version 16+) privilege is required. More details here.
A user created with the CREATEROLE permission can create ephemeral credentials but cannot revoke them.
In the case where a TRE cannot not use admin credentials, the TRE database can be hosted in the same postgresql database instance, which keeps the local state of TRE Agent, and superuser credentials of the postgresql can be used in the environment variables TRE_DATA_USER and TRE_DATA_PASSWORD.

Get Started

Prerequisites

Docker and Docker Compose installed. For Linux/Ubuntu VMs, you can follow this guide
Git installed on your machine.
TES executing agent (e.g., Funnel) installed and running.
💡
Instructions on how to install Funnel can be found here

Deployment Steps

To deploy a TRE Agent and Data Egress:

Clone the repository

git clone https://github.com/SwanseaUniversityMedical/5S-TES-deployment.git

Prepare the necessary files

To deploy a TRE Agent and Data Egress, we need the following files and directories in 5S-TES-deployment repository:

DeploymentStack/TRE/docker-compose.yml
DeploymentStack/TRE/config/
ServiceStack/
.env, which needs to be configured with the appropriate environment variables based on the DeploymentStack/TRE/.env.example file.

The docker-compose.yml file in the DeploymentStack/TRE/ directory includes the services for the TRE Agent and Data Egress using relative paths to the compose manifests in the ServiceStack/ directory, where they are defined.

If you need to modify the compose manifests or the docker-compose.yml file, we recommend to move these four components to a separate directory, e.g., /srv/tre, and update the relative paths in the docker-compose.yml and CONFIG_PATH in the .env file accordingly accordingly. When in sync with the remote repository, the local 5S-TES-deployment repository will then only serve as a reference for your own deployment, rather than the source of truth.

The compose manifests used in the TRE Agent and Data Egress are:

ServiceStack/compose-manifests/applications/tre-layer.yml which defines the TRE Layer services, i.e., TRE Agent API, TRE Agent GUI.
ServiceStack/compose-manifests/applications/egress-layer.yml which defines the Data Egress Layer services, i.e., Data Egress API, Data Egress GUI.
ServiceStack/compose-manifests/shared/platform.yml which defines the platform services, i.e., PostgreSQL, Adminer, Serilog, RabbitMQ.
ServiceStack/compose-manifests/shared/credentials.yml which defines the services for ephemeral credentials, i.e., Camunda, Connectors, Vault, OpenLDAP, LDAP Init, phpLDAPadmin, Elastic Search.
ServiceStack/compose-manifests/shared/auth.yml which defines the authentication services, i.e., Keycloak. This is optional and only needed if you want to use Keycloak for authentication internally within the TRE Layer.
💡
Comment or remove the line ../../ServiceStack/compose-manifests/shared/auth.yml in the docker-compose.yml file if you don’t want to use it.
ServiceStack/compose-manifests/shared/omop-lite.yml which defines the OMOP Lite service, i.e., OMOP Lite. This is optional and only needed if you want to populate a synthetic OMOP CDM for the TRE data.
💡
Comment or remove the line ../../ServiceStack/compose-manifests/shared/omop-lite.yml in the docker-compose.yml file if you don’t want to use it.

Configure the .env file

Create a new .env file in the directory where you store the docker-compose.yml file, and configure the environment variables based on the .env.example file in the DeploymentStack/TRE/ directory or here.

The descriptions of the essential environment variables and the guide to set them are as follows:

Environment Variable	Description
`DEPLOYMENT_VERSION`	The version of the TRE Agent app you are deploying. Find the version here, between `v` and `-5S-TES-containers`. For example, `1.5.1` is the version for the release `v1.5.1-5S-TES-containers`.
`PGLOGIN` and `PGPASSWORD`	The admin credentials for the PostgreSQL database used by the TRE Agent.
`SerilogLevel`	The Serilog logging level for the TRE Agent. Default is `Information`. Change this depending on the logging level you want to set for the TRE Agent. More details here.
`EncryptionSettingsKey`	The encryption settings (in 16 bytes Base64 String format) for the TRE Agent. They must be set and can be generated using the `openssl` command: `openssl rand -base64 16` (MacOS/Linux) or `$key = New-Object byte[] 16;[System.Security.Cryptography.RandomNumberGenerator]::Fill($key);[Convert]::ToBase64String($key)` (Windows).
`TRE_DATA_SERVER` `TRE_DATA_PORT` `TRE_DATA_DATABASE` `TRE_DATA_USER` `TRE_DATA_PASSWORD`	The credentials for the PostgreSQL database used to hold the TRE data, e.g., OMOP CDM data. Camunda will use these credentials to create the ephemeral user accounts for the TES executing agent (e.g., Funnel) to access the TRE data.
`TreName`	The name identifier for this TRE instance.
`KeycloakHostName`	The hostname of the Keycloak server, for example,`localhost` or `my-keycloak`. This is only used if you are using a local Keycloak service defined in the `ServiceStack/compose-manifests/shared/auth.yml` file.
`TRE_KEYCLOAK_URL`	The URL of the TRE Layer’s Keycloak server. For example, `http://keycloak:8080` (for default local Keycloak server) or `https://tre-keycloak.net` (for a remote Keycloak server).
`SUBMISSION_KEYCLOAK_URL`	The URL of the Submission Layer’s Keycloak server. For example, `http://keycloak:8080` (for default local Keycloak server) or `https://submission-keycloak.net` (for a production Submission Layer Keycloak server).
`SubmissionAPIAddressURL`	The Submission API address URL. For example, `http://host-of-submission-layer:5034`.
`TesAPIUrl`	Where TESK or Funnel API is hosted. For example, `http://host.docker.internal:8000/v1/tasks` (when Funnel is hosted locally) or `http://my-funnel.com/tasks`.
`MINIO_TRE_UI_URL`	Where the MinIO TRE UI is hosted. For example, `http://localhost:9003` or `http://minio-tre-ui:9000`.
`MinioTreOpenidSecret`	The OpenID secret for the Minio client of Dare-TRE realm. There is a default value in the realm configuration, but you should regenerate it for production deployments. Do this by navigating to `Dare-TRE` realm -> `Clients` -> `Dare-TRE-Minio` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`TreMinioAdminUser`	The admin user for the TRE Minio server.
`TreMinioAdminPassword`	The admin password for the TRE Minio server.
`MinioRootUser`	The root user for the Submission Layer Minio server.
`MinioRootPass`	The root password for the Submission Layer Minio server.
`submissionMinioUrl`	Point to Submission Layer Minio server, used by TRE Agent. For example, `http://host-of-submission-layer:9000`.
`submissionMinioAdminConsole`	The URL for the Submission Layer’s Minio admin console. For example, `http://host-of-submission-layer:9001`.
`TreApiPublicUrl`	The public URL for the TRE Agent API. For example, `http://my-TRE-host:8072`.
`syncSchedule`	Sync Projects & Users between TRE & Submission Layer - default every 10 minutes (in minutes).
`scanSchedule`	Scan Submission Layer for available submissions - default every 1 minute (in minutes).
`EgressKeyCloakClientUIRedirectURL`	The URL for the Egress Layer’s Keycloak redirect URL. For example, `https://localhost:8100/`.
`EgressKeyCloakSecret`	The OpenID secret for the Egress Keycloak client `Data-Egress-API`. Find and regenerate this secret by navigating to `Data-Egress` realm -> `Clients` -> `Data-Egress-API` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`SubmissionAPIKeyCloakClientUIRedirectURL`	The URL for the Submission API’s Keycloak redirect URL. For example, `http://host-of-submission-layer:8989/`.
`SubmissionAPIKeyCloakSecret`	The OpenID secret for the Submission API Keycloak client `Dare-Control-API`. Find and regenerate this secret by navigating to `Dare-Control` realm -> `Clients` -> `Dare-Control-API` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`TreKeyCloakClientUIRedirectURL`	The URL for the TRE UI’s Keycloak redirect URL. For example, `http://host-of-tre:8989/`.
`TreKeyCloakSecret`	The OpenID secret for the TRE UI Keycloak client `Dare-TRE-UI`. Find and regenerate this secret by navigating to `Dare-TRE` realm -> `Clients` -> `Dare-TRE-UI` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`TreAPIKeyCloakClientUIRedirectURL`	The URL for the TRE API’s Keycloak redirect URL. For example, `http://localhost:8989/`.
`TreAPIKeyCloakSecret`	The OpenID secret for the TRE API Keycloak client `Dare-TRE-API`. Find and regenerate this secret by navigating to `Dare-TRE` realm -> `Clients` -> `Dare-TRE-API` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`BETTER_AUTH_SECRET`	Base-64 String for the secret key for the Better Auth service. Only required if you are using the NextJS Agent web app. Generated using the `openssl` command: `openssl rand -base64 32` (MacOS/Linux) or `$key = New-Object byte[] 32;[System.Security.Cryptography.RandomNumberGenerator]::Fill($key);[Convert]::ToBase64String($key)` (Windows).
`BETTER_AUTH_URL`	The URL for the Better Auth service. Typically, this is the URL of the Agent Web App. For example, `http://localhost:3000/` or `http://my-agent-web-app.com/`. Only required if you are using the NextJS Agent web app.
`NEXT_PUBLIC_KEYCLOAK_URL`	The URL of the Keycloak Client portal. For example, `http://localhost:8085/`. Only required if you are using the NextJS Agent web app.
`CONFIG_PATH`	The path for the to the shared configuration files (e.g., realm config, ldap, vault, init scripts) for the TRE Layer. Default is `../../../DeploymentStack/TRE/config`.

Run docker compose

At the directory where docker-compose.yml and .env files are located, run the following command to start the TRE Agent and Data Egress:

docker compose up -d

Check-in

After the containers are up and running (check with docker ps), if you have configured the ports for the TRE Agent components for user accessibility, you can access the TRE Agent UI and Egress UI by navigating to http://<hostname>:8989 and http://<hostname>:8100, respectively, in your browser.

If you are using the default Keycloak service defined in the ServiceStack/compose-manifests/shared/auth.yml file, you can access the Keycloak Client portal by navigating to port 8085 of the host machine, and Then, the default to access KeyCloak admin portal are username admin and admin; the default username and password for the TRE Agent and Egress are globaladminuser and password123.

💡

If you encounter the error “Invalid parameter: redirect_uri” from Keycloak when trying to login into TRE UI or Egress UI, please make sure the redirectURL environment variables in the .env file, the URL you put on the browser’s address bar, and the Valid redirect URIs registered in Keycloak’s client settings are the same. For more details, please refer to the troubleshooting guide.

Connect TRE Agent to Submission Layer and Egress

TRE and Submission Layer connection

The TRE Agent needs user credentials from the Submission Layer Keycloak to connect to the Submission Layer. This user should be created in the Dare-Control realm with the dare-tre-admin role and group assigned to it.

If you are hosting your own Submission Layer create the user in the Submission Layer Keycloak. Otherwise, the TRE Admin user’s credentials are typically provided by the Submission Layer admin.

Given these credentials, put them in the page Update Submission Credentials in the TRE Agent UI.

TRE and Egress connection

At the page Update Data Egress Credentials in the TRE Agent UI (port 8989 of the host machine), put the credentials of a user with the dare-tre-admin role in the Dare-Egress Keycloak realm.

If you are using the default local Keycloak service defined in the ServiceStack/compose-manifests/shared/auth.yml file, the user already exists and its credentials are:

Username: accessfromtretoegress
Password: password123

At the page Update TRE Credentials in the Egress UI (port 8100 of the host machine), put the credentials of a user with the dare-egress-admin role in the Dare-TRE Keycloak realm.

If you are using the default local Keycloak service defined in the ServiceStack/compose-manifests/shared/auth.yml file, the user already exists and its credentials are:

Username: accessfromegresstotre
Password: password123

Update DMN rules

At the page DMN Rules in the TRE Agent UI (port 8989 of the host machine), update and deploy the DMN rules to the correct values of the TRE DB credentials.

Output:tag	Output:value
postgresDatabase	”change to the TREDBDatabase above”
postgresServer	”change to the TREDBHost above”
postgresPort	”change to the TREDBPort above”
postgresSchema	”change to the schema of the OMOP CDM in TRE DB”

More details can be found in the guide.

Approve projects and memberships

Once a Submission layer Admin has created a Project, assigned Users to the Project and added your TRE to the Project, you need to approve the Project and User Memberships from the TRE Agent UI (port 8989 of the host machine), at the page Projects.

More details can be found in the guide.

Recommended next steps

Make sure TRE’s containers, database and TES executing agent are running and accessible, then test the system by sending the Hello world submission or submission with basic analysis task from the Submission layer side.
Add a new user in the Dare-TRE Keycloak realm with appropriate permissions, then use this user credentials to access the TRE Agent UI.
Add a new user in the Data-Egress Keycloak realm with appropriate permissions, then use this user credentials to access the Egress UI.
Update passwords or secrets for the TRE Agent and Egress, including the Keycloak client secrets, Minio server credentials, etc.

Debugging

If you encounter any issues, you can check the logs of the containers by running the following command:

docker logs <container-name> -f

Alternatively, you can check the logs of the stack by accessing port 5341 (Seq service) of the host machine.

Requirements Set TES API