Deploy a TRE Agent and Data Egress

This page contains:

a summary of what is involved when deploying a Five Safes TES TRE Layer
a Sample Deployment Guide
- with recommendations for deploying in Production

Component Summary

Here is a summary of the required components which make up the TRE Layer:

Component	Notes
KeyCloak	The TRE Agent and Egress apps each require a Keycloak realm containing users and clients authorised to access them
Minio	An Amazon S3 compatible TRE Storage service
RabbitMQ	A message broker used for queueing
PostgreSQL	A database for the TRE Agent and Egress apps to keep local state
Hashicorp Vault	Used for accessing ephemeral credentials as secrets
OpenLDAP	Used to provide ephemeral credentials for Trino (if in use as a datasource)
Camunda Connectors	A REST API allowing other services (e.g. TRE Agent) to integrate with Camunda
Camunda Orchestration	A consolidated service that combines Camunda functionality via Zeebe + Operate + Tasklist
ElasticSearch	Used for Camunda’s local datastore such as workflow state
TRE-Camunda	A service defining the handlers for ephemeral credentials management. Registers the handlers with Camunda, via Zeebe
TRE Agent API	A REST API for TRE Agent functionality
TRE Agent GUI	A Web Frontend for TRE Admins to log into and interact with the TRE Agent
Egress API	A REST API for Egress Portal functionality
Egress GUI	A Web Frontend for Egress Officers to log into and interact with the Egress Portal
TES Backend	A standard GA4GH TES implementation to execute the analysis

You may choose how to distribute your own deployment of these components, as long as they are able to communicate with each other over the network.

For some components, such as KeyCloak or MinIO, you may wish to use your own existing deployments.

User accessible components

A subset of the components will need to be accessible by users, outside of the TRE Layer environent:

Component	Reason
KeyCloak	So that users can authenticate and admins can manage the TRE Agent and Egress KeyCloak Realms
TRE Agent GUI	So that TRE Admins can log into and interact with the TRE Agent
TRE Agent API	So that TRE Agent GUI browser functionality, and optionally other services, can interact with the TRE Agent REST API
Egress GUI	So that Egress Officers can log into and interact with the Egress Portal
Egress API	Optional. So that other services can integrate with the Egress REST API

Sample Deployment Guide

Note that this guide is a sample. It can be modified to reflect the your own infrastructure and configuration choices.

This guide deploys all the components of the TRE Layer (except the TES Backend) on a single host (e.g. a Virtual Machine) using Docker Compose.

All the components are deployed on suitable Docker networks and can communicate with each other as required.

It also automatically configures the necessary KeyCloak realms, ready for following our other guides.

💡

Because the TES Backend is not part of this deployment, it will need to be deployed separately and configured to communicate with the required components.

We also provide guidance for installing Funnel as a TES Backend.

User accessible components

Components which may need to be accessed from outside the host are forwarded to the host on specific ports

Component	Ports
KeyCloak	HTTP: `8085`
TRE Agent GUI	HTTP: `8989`
TRE Agent API	HTTP: `8072`
Egress GUI	HTTP: `8100`
Egress API	HTTP: `8101`

In this sample, where the TES Backend is deployed elsewhere, the TES Backend will also need access to some components:

Component	Ports
MinIO	API HTTP: `9002`
TRE Agent API	HTTP: `8072`

💡

Remember the TES Backend’s environment will also need network access to project data sources, in order for analysis to run against them.

This sample also includes some additional components useful for diagnostic purposes:

Component	Notes	Ports
Seq	Logs aggregator and web-based viewer	HTTP: `5341`
Adminer	Web-based PostgreSQL client	HTTP: `8087`

Production Recommendations

To use this sample as a starting point for a production deployment, some general recommendations should be considered.

The ports specified above can be used for configuring a reverse proxy for access to the necessary components.

💡

If you are doing reverse proxy for MinIO, the Websockets Support should be enabled for MinIO GUI Proxy host.

Get Started

Prerequisites

Docker and Docker Compose installed. For Linux/Ubuntu VMs, you can follow this guide
Git installed on your machine.

Deployment Steps

To deploy a TRE Agent and Data Egress:

Clone the repository

git clone https://github.com/SwanseaUniversityMedical/5S-TES-deployment.git

Go to the TRE Layer directory

Within the 5S-TES-deployment directory, the TRE Layer docker-compose and configuration files are in the DeploymentStack/TRE directory:

cd DeploymentStack/TRE

Configure the .env file

Open the .env file in the TRE directory and configure the environment variables.

The descriptions of the environment variables and the guide to set them are as follows:

Environment Variable	Description
`dareVer`	The version of the TRE Agent app you are deploying. Find the version here, between `v` and `-5S-TES-containers`. For example, `1.5.1` is the version for the release `v1.5.1-5S-TES-containers`.
`PGLOGIN` and `PGPASSWORD`	The admin credentials for the PostgreSQL database used by the TRE Agent.
`EncryptionSettingsKey` and `EncryptionSettingsBase`	The encryption settings (in 16 bytes Base64 String format) for the TRE Agent. They must be set and can be generated using the `openssl` command: `openssl rand -base64 16` (MacOS/Linux) or `$key = New-Object byte[] 16;[System.Security.Cryptography.RandomNumberGenerator]::Fill($key);[Convert]::ToBase64String($key)` (Windows).
`TRE_DATA_SERVER` `TRE_DATA_PORT` `TRE_DATA_DATABASE` `TRE_DATA_USER` `TRE_DATA_PASSWORD`	The credentials for the PostgreSQL database used to hold the TRE data, e.g., OMOP CDM data. Camunda will use these credentials to create the ephemeral user accounts for the TES executing agent (e.g., Funnel) to access the TRE data.
`DemoMode`	Set to `true` if you’d like to simulate execution, otherwise default to `false`.
`TreName`	The name identifier for this TRE instance.
`KeyCloakDemoMode`	Allows Keycloak to not require https. Default is `true`.
`EphemeralCredentials`	Enable ephemeral credentials functionality. Set to `true` to enable.
`KeycloakHostName`	The hostname of the Keycloak server. For example, `http://localhost:8085` or `https://my-keycloak.net`.
`UseTESK`	Set to `true` to execute with a TES implementation.
`UseRabbit`	Set to `true` to use RabbitMQ message broker, otherwise set to `false`.
`TesAPIUrl`	Where TESK or Funnel API is hosted. For example, `http://host.docker.internal:8000/v1/tasks` (when Funnel is hosted locally) or `http://my-funnel.com/tasks`.
`TesOutputBucketPrefix`	Output bucket prefix for the TES executing agent to write results to. For example, `s3://`.
`MinioTreIdentityID`	Name of the TRE’s Minio client, i.e., `Dare-TRE-Minio`.
`MinioTreOpenidSecret`	The OpenID secret for the Minio client of Dare-TRE realm. There is a default value in the realm configuration, but you should regenerate it for production deployments. Do this by navigating to `Dare-TRE` realm -> `Clients` -> `Dare-TRE-Minio` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`MinioTreIdentityConfigURL`	The OpenID configuration URL for the Dare-TRE realm. For example, `http://keycloak:8080/realms/Dare-TRE/.well-known/openid-configuration` (if Keycloak is running on the same Docker network) or `https://my-keycloak.net/realms/Dare-TRE/.well-known/openid-configuration`.
`MinioRootUser`	The root user for the Submission Layer Minio server.
`MinioRootPass`	The root password for the Submission Layer Minio server.
`TreMinioAdminUser`	The admin user for the TRE Minio server.
`TreMinioAdminPassword`	The admin password for the TRE Minio server.
`MinioBrowserHost`	This is useful if you are using a reverse proxy to access the Minio server. Read more about this here.
`submissionMinioUrl`	Point to Submission Layer Minio server, used by TRE Agent. For example, `http://host-of-submission-layer:9000`.
`submissionMinioAdminConsole`	The URL for the Submission Layer’s Minio admin console. For example, `http://host-of-submission-layer:9001`.
`TreUiPublicUrl`	The public URL for the TRE Agent GUI. For example, `http://my-TRE-host:8989`.
`TreApiPublicUrl`	The public URL for the TRE Agent API. For example, `http://my-TRE-host:8072`.
`CAMUNDA_VERSION`	The version of Camunda to deploy.
`CAMUNDA_BUNDLE_VERSION`	The version of Camunda Bundle to deploy.
`ELASTIC_VERSION`	The version of Elasticsearch to deploy for Camunda’s local datastore.
`CredentialAPISettingsStartWebhookUrl`	The webhook URL for starting credentials via Camunda connectors. For example, `http://connectors:8080/inbound/StartCredentials`.
`CredentialAPISettingsRevokeWebhookUrl`	The webhook URL for revoking credentials via Camunda connectors. For example, `http://connectors:8080/inbound/RevokeCredentials`.
`syncSchedule`	Sync Projects & Users between TRE & Submission Layer - default every 10 minutes (in minutes).
`scanSchedule`	Scan Submission Layer for available submissions - default every 1 minute (in minutes).
`EnableExternalHangfire`	Set to `true` to enable external Hangfire, otherwise set to `false`.
`EgressKeyCloakUseRedirect`	If this is set to `true`, the Egress Layer will use redirects for Keycloak authentication. Default is `false`.
`EgressKeyCloakBaseRealmAddress`	This is the realm address for the Egress’s Keycloak server. For example, `http://keycloak:8080/realms/Data-Egress` or `https://my-keycloak.net/realms/Data-Egress`.
`EgressKeyCloakAuthority`	The OpenID authority URL for the Egress Keycloak realm. For example, `http://keycloak:8080/realms/Data-Egress/.well-known/openid-configuration`.
`EgressKeyCloakMetadataAddress`	The OpenID metadata address URL for the Egress Keycloak realm. For example, `http://keycloak:8080/realms/Data-Egress/.well-known/openid-configuration`.
`EgressValidAudiences`	The valid audiences for the Egress Keycloak token. Default is `Data-Egress-UI,Data-Egress-API`.
`EgressKeyCloakClientUIRedirectURL`	The URL for the Egress Layer’s Keycloak redirect URL. For example, `https://localhost:8100/`.
`EgressKeyCloakTokenExpiredAddressUI`	The URL for the Egress Layer’s Keycloak token expired address. For example, `http://localhost:8100/Account/LoginAfterTokenExpired`.
`EgressKeyCloakSecret`	The OpenID secret for the Egress Keycloak client `Data-Egress-API`. Find and regenerate this secret by navigating to `Data-Egress` realm -> `Clients` -> `Data-Egress-API` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`EgressKeyCloakClientID`	The client ID for the Egress Keycloak client, i.e., `Data-Egress-API`.
`SubmissionAPIAddressURL`	The Submission API address URL. For example, `http://host-of-submission-layer:5034`.
`SubmissionAPIKeyCloakUseRedirect`	If this is set to `true`, the Submission API will use redirects for Keycloak authentication. Default is `false`.
`SubmissionAPIKeyCloakClientUIRedirectURL`	The URL for the Submission API’s Keycloak redirect URL. For example, `http://host-of-submission-layer:8989/`.
`SubmissionAPIKeyCloakClientId`	The client ID for the Submission API Keycloak client, i.e., `Dare-Control-API`.
`SubmissionAPIKeyCloakBaseRealmAddress`	This is the realm address for the Submission’s Keycloak server. For example, `http://keycloak:8080/realms/Dare-Control` (if Keycloak is running on the same Docker network) or `https://my-keycloak.net/realms/Dare-Control`.
`SubmissionAPIKeyCloakAuthority`	The OpenID authority URL for the Submission Keycloak realm. For example, `http://keycloak:8080/realms/Dare-Control/.well-known/openid-configuration` or `https://my-keycloak.net/realms/Dare-Control/.well-known/openid-configuration`.
`SubmissionAPIKeyCloakMetadataAddress`	The OpenID metadata address URL for the Submission Keycloak realm. Same as `SubmissionAPIKeyCloakAuthority`.
`SubmissionAPIValidAudiences`	The valid audiences for the Submission API Keycloak token. Default is `Dare-Control-UI,Dare-Control-API,Dare-Control-Minio`.
`SubmissionAPIKeyCloakTokenExpiredAddressUI`	The URL for the Submission API’s Keycloak token expired address. For example, `http://host-of-submission-layer:8989/Account/LoginAfterTokenExpired`.
`SubmissionAPIKeyCloakSecret`	The OpenID secret for the Submission API Keycloak client `Dare-Control-API`. Find and regenerate this secret by navigating to `Dare-Control` realm -> `Clients` -> `Dare-Control-API` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`TreKeyCloakUseRedirect`	If this is set to `true`, the TRE UI will use redirects for Keycloak authentication. Default is `false`.
`TreKeyCloakClientUIRedirectURL`	The URL for the TRE UI’s Keycloak redirect URL. For example, `http://host-of-tre:8989/`.
`TreKeyCloakTokenExpiredAddressUI`	The URL for the TRE UI’s Keycloak token expired address. For example, `http://host-of-tre:8989/Account/LoginAfterTokenExpired`.
`TreKeyCloakSecret`	The OpenID secret for the TRE UI Keycloak client `Dare-TRE-UI`. Find and regenerate this secret by navigating to `Dare-TRE` realm -> `Clients` -> `Dare-TRE-UI` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`TreKeyCloakBaseRealmAddress`	This is the realm address for the TRE’s Keycloak server. For example, `http://keycloak:8080/realms/Dare-TRE` or `https://my-keycloak.net/realms/Dare-TRE`.
`TreKeyCloakAuthority`	The OpenID authority URL for the TRE Keycloak realm. For example, `http://keycloak:8080/realms/Dare-TRE/.well-known/openid-configuration` or `https://my-keycloak.net/realms/Dare-TRE/.well-known/openid-configuration`.
`TreKeyCloakClientId`	The client ID for the TRE UI Keycloak client, i.e., `Dare-TRE-UI`.
`TreKeyCloakMetadataAddress`	The OpenID metadata address URL for the TRE Keycloak realm. Same as `TreKeyCloakAuthority`.
`TreAccountManagementURLUI`	The URL for the TRE UI’s Keycloak account management. For example, `http://localhost:8085/realms/Dare-TRE/account`.
`TreValidAudiences`	The valid audiences for the TRE Keycloak token. Default is `Dare-TRE-API,Dare-TRE-UI`.
`TreAPIKeyCloakUseRedirect`	If this is set to `true`, the TRE API will use redirects for Keycloak authentication. Default is `false`.
`TreAPIKeyCloakClientUIRedirectURL`	The URL for the TRE API’s Keycloak redirect URL. For example, `http://localhost:8989/`.
`TreAPIKeyCloakTokenExpiredAddressUI`	The URL for the TRE API’s Keycloak token expired address. For example, `http://localhost:8989/Account/LoginAfterTokenExpired`.
`TreAPIKeyCloakSecret`	The OpenID secret for the TRE API Keycloak client `Dare-TRE-API`. Find and regenerate this secret by navigating to `Dare-TRE` realm -> `Clients` -> `Dare-TRE-API` -> `Credentials`. Then click `Regenerate` and copy the new value into this environment variable.
`TreAPIKeyCloakBaseRealmAddress`	This is the realm address for the TRE API’s Keycloak server. For example, `http://keycloak:8080/realms/Dare-TRE`.
`TreAPIKeyCloakAuthority`	The OpenID authority URL for the TRE API Keycloak realm. For example, `http://keycloak:8080/realms/Dare-TRE/.well-known/openid-configuration`.
`TreAPIKeyCloakClientId`	The client ID for the TRE API Keycloak client, i.e., `Dare-TRE-API`.
`TreAPIKeyCloakMetadataAddress`	The OpenID metadata address URL for the TRE API Keycloak realm. For example, `http://keycloak:8080/realms/Dare-TRE/.well-known/openid-configuration`.
`TreAPIAccountManagementURLUI`	The URL for the TRE API’s Keycloak account management. For example, `http://localhost:8085/realms/Dare-TRE/account`.
`TreAPIValidAudiences`	The valid audiences for the TRE API Keycloak token. Default is `Dare-TRE-API,Dare-TRE-UI`.
`CONFIG_PATH`	The path for the to the shared configuration files (e.g., realm config, ldap, vault, init scripts) for the TRE Layer. Default is `../../../DeploymentStack/TRE/config`.

💡

Instructions on how to set up a Funnel TES API executing agent can be found here

You can find an example .env file here.

Run docker compose

docker compose up -d

Check-in

After the containers are running, if you have configured the ports for the TRE Agent components for user accessibility, you can access the TRE Agent UI and Egress UI by navigating to http://<hostname>:8989 and http://<hostname>:8100, respectively, in your browser.

Next steps

Add a new user in the Dare-TRE Keycloak realm with appropriate permissions, then use this user credentials to access the TRE Agent UI.
Add a new user in the Data-Egress Keycloak realm with appropriate permissions, then use this user credentials to access the Egress UI.
Connect the TRE Agent to the Submission Layer by following this guide.
Connect the TRE Agent to Egress by following this guide.

Requirements Set TES API