Five Safes TESReference ImplementationTask Execution ServiceInstall Funnel TES

Funnel TES

Funnel is an implementation of the GA4GH Task Execution Schemas. It provides an API server, multiple storage backends, multiple compute backends and a web UI. For the purposes of the All In One stack we will be configuring the TRE Agent to use Funnel as the TES compatible execution backend and any outputs from the execution will be stored in the TRE Layer MinIO instance.

Set up MinIO Access key

💡

This guidance refers to the preset credentials for the All In One quickstart. If you are setting up Funnel for a regular deployment, you’ll need to use your TRE Agent realm credentials.

Navigate to the MinIO instance of the TRE Layer (hosted at: localhost:9002). If you see the option to login with SSO_Identifier, use the following credentials:

Username: globaladminuser
Password: password123

You can also login to Minio with root credentials using the option Use Credentials under Other Authentication methods:

Username: minio
Password: minio123

To create the access keys, in the MinIO console navigate to Access Keys -> Create access key

MinIO

Install Funnel

Instructions on how to install Funnel can be found here

Funnel is currently only available on linux and macOS.

This installs the latest release candidate at the time of writing v0.11.7-rc.12.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/ohsu-comp-bio/funnel/refs/heads/develop/install.sh)" -- v0.11.7-rc.12
💡

For a production deployment, some general recommendations should be considered.

The ports Funnel runs on can be configured in the next step, if needed for configuring a reverse proxy.

Funnel’s own deployment documentation also offers suggested approaches to deployment.

Check if funnel has been successfully installed

funnel --help

Funnel config file

Funnel will need access to the TRE layer MinIO. Once funnel is installed, create a config.yml file.

  • Endpoint is where the instance of MinIO in the TRE Layer is hosted.
  • Access key & Secret are created in the previous step.
config.yml
GenericS3:
  - Disabled: false
    Endpoint: "localhost:9002" # URL to MinIO. Change to match your environment. `9002` is the default port in our deployment samples.
    Key: "<access key>"
    Secret: "<secret>"

You may also wish to configure other elements of the Funnel installation, e.g.:

config.yml
Server:
  HTTPPort: 8080 # Control which port Funnel binds on
Worker:
  WorkDir: "/var/opt/funnel" # explicitly specify Funnel's working directory
# ...

Funnel’s documentation offers configuration details in the relevant pages for each feature area.

Run Funnel

Run the funnel server using the config created above.

funnel server run -c config.yml

You can decide the location of the config file, but the command above will need to be changed to point to the correct config file location. For example, if the config file is located in /etc/funnel/:

funnel server run -c /etc/funnel/config.yml

Optionally, to run Funnel in the background on a typical linux server, you may wish to setup a systemd service.

For example, if Funnel is installed manually to /opt/funnel/funnel:

funnel.service
[Unit]
Description=Funnel Server

[Service]
ExecStart=/opt/funnel/funnel server run --config /etc/funnel/config.yml
SyslogIdentifier=funnel
LimitNOFILE=1000000
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target
QuickstartDeploying to Production