Funnel TES

Funnel is an implementation of the GA4GH Task Execution Schemas. It provides an API server, multiple storage backends, multiple compute backends and a web UI. For the purposes of the DemoStack quickstart we will be configuring the TRE Agent to use Funnel as the TES compatible execution backend and any outputs from the execution will be first stored in the TRE Layer S3 instance and then copied over to the Submission Layer S3 instance.

Set up S3 Access key

💡

This guidance refers to the preset credentials for the DemoStack quickstart. If you are setting up Funnel for a regular deployment, you’ll need to use your TRE Agent realm credentials.

Navigate to the S3 instance of the TRE Layer. If you are using Demostack Quickstart, it will be on the same host as the TRE Agent, at port 9002. If you’ve configured it to be different, you’ll need to use the correct port.
Login with S3 TRE root credentials (which are set up in the environment variables in the .env file).

If you are using Demostack Quickstart, the S3 TRE root credentials are:

Username: s3-tre
Password: s3-tre-pass

To create the access keys, in the S3 (currently RustFS) console navigate to Access Keys -> Add Access Key

💡

While you can log in with your KeyCloak credentials, using the option Log in with KeyCloak, to create access keys with efficient privileges, you need to log in with S3 root credentials.

RustFS

Install Funnel

Instructions on how to install Funnel can be found here

Funnel is currently only available on linux and macOS.

This installs the latest release candidate at the time of writing v0.11.12.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/funnel/develop/install.sh)" -- "v0.11.12"

You can add the destination of Funnel by adding it in the end of the command. For example, to install Funnel to /opt/funnel:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/calypr/funnel/develop/install.sh)" -- "v0.11.12" "/opt/funnel"

The latest version available via brew is v0.11.3.

brew tap ohsu-comp-bio/formula
 
brew install funnel

This installs the most recent Funnel release at the time of writing v0.11.6.

For Intel (AMD64) architecture:

wget https://github.com/ohsu-comp-bio/funnel/releases/download/v0.11.6/funnel-darwin-amd64-v0.11.6.tar.gz
tar -xzvf funnel-darwin-amd64-v0.11.6.tar.gz
sudo mv funnel /usr/local/bin/

For Apple Silicon (ARM64) architecture:

wget https://github.com/ohsu-comp-bio/funnel/releases/download/v0.11.6/funnel-darwin-arm64-v0.11.6.tar.gz
tar -xzvf funnel-darwin-arm64-v0.11.6.tar.gz
sudo mv funnel /usr/local/bin/

In order to build the latest development code, run: bash copy git clone https://github.com/ohsu-comp-bio/funnel.git cd funnel make This requires Go 1.21+ to be installed.

💡

For a production deployment, some general recommendations should be considered.

The ports Funnel runs on can be configured in the next step, if needed for configuring a reverse proxy.

Funnel’s own deployment documentation also offers suggested approaches to deployment.

Check if funnel has been successfully installed

funnel --help

Funnel config file

Funnel will need access to the TRE layer S3 storage. Once funnel is installed, create a config.yml file.

Endpoint is where the instance of S3 in the TRE Layer is hosted.
Access key & Secret are created in the previous step.

config.yml

GenericS3:
  - Disabled: false
    Endpoint: "localhost:9002" # URL to S3 API. Change to match your environment. `9002` is the default port in our deployment samples.
    Key: "<access key>"
    Secret: "<secret>"

You may also wish to configure other elements of the Funnel installation, e.g.:

config.yml

Server:
  HTTPPort: 8080 # Control which port Funnel binds on
Worker:
  WorkDir: "/var/opt/funnel" # explicitly specify Funnel's working directory
# ...

Funnel’s documentation offers configuration details in the relevant pages for each feature area.

Run Funnel

Run the funnel server using the config created above.

funnel server run -c config.yml

You can decide the location of the config file, but the command above will need to be changed to point to the correct config file location. For example, if the config file is located in /etc/funnel/: bash copy funnel server run -c /etc/funnel/config.yml

Optionally, to run Funnel in the background on a typical linux server, you may wish to setup a systemd service.

For example, if Funnel is installed manually to /opt/funnel/funnel:

funnel.service

[Unit]
Description=Funnel Server

[Service]
ExecStart=/opt/funnel/funnel server run --config /etc/funnel/config.yml
SyslogIdentifier=funnel
LimitNOFILE=1000000
RestartSec=10
Restart=always

[Install]
WantedBy=multi-user.target

Access to Funnel

For DemoStack quickstart, by default, Funnel Web Dashboard can be accessed at localhost:8000.

⚠️

For regular deployments, it is strongly recommended NOT to open public internet access to Funnel Dashboard and to limit the Funnel network access to specific IP addresses, e.g., IP of developer’s machine for debugging purposes, only.

Quickstart Deploying to Production