All In One StackRun with Funnel TES

Funnel TES

Funnel is an implementation of the GA4GH Task Execution Schemas. It provides an API server, multiple storage backends, multiple compute backends and a web UI. For the purposes of the All In One stack we will be configuring the TRE Agent to use Funnel as the TES compatible execution backend and any outputs from the execution will be stored in the TRE Layer MinIO instance.

Set up MinIO Access key

Navigate to the TRE layer MinIO instance, hosted at http://localhost:9002 and create access key credentials. To login to MinIO use:

Username: minio
Password: minio123

To create the access keys, in the MinIO console navigate to Access Keys -> Create access key

MinIO

💡

In the Object Browser note the name of the TRE output bucket of the Project that you want to submit to. The naming pattern is: <projectName> + <uniqueID> + outputtre. You will need to reference this in the outputs section in the TES message

Install Funnel

Instructions on how to install Funnel can be found here

💡
It is currently only available on linux and macOS.

Funnel config file

Funnel will need access to the TRE layer MinIO. Create a yaml file, point it to the instance of MinIO in the TRE Layer and set the access key & secret created in the previous step.

config.yml
GenericS3:
  - Disabled: false
    Endpoint: "localhost:9002" // URL where the MinIO instance is hosted.
    Key: “<access key>”
    Secret: “<secret>"

Run Funnel

Run the funnel server using the config created above.

funnel server run -c config.yml

Set Funnel as the TES API for All In One

In the All In One repository change these settings in the .env file:

UseTESK = true

In the docker-compose.yml set:

// Set all instances of DemoMode and KeycloakDemoMode in the file to:
DemoMode = false
KeyCloakDemoMode = true
 
// In the tre-api section set:
TESKAPIURL = "http://localhost:8000/v1/tasks" or "http://host.docker.internal:8000/v1/tasks" // point to the funnel server TASK API, if trying to access from inside docker you need to use host.docker.internal
TESKOutputBucketPrefix = "s3://" // set the prefix for GenericS3 storage

Submit a TES message

Once the Funnel is running and the All In One is up and configured, you can submit a TES payload to the Submission Layer.

Get Access Token

From the Submission Layer navigate to API Access Token -> Copy to Clipboard

Authorise request

You can make the request from the swagger API hosted at http://localhost:5034/swagger. Authorise the request with the Token from the Authorise button on the top right.

Make submission

More information on the TES payload can be found at the TES API specification website.

 
POST localhost:5034/v1/tasks
{
    "state": 0,
    "name": "Hello world",
    "inputs": [
        {
            "url": "s3://<TREOutputBucket>/hello.txt",
            "path": "/inputs/hello.txt"
        }
    ],
    "outputs": [
        {
            "url": "s3://<TREOutputBucket>/output.txt", // TREOutputBucket needs to match the bucket name from Step 1
            "path": "/outputs/stdout"
        }
    ],
    "executors": [
        {
            "image": "alpine",
            "command": [
                "cat",
                "/inputs/hello.txt"
            ],
            "stdout": "/outputs/stdout"
        }
    ],
    "volumes": null,
    "tags": {
        "project": "<ProjectName>", // Project that you would like to submit to.
        "tres": "<TREName>" // TRE related to that Project
    },
    "logs": null,
    "creation_time": null
}
Getting StartedStack Components