=========================================== Deploy a Command-Line Tool on Seven Bridges =========================================== Introduction ============ Seven Bridges (SB) offers a platform for building and running a wide range of tools and workflows. While the `official documentation `_ provides a tutorial on creating tools, this case study goes further by focusing on a real-world example specific to the AIMPF. We will create a tool that wraps Python code into a command-line tool. This tool will take experiment names as input, query the db38 database, retrieve records, and generate a directory containing individual Excel files for each experiment. To build this tool, we need a Command-Line Interface (CLI) for the Python code. If you are unfamiliar with CLI, you can learn more in the following tutorial: :doc:`build-cli-for-python`. Once the CLI is ready, we can proceed to build the tool on SB. Build the Tool ============== Create the Tool --------------- To create a tool on SB, follow these steps (adapted from the `official documentation `_): 1. Select your division ``AIMPF`` and navigate to your project ``sandbox-chen`` (or your own project). 2. Click the ``Apps`` tab. 3. Click ``Create app`` in the top-right corner to open the app creation dialog. 4. Select ``Command Line Tool``. 5. In the ``Name`` field, enter ``db38pulling`` for your tool. 6. Select the CWL Version to use. The latest recommended CWL version is selected by default. 7. Click ``Create``. You will be taken to the tool editor to start entering the tool information in the sections described below. Register the Docker Image ------------------------- .. image:: images/docker-image.png :align: center :width: 90% .. raw:: html
In the ``DOCKER IMAGE`` section, register the Docker image containing the Python code for the tool: ``images.sbgenomics.com/contextualize/db38pulling:0.1.1``. For more details about Docker image formats, refer to the `official documentation `_. Set the Arguments and Input Ports --------------------------------- .. image:: images/base-command.png :align: center :width: 90% .. raw:: html
The ``BASE COMMAND`` is not required for this tool, so you can leave it empty. .. image:: images/arguments.png :align: center :width: 90% .. raw:: html
In the ``ARGUMENTS`` section, define two arguments: ``SBGSECRET_USER`` for the username and ``SBGSECRET_PASS`` for the password. These are used to authenticate the user ``Contextualize`` to query the db38 database. .. image:: images/arguments-secret-user.png :align: center :width: 40% .. raw:: html
For ``SBGSECRET_USER``, enable ``Use command line binding``, set ``Prefix`` to ``--username``, and set ``Value`` to ``SBGSECRET_USER``. Enable ``Separate value and prefix`` and set ``Position`` to ``0`` (optional in this case). Apply similar settings for ``SBGSECRET_PASS``. .. image:: images/input-ports.png :align: center :width: 90% .. raw:: html
In the ``INPUT PORTS`` section, define the input ports for the tool: ``USER``, ``PASS``, ``experiment_list``, ``experiment_file``, ``master_folder``, and ``motion_toggle``. All are of type ``string`` except for ``experiment_file``, which is a ``file`` type. .. image:: images/output-ports.png :align: center :width: 90% .. raw:: html
In the ``OUTPUT PORTS`` section, define the output ports for the tool. The output port ``output_folder`` is of type ``directory``. The output folder name will be determined by the ``master_folder`` input port, with the Glob value set to ``$(inputs.master_folder)``. .. image:: images/environmental-variables.png :align: center :width: 90% .. raw:: html
In the ``ENVIRONMENT VARIABLES`` section, set the environmental variables ``SBGSECRET_USER`` and ``SBGSECRET_PASS`` to authenticate the user for querying the db38 database. .. image:: images/command.png :align: center :width: 90% .. raw:: html
After setting up the tool, the system will automatically generate a command, visible in the ``COMMAND`` section. Note that secrets must be specified in the ``ARGUMENTS``, ``INPUT PORTS``, and ``ENVIRONMENT VARIABLES`` sections. Update the Tool --------------- You can update the tool by following these methods: - Click the ``Apps`` tab, find ``db38pulling``, click the ``...`` next to the tool name, and select ``Edit``. - Click the ``Apps`` tab, find ``db38pulling``, click its name, then click the ``Edit`` button in the top-right corner. - If there is any run history in the dashboard, click the event, then click the app name on the left side of the screen. On the next page, click the ``Edit`` button in the top-right corner. Updating the tool is similar to creating it. You can update the Docker image, arguments, input/output ports, environmental variables, and command. After updating, click the ``Save`` button in the top-right corner to save changes. .. Tip:: After updating, the revision number will increment by 1. When using the tool later, you can select the appropriate revision. Additionally, adjust the secret whitelist to grant the new revision access to secrets. Use the Tool ============ Configure the Secret -------------------- To securely store and use sensitive information like usernames and passwords in the tool, follow these steps: 1. In the top menu, click ``Developer`` and select ``Secrets management``. .. image:: images/add-new-secret.png :align: center :width: 40% .. raw:: html
2. Click ``Add new secret`` and enter the secret name, value, and a short description. Secrets can be shared within the same division. Once done, click ``Add secret``. 3. To edit an existing secret, click the pencil icon next to the secret name. To delete a secret, click ``...`` and select ``Delete secret``. .. image:: images/secret-allow-app.png :align: center :width: 90% .. raw:: html
4. To allow apps to access the secret, add it to the secret whitelist. Click ``...`` next to the secret name, select ``Allow app``, enter the ``App ID`` and ``App revision`` in the pop-up window, and click ``Add``. .. image:: images/appid-revision.png :align: center :width: 90% .. raw:: html
.. tip:: The ``App ID`` and ``App revision`` can be found on the tool's main information page, typically formatted as ``///``. .. note:: Secrets are shared across the division, so you don't need to create the same secret multiple times. Secrets can also be shared with others, configurable in the secret settings. Run the Tool ------------ To run the tool, follow these steps: 1. Select your division ``AIMPF`` and navigate to your project ``sandbox-chen`` (or your own project). 2. Click the ``Apps`` tab. 3. Find ``db38pulling`` and click ``Run`` to open the ``Tasks`` tab for configuration. .. image:: images/run-app-settings.png :align: center :width: 90% .. raw:: html
4. In the ``Task Input`` tab, you will see three columns: ``Inputs``, ``App Settings``, and ``Output Settings``. Use the input file selector for the ``experiment_file`` input port. ``USER`` and ``PASS`` are dropdown lists, while text fields are provided for ``experiment_list``, ``master_folder``, and ``motion_toggle`` input ports. Fill in the required values. .. note:: Leave ``experiment_list`` empty, as the CLI enforces a mutual-exclusivity rule between ``experiment_list`` and ``experiment_file``, allowing you to submit either a comma-separated string or a text file listing experiment names. .. tip:: The file selected in the ``experiment_file`` input port must be uploaded to the platform first. Upload the file in the ``Files`` tab via ``Add files``, then select it in the ``experiment_file`` input port. Currently, the platform does not support direct text file creation. .. image:: images/run-app-complete.png :align: center :width: 90% .. raw:: html
5. Click ``Run`` in the top-right corner to start the task. You will be taken to the ``Tasks`` tab to monitor progress. The status will change from ``Queued`` to ``Running`` to ``Completed`` if no errors occur. You will receive an email notification regardless of the job's outcome. .. image:: images/run-app-result.png :align: center :width: 90% .. raw:: html
6. Access the results by clicking the folder name in the ``Output`` column. The output folder contains individual Excel files for each experiment, which can be downloaded separately or as a zip file. .. note:: When a file/folder with the same name already exists on the platform, it will not be overwritten. Instead, a new file/folder will be created with a name preceded by a pattern like ``_#_``, e.g., ``test-folder-1`` becomes ``_1_test-folder-1``. Summary ======= This guide outlines the steps to deploy a command-line tool on the Seven Bridges platform, using the example tool ``db38pulling`` to query the db38 database and generate Excel files. Steps ----- 1. **Create the Tool:** - Navigate to your project, select the ``Apps`` tab, and create a new command-line tool. - Register the Docker image and configure the necessary input/output ports, arguments, and environment variables. 2. **Configure Secrets:** - Store sensitive information as secrets on the platform. - Add these secrets to the whitelist to allow the tool access. 3. **Run the Tool:** - Execute the tool from the ``Apps`` tab, set up inputs and outputs, and monitor the task in the ``Tasks`` tab. - Retrieve the results from the output folder, which contains individual Excel files. 4. **Manage Results:** - The platform prevents overwriting by renaming files/folders if duplicates exist. This summary provides a streamlined overview of creating, configuring, and running a command-line tool on Seven Bridges.