=============================== Running MATLAB on Seven Bridges =============================== Introduction ============ This tutorial provides a comprehensive guide on how to run MATLAB applications on the Seven Bridges (SB) platform using a custom Docker container. You will learn how to prepare a MATLAB Docker image, configure an SB app using CWL (Common Workflow Language), and set up conditional logic to handle various input scenarios. By the end of this tutorial, you will have a fully functional MATLAB app on SB that can execute commands or scripts based on user inputs. Prepare MATLAB Docker Container =============================== This section covers the preparation of a Docker container that includes MATLAB and is configured to run on the SB platform. We will start from the official MATLAB Docker image and make necessary modifications to ensure compatibility with SB. *mathworks/matlab:r2024b* ------------------------- Begin by pulling the official MATLAB Docker image:: docker pull mathworks/matlab:r2024b This image serves as the base for our custom container. *images.sbgenomics.com/contextualize/matlab_cchen886:0.1.1* ----------------------------------------------------------- Starting from the base image, we perform the following modifications: - **Install Additional Tools**: Install ``vim`` and ``tree`` utilities for convenience. - **Modify ``/bin/util.sh``**: Replace the original script with a modified version that includes the ``-licmode onlinelicensing`` option. This change allows you to skip typing ``-licmode onlinelicensing`` when running MATLAB commands. - **Update Entry Point**: Change the entry point in the Dockerfile to:: ENTRYPOINT ["tini", "-s", "--", "sudo", "-u", "matlab", "/bin/run.sh"] This ensures that the container runs as the ``matlab`` user, which is critical for compatibility with the SB platform. *images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2* ----------------------------------------------------------- To create a Docker image that doesn't require authentication every time it runs, follow these steps: 1. **Run the Container and Authenticate**: .. code-block:: bash docker run --platform linux/amd64 -it --rm images.sbgenomics.com/contextualize/matlab_cchen886:0.1.1 At the prompt, enter your MathWorks Account email address and follow the instructions to authenticate using a one-time password. 2. **Commit the Authenticated Container**: Find the running container's ID (e.g., ``a85e56b141b7``) and commit it to create a new image: .. code-block:: bash docker commit a85e56b141b7 images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 You now have an authenticated Docker image. Test it with the following commands: - **Run a Simple Command**: .. code-block:: bash docker run --platform linux/amd64 --rm images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 -batch 'disp("Hello, World!")' - **Run a MATLAB Script from Your Current Directory**: .. code-block:: bash docker run --platform linux/amd64 --rm -v $(PWD):/home/matlab/Documents/MATLAB images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 -batch 'myscript' Ensure that ``myscript.m`` is in your current directory. An example of ``myscript.m``: .. code-block:: matlab % myscript.m % A simple MATLAB script for testing disp('Hello, MATLAB!'); x = 1:5; % Define a simple array y = x.^2; % Square each element of x disp('x = '); disp(x); disp('y = '); disp(y); *images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3* ----------------------------------------------------------- I encountered difficulties installing additional MATLAB toolboxes in the official image. As a result, I decided to remove the existing MATLAB installation in the image and reinstall it. The following command starts a container with privileged access, mounting the MATLAB installer ISO file. The ISO file is located at ``/Users/cchen/MyDrive/archive/utility/linux/MATLAB_R2024b/R2024b_Linux.iso`` on your local machine. Additionally, the command exposes ports for VNC and a web interface, providing access to the MATLAB GUI. .. code-block:: bash # More capable command allowing access to the GUI docker run --init -it --rm --privileged -v /Users/cchen/MyDrive/archive/utility/linux/MATLAB_R2024b/R2024b_Linux.iso:/mnt/matlab_iso.iso:ro -p 5901:5901 -p 6080:6080 --shm-size=512M images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 -vnc # Minimal command to run the container on a headless environment docker run --init -it --rm --privileged -v /Users/cchen/MyDrive/archive/utility/linux/MATLAB_R2024b/R2024b_Linux.iso:/mnt/matlab_iso.iso:ro --shm-size=512M images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 Now in your browser navigate to ``http://localhost:6080`` and log in with the credentials. Next, run the following command in the terminal: .. code-block:: bash sudo mkdir /mnt/matlab_installer sudo mount -o loop /mnt/matlab_iso.iso /mnt/matlab_installer sudo rm -rf /opt/matlab/R2024b cd /home/matlab/Documents/MATLAB cp /mnt/matlab_installer/installer_input.txt . chmod 755 installer_input.txt The content of the modified installer_input.txt is as follows: .. code-block:: text destinationFolder=/opt/matlab/R2024b fileInstallationKey=08123-49505-***-32648 agreeToLicense=yes enableLNU=yes improveMATLAB=yes product.MATLAB product.Signal_Processing_Toolbox product.Statistics_and_Machine_Learning_Toolbox Run the following command to install MATLAB: .. code-block:: bash sudo /mnt/matlab_installer/install -inputFile installer_input.txt The log info will be saved in ``/tmp/mathworks_root.log`` by default (``/tmp/mathworks_matlab.log`` if you used user ``matlab`` to install the package). After installation, while the docker container is active, run the following command in a separate terminal to create a new image: .. code-block:: bash docker commit images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3 The new image is 10.54 GB in size, and whether or not you run ``sudo umount /mnt/matlab_installer`` is not affecting the size of the docker image. Configure the App on SB ======================= This section guides you through setting up a MATLAB app on the SB platform, including defining inputs and outputs, setting up the command-line logic, and handling various input scenarios. Prerequisites ------------- - Access to your Seven Bridges platform account. - Familiarity with Docker and CWL. - The authenticated Docker image: `images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3` Step-by-Step Instructions ------------------------- 1. **Create a New App on SB**: - Log in to your SB platform account. - Navigate to the **Apps** section and click **Create New App**. - Choose **Command Line Tool** and select the **CWL Editor**. 2. **Define App Metadata**: - **App Name**: `matlab-hello-world` - **Description**: A MATLAB app that runs commands or scripts based on user input. - **Project**: Select your project (e.g., `sandbox-chen`). 3. **Set Up Docker Image**: Under **Docker Pull**, specify your Docker image: `images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3` 4. **Define Inputs**: - `matlab_command`: - **ID**: `matlab_command` - **Type**: `string?` (optional) - **Default**: Leave blank or `null` - `m_file`: - **ID**: `m_file` - **Type**: `File?` (optional) - `workdir`: - **ID**: `workdir` - **Type**: `Directory?` (optional) 5. **Define Outputs**: - `output`: - **ID**: `output` - **Type**: `File?` - **Output Binding**: .. code-block:: javascript { "glob": "*.out" } 6. **Specify Base Command**: Leave the `baseCommand` field empty. We will construct the command using arguments. 7. **Add Arguments**: We will define arguments that build the command-line instruction based on the inputs. - **Argument 1**: - **Prefix**: `-batch` - **Position**: `0` - **Shell Quote**: `false` - **Value From**: .. code-block:: javascript ${ if (!inputs.matlab_command && !inputs.m_file) { return "'disp(\"To run the application successfully, please provide either a matlab_command or an m_file.\")'"; } else if (inputs.matlab_command && inputs.m_file) { return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); " + inputs.matlab_command + "; run(\"" + inputs.m_file.basename + "\")'"; } else if (inputs.matlab_command) { return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo chmod a+r *\"); " + inputs.matlab_command + "'"; } else if (inputs.m_file) { return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); run(\"" + inputs.m_file.basename + "\")'"; } } - **Argument 2**: - **Prefix**: *(Leave blank)* - **Position**: `1` - **Shell Quote**: `false` - **Value From**: `>> stdout.out || echo "MATLAB failed with exit code $?" >> stdout.out` 8. **Set Up Requirements**: Add the following requirements to the app: - **ShellCommandRequirement**: Allows the use of shell commands in the CWL script. - **DockerRequirement**: Specifies the Docker image. - **InlineJavascriptRequirement**: Enables JavaScript expressions. 9. **Include Full CWL Code of the App**: .. code-block:: text cwlVersion: v1.2 class: CommandLineTool $namespaces: sbg: https://sevenbridges.com label: matlab-hello-world baseCommand: [] inputs: - id: matlab_command type: 'string?' default: null - id: m_file type: 'File?' - id: workdir type: 'Directory?' outputs: - id: output type: 'File?' outputBinding: glob: '*.out' arguments: - prefix: -batch shellQuote: false position: 0 valueFrom: ${ if (!inputs.matlab_command && !inputs.m_file) { return "'disp(\"To run the application successfully, please provide either a matlab_command or an m_file.\")'"; } else if (inputs.matlab_command && inputs.m_file) { return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); " + inputs.matlab_command + "; run(\"" + inputs.m_file.basename + "\")'"; } else if (inputs.matlab_command) { return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo chmod a+r *\"); " + inputs.matlab_command + "'"; } else if (inputs.m_file) { return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); run(\"" + inputs.m_file.basename + "\")'"; } } - prefix: '' shellQuote: false position: 1 valueFrom: '>> stdout.out || echo "MATLAB failed with exit code $?" >> stdout.out' requirements: - class: ShellCommandRequirement - class: DockerRequirement dockerPull: images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3 - class: InlineJavascriptRequirement hints: sbg:projectName: sandbox-chen Explanation of the Logic ------------------------ The CWL script uses JavaScript expressions to handle inputs and file management: - **No Inputs Provided**: Displays a message requesting input: .. code-block:: matlab 'disp("To run the application successfully, please provide either a matlab_command or an m_file.")' - **Both Inputs Provided**: Copies working directory contents, runs MATLAB command, then executes the m-file: .. code-block:: bash 'system("sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .; sudo cp m_file .; sudo chmod a+r *"); matlab_command; run("m_file")' - **Only `matlab_command` Provided**: Copies working directory contents and executes MATLAB command: .. code-block:: bash 'system("sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .; sudo chmod a+r *"); matlab_command' - **Only `m_file` Provided**: Copies working directory contents and runs the m-file: .. code-block:: bash 'system("sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .; sudo cp m_file .; sudo chmod a+r *"); run("m_file")' File Management ~~~~~~~~~~~~~~~ The app now includes comprehensive file management: 1. Copies all files from the working directory: .. code-block:: bash sudo ls workdir/ | xargs -I {} sudo cp workdir/{} . .. note:: The goal here is just copying files to the current directory. The reason for using this is that when you use ``sudo cp /path/to/file/* .`` it will not work if the file is not owned by the user running the command even if you are running it using sudo. The workaround is to use ``xargs`` to copy files one by one, since I found that the command ``ls`` worked and copying single files worked as well. 2. Copies the m-file if provided: .. code-block:: bash sudo cp m_file . 3. Sets read permissions for all files: .. code-block:: bash sudo chmod a+r * Testing the App --------------- Test each scenario: 1. **No Inputs**: - Should display the help message 2. **With Working Directory**: - Verify all files are copied correctly - Check file permissions 3. **With MATLAB Command**: - Confirm working directory files are available - Verify command execution 4. **With M-File**: - Check file copying - Verify script execution 5. **All Inputs**: - Verify complete workflow - Check output generation Additional Caveats ------------------ There are a few caveats I encountered during the process: - For memory-intensive MATLAB applications, consider adjusting the Docker container's memory limits. Specifically, when running mass flow script, in the task configuration state, you will need to go to the ``Execution Settings`` tab and choose custom instance type ``c4.4xlarge`` or higher, which allocates 30 GB of memory as compared to the default of 15 GB in the ``c4.2xlarge`` instance type. Conclusion ========== By following this tutorial, you have successfully set up a MATLAB-based app on the Seven Bridges platform. The app is capable of executing MATLAB commands or scripts based on user inputs, with proper handling of different input scenarios. This flexible configuration allows for a wide range of MATLAB applications to be run on the SB platform, leveraging the power of Docker containers and CWL. Additional Resources ==================== - **Seven Bridges Documentation**: https://docs.sevenbridges.com - **CWL Specification**: https://www.commonwl.org - **MATLAB Docker Repository**: https://hub.docker.com/r/mathworks/matlab