Running MATLAB on Seven Bridges

Introduction

This tutorial provides a comprehensive guide on how to run MATLAB applications on the Seven Bridges (SB) platform using a custom Docker container. You will learn how to prepare a MATLAB Docker image, configure an SB app using CWL (Common Workflow Language), and set up conditional logic to handle various input scenarios. By the end of this tutorial, you will have a fully functional MATLAB app on SB that can execute commands or scripts based on user inputs.

Prepare MATLAB Docker Container

This section covers the preparation of a Docker container that includes MATLAB and is configured to run on the SB platform. We will start from the official MATLAB Docker image and make necessary modifications to ensure compatibility with SB.

mathworks/matlab:r2024b

Begin by pulling the official MATLAB Docker image:

docker pull mathworks/matlab:r2024b

This image serves as the base for our custom container.

images.sbgenomics.com/contextualize/matlab_cchen886:0.1.1

Starting from the base image, we perform the following modifications:

  • Install Additional Tools: Install vim and tree utilities for convenience.

  • Modify ``/bin/util.sh``: Replace the original script with a modified version that includes the -licmode onlinelicensing option. This change allows you to skip typing -licmode onlinelicensing when running MATLAB commands.

  • Update Entry Point:

    Change the entry point in the Dockerfile to:

    ENTRYPOINT ["tini", "-s", "--", "sudo", "-u", "matlab", "/bin/run.sh"]

    This ensures that the container runs as the matlab user, which is critical for compatibility with the SB platform.

images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2

To create a Docker image that doesn’t require authentication every time it runs, follow these steps:

  1. Run the Container and Authenticate:

    docker run --platform linux/amd64 -it --rm images.sbgenomics.com/contextualize/matlab_cchen886:0.1.1

    At the prompt, enter your MathWorks Account email address and follow the instructions to authenticate using a one-time password.

  2. Commit the Authenticated Container:

    Find the running container’s ID (e.g., a85e56b141b7) and commit it to create a new image:

    docker commit a85e56b141b7 images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2

You now have an authenticated Docker image. Test it with the following commands:

  • Run a Simple Command:

    docker run --platform linux/amd64 --rm images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 -batch 'disp("Hello, World!")'

  • Run a MATLAB Script from Your Current Directory:

    docker run --platform linux/amd64 --rm -v $(PWD):/home/matlab/Documents/MATLAB images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 -batch 'myscript'

    Ensure that myscript.m is in your current directory. An example of myscript.m:

    % myscript.m
% A simple MATLAB script for testing

disp('Hello, MATLAB!');
x = 1:5;        % Define a simple array
y = x.^2;       % Square each element of x
disp('x = ');
disp(x);
disp('y = ');
disp(y);

images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3

I encountered difficulties installing additional MATLAB toolboxes in the official image. As a result, I decided to remove the existing MATLAB installation in the image and reinstall it.

The following command starts a container with privileged access, mounting the MATLAB installer ISO file. The ISO file is located at /Users/cchen/MyDrive/archive/utility/linux/MATLAB_R2024b/R2024b_Linux.iso on your local machine. Additionally, the command exposes ports for VNC and a web interface, providing access to the MATLAB GUI.

# More capable command allowing access to the GUI
docker run --init -it --rm --privileged -v /Users/cchen/MyDrive/archive/utility/linux/MATLAB_R2024b/R2024b_Linux.iso:/mnt/matlab_iso.iso:ro -p 5901:5901 -p 6080:6080 --shm-size=512M images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2 -vnc

# Minimal command to run the container on a headless environment
docker run --init -it --rm --privileged -v /Users/cchen/MyDrive/archive/utility/linux/MATLAB_R2024b/R2024b_Linux.iso:/mnt/matlab_iso.iso:ro --shm-size=512M images.sbgenomics.com/contextualize/matlab_cchen886:0.1.2

Now in your browser navigate to http://localhost:6080 and log in with the credentials. Next, run the following command in the terminal:

sudo mkdir /mnt/matlab_installer
sudo mount -o loop /mnt/matlab_iso.iso /mnt/matlab_installer
sudo rm -rf /opt/matlab/R2024b
cd /home/matlab/Documents/MATLAB
cp /mnt/matlab_installer/installer_input.txt .
chmod 755 installer_input.txt

The content of the modified installer_input.txt is as follows:

destinationFolder=/opt/matlab/R2024b
fileInstallationKey=08123-49505-***-32648
agreeToLicense=yes
enableLNU=yes
improveMATLAB=yes
product.MATLAB
product.Signal_Processing_Toolbox
product.Statistics_and_Machine_Learning_Toolbox

Run the following command to install MATLAB:

sudo /mnt/matlab_installer/install -inputFile installer_input.txt

The log info will be saved in /tmp/mathworks_root.log by default (/tmp/mathworks_matlab.log if you used user matlab to install the package). After installation, while the docker container is active, run the following command in a separate terminal to create a new image:

docker commit <container_id> images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3

The new image is 10.54 GB in size, and whether or not you run sudo umount /mnt/matlab_installer is not affecting the size of the docker image.

Configure the App on SB

This section guides you through setting up a MATLAB app on the SB platform, including defining inputs and outputs, setting up the command-line logic, and handling various input scenarios.

Prerequisites

  • Access to your Seven Bridges platform account.

  • Familiarity with Docker and CWL.

  • The authenticated Docker image: images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3

Step-by-Step Instructions

  1. Create a New App on SB:

    • Log in to your SB platform account.

    • Navigate to the Apps section and click Create New App.

    • Choose Command Line Tool and select the CWL Editor.

  2. Define App Metadata:

    • App Name: matlab-hello-world

    • Description: A MATLAB app that runs commands or scripts based on user input.

    • Project: Select your project (e.g., sandbox-chen).

  3. Set Up Docker Image:

    Under Docker Pull, specify your Docker image: images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3

  4. Define Inputs:

    • matlab_command:

    • ID: matlab_command

    • Type: string? (optional)

    • Default: Leave blank or null

    • m_file:

    • ID: m_file

    • Type: File? (optional)

    • workdir:

    • ID: workdir

    • Type: Directory? (optional)

  5. Define Outputs:

    • output:

    • ID: output

    • Type: File?

    • Output Binding:

      {
    "glob": "*.out"
}

  6. Specify Base Command:

    Leave the baseCommand field empty. We will construct the command using arguments.

  7. Add Arguments:

    We will define arguments that build the command-line instruction based on the inputs.

    • Argument 1:

      • Prefix: -batch

      • Position: 0

      • Shell Quote: false

      • Value From:

        ${
    if (!inputs.matlab_command && !inputs.m_file) {
        return "'disp(\"To run the application successfully, please provide either a matlab_command or an m_file.\")'";
    } else if (inputs.matlab_command && inputs.m_file) {
        return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); " + inputs.matlab_command + "; run(\"" + inputs.m_file.basename + "\")'";
    } else if (inputs.matlab_command) {
        return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo chmod a+r *\"); " + inputs.matlab_command + "'";
    } else if (inputs.m_file) {
        return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); run(\"" + inputs.m_file.basename + "\")'";
    }
}

    • Argument 2:

      • Prefix: (Leave blank)

      • Position: 1

      • Shell Quote: false

      • Value From: >> stdout.out || echo “MATLAB failed with exit code $?” >> stdout.out

  8. Set Up Requirements:

    Add the following requirements to the app:

    • ShellCommandRequirement: Allows the use of shell commands in the CWL script.

    • DockerRequirement: Specifies the Docker image.

    • InlineJavascriptRequirement: Enables JavaScript expressions.

  9. Include Full CWL Code of the App:

    cwlVersion: v1.2
class: CommandLineTool
$namespaces:
    sbg: https://sevenbridges.com
label: matlab-hello-world
baseCommand: []
inputs:
    - id: matlab_command
      type: 'string?'
      default: null
    - id: m_file
      type: 'File?'
    - id: workdir
      type: 'Directory?'
outputs:
    - id: output
      type: 'File?'
      outputBinding:
          glob: '*.out'
arguments:
    - prefix: -batch
      shellQuote: false
      position: 0
      valueFrom: ${
          if (!inputs.matlab_command && !inputs.m_file) {
              return "'disp(\"To run the application successfully, please provide either a matlab_command or an m_file.\")'";
          } else if (inputs.matlab_command && inputs.m_file) {
              return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); " + inputs.matlab_command + "; run(\"" + inputs.m_file.basename + "\")'";
          } else if (inputs.matlab_command) {
              return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo chmod a+r *\"); " + inputs.matlab_command + "'";
          } else if (inputs.m_file) {
              return "'system(\"sudo ls " + inputs.workdir.path + "/ | xargs -I {} sudo cp " + inputs.workdir.path + "/{} .; sudo cp " + inputs.m_file.path + " .; sudo chmod a+r *\"); run(\"" + inputs.m_file.basename + "\")'";
          }
      }
    - prefix: ''
      shellQuote: false
      position: 1
      valueFrom: '>> stdout.out || echo "MATLAB failed with exit code $?" >> stdout.out'
requirements:
    - class: ShellCommandRequirement
    - class: DockerRequirement
      dockerPull: images.sbgenomics.com/contextualize/matlab_cchen886:0.1.3
    - class: InlineJavascriptRequirement
hints:
    sbg:projectName: sandbox-chen

Explanation of the Logic

The CWL script uses JavaScript expressions to handle inputs and file management:

  • No Inputs Provided:

    Displays a message requesting input:

    'disp("To run the application successfully, please provide either a matlab_command or an m_file.")'

  • Both Inputs Provided:

    Copies working directory contents, runs MATLAB command, then executes the m-file:

    'system("sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .; sudo cp m_file .; sudo chmod a+r *"); matlab_command; run("m_file")'

  • Only `matlab_command` Provided:

    Copies working directory contents and executes MATLAB command:

    'system("sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .; sudo chmod a+r *"); matlab_command'

  • Only `m_file` Provided:

    Copies working directory contents and runs the m-file:

    'system("sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .; sudo cp m_file .; sudo chmod a+r *"); run("m_file")'

File Management

The app now includes comprehensive file management:

  1. Copies all files from the working directory:
    sudo ls workdir/ | xargs -I {} sudo cp workdir/{} .

    Note

    The goal here is just copying files to the current directory. The reason for using this is that when you use sudo cp /path/to/file/* . it will not work if the file is not owned by the user running the command even if you are running it using sudo. The workaround is to use xargs to copy files one by one, since I found that the command ls worked and copying single files worked as well.

  2. Copies the m-file if provided:
    sudo cp m_file .
  3. Sets read permissions for all files:
    sudo chmod a+r *

Testing the App

Test each scenario:

  1. No Inputs:

    • Should display the help message

  2. With Working Directory:

    • Verify all files are copied correctly

    • Check file permissions

  3. With MATLAB Command:

    • Confirm working directory files are available

    • Verify command execution

  4. With M-File:

    • Check file copying

    • Verify script execution

  5. All Inputs:

    • Verify complete workflow

    • Check output generation

Additional Caveats

There are a few caveats I encountered during the process:

  • For memory-intensive MATLAB applications, consider adjusting the Docker container’s memory limits. Specifically, when running mass flow script, in the task configuration state, you will need to go to the Execution Settings tab and choose custom instance type c4.4xlarge or higher, which allocates 30 GB of memory as compared to the default of 15 GB in the c4.2xlarge instance type.

Conclusion

By following this tutorial, you have successfully set up a MATLAB-based app on the Seven Bridges platform. The app is capable of executing MATLAB commands or scripts based on user inputs, with proper handling of different input scenarios. This flexible configuration allows for a wide range of MATLAB applications to be run on the SB platform, leveraging the power of Docker containers and CWL.

Additional Resources