Working with Custom Application Images

Ex 1) Pulling Stock Docker Image (URL)

Docker images can be placed into Lancium’s image area either through uploading a local image file or directing the CLI towards a Docker Hub image URL. The CLI differentiates the two types: local file (docker_file) and Docker Hub url (docker_image). Note in the below code snippet the format of the URL at which the image lives: docker://{image_name}:{image_version} .

# The following command fetches and builds a Python Docker Image with an environment variable

$ lcli image add --name pythonDockerURL --url docker://python --type docker_image test/pythonDockerURL --env test=hello

{
    "path": "test/pythonDockerURL",
    "name": "pythonDockerURL",
    "source_type": "docker_image",
    "source_url": "docker://python",
    "environment": [
        {
            "variable": "test",
            "value": "hello"
        }
    ]
}


# Let's look at the image after build completion
$ lcli image show test/pythonDockerURL

[
    {
        "path": "test/pythonDockerURL3",
        "name": "pythonDockerURL3",
        "source_type": "docker_image",
        "source_url": "docker://python",
        "status": "ready",
        "build_output": {'metadata'},
        "created_at": "2022-05-17T14:36:27.322Z",
        "updated_at": "2022-05-17T14:41:33.959Z",
        "built_at": "2022-05-17T14:41:33.957Z"
    }
]

Ex 2) Writing a Docker Image

Docker images do not need to come from Docker Hub URL’s or downloaded image files; it is fairly simple to write one’s own docker container.

The following is a Docker File written on an ubuntu base.

#syntax=docker/dockerfile:1

FROM debian:latest

RUN apt update -y
RUN apt install python3 -y

ENV to_test='hello'

The top line indicating the syntax is an optional parser directive. It lets the Docker builder know what syntax to use when parsing this file. The FROM debian:latest command sets the base image to extend upon as the latest, public Debian distributed image. The WORKDIR command sets the working directory for the Docker container; if the directory does not exist, it will create a working directory of that name and path. WORKDIR can be used as many times as needed to set a new working directory for the container. COPY is a command that will copy a local file to the destination path. The RUN command is used to run commands when an image is instatiated. The ENV command can be used to set environment variables within a container.

Let’s add this image your Lancium account’s image storage:

$ lcli image add --name "Debian Custom Docker" --type docker_file --build-script ~/bin/pyExtendImage test/DebianDocker

{
    "path": "test/DebianDocker",
    "name": "Debian Custom Docker",
    "source_type": "docker_file",
    "build_script": "#syntax=docker/dockerfile:1\n\nFROM debian:latest\n\nRUN apt update -y\nRUN apt install python3 -y\n\nENV to_test='hello'\n"
}

$ lcli image show test/DebianDocker
[
    {
        "path": "test/DebianDocker",
        "name": "Debian Custom Docker",
        "source_type": "docker_file",
        "build_script": "#syntax=docker/dockerfile:1\n\nFROM debian:latest\n\nRUN apt update -y\nRUN apt install python3 -y\n\nENV to_test='hello'\n",
        "status": "ready",
        "build_output": "{metadata}",
        "created_at": "2022-05-19T19:41:48.002Z",
        "updated_at": "2022-05-19T19:46:52.111Z",
        "built_at": "2022-05-19T19:46:52.110Z"
    }
]

Once the image is ready, we can run a job to print out the environment variable to_test. We will use a simply python script that fetches the environment variable and prints it.

# Let's take a look at the python script that we will inject into the job
$ cat ~/bin/testEnvVar.py
#output
'''
import os

print(os.getenv('to_test'))
'''

# Let's run a job - note that because we installed the python package in the 
# custom Docker image, we can use "python3" commands
$ lcli job run --name "print to_test" --image test/DebianDocker --command "python3 testEnvVar.py" --input-file ~/bin/testEnvVar.py 

{
    "id": 48281,
    "name": "print to_test",
    "status": "created",
    "qos": "high",
    "command_line": "python3 testEnvVar.py",
    "image": "test/DebianDocker",
    "resources": {
        "core_count": 2,
        "gpu_count": null,
        "memory": 4,
        "gpu": null,
        "scratch": null
    },
    "max_run_time": 259200,
    "input_files": [
        {
            "id": 11257,
            "name": "testEnvVar.py",
            "source_type": "file",
            "source": "/home/rap/bin/testEnvVar.py",
            "cache": false,
            "upload_complete": true,
            "chunks_received": [
                [
                    1,
                    38
                ]
            ]
        }
    ],
    "created_at": "2022-05-19T20:11:57.815Z",
    "updated_at": "2022-05-19T20:11:57.815Z"
}

# Let's take a look at the output once the job is finished.
$ lcli job output get --view --file "stdout.txt" 48281
'hello'

Ex 3) Writing a Singularity Image

Now let’s write a singularity recipe:

# Header
Bootstrap: docker
From: debian


%environment
    export to_test='hello'
    
%post
	apt update -y
	apt install python3 -y

The header is written at the top of a singularity file and describes the base operating system. The library key word indicates going to the Container Library for the OS base. Other options to the Container Library include: docker (Docker Hub), shub (Singularity Hub), oras (OCI registries), and scratch (build a container from scratch). In addition to the header, there are a few different sections that can be included in a singularity definition:

%setup : commands here are first executed in the host operating system outside of the container for “setup” purposes. However, the container can be accessed by specifying %SINGULARITY_ROOTFS.
%files : this can be used to copy files into the container
%post : this can be used to specify downloading files from the internet.
%test : this runs at the end of the container build to validate the container.
%environment : this allows environment variable exporting to the container.
%startscript : this runs within the container at build time
%runscript : this is written to a file within the container and executed when the image is run.
%labels : this adds metadata to your “labels.json” file within the container
%help this indicates insertion into the metadata file.

Now, let’s add the Singularity image to your Lancium image directory:

$ lcli image add --name "Debian Custom Simg" --type singularity_file --build-script ~/bin/pyCustomSimg.simg test/DebianSingularity 

{
    "path": "test/DebianSingularity",
    "name": "Debian Custom Simg",
    "source_type": "singularity_file",
    "build_script": "Bootstrap: docker\nFrom: debian\n\n\n%environment\n    export to_test='hello'\n    \n%post\n\tapt update -y\n\tapt install python3 -y\n"
}

$ lcli image show test/customPySingularity

[
    {
        "path": "test/DebianSingularity",
        "name": "Debian Custom Simg",
        "source_type": "singularity_file",
        "build_script": "Bootstrap: docker\nFrom: debian\n\n\n%environment\n    export to_test='hello'\n    \n%post\n\tapt update -y\n\tapt install python3 -y\n",
        "status": "ready",
        "build_output": "{metadata}",
        "created_at": "2022-05-19T20:34:00.755Z",
        "updated_at": "2022-05-19T20:39:03.780Z",
        "built_at": "2022-05-19T20:39:03.779Z"
    }
]

Finally, let’s run a simple job to test whether the to_test variable was exported to the environment and whether python3 was successfully installed within the singularity container:

# Let's take a look at the python script that will be injected into the
# job.
$ cat ~/bin/testEnvVar.py
#output
'''
import os

print(os.getenv('to_test'))
'''

# Let's run the job -- note that we are using python3 in the command
# to ensure that the python package was successfully installed.
$ lcli job run --name "testing Singularity Custom Image" --image test/DebianSingularity --command "python3 testEnvVar.py" --input-file ~/bin/testEnvVar.py 

{
    "id": 48283,
    "name": "testing Singularity Custom Image",
    "status": "created",
    "qos": "high",
    "command_line": "python3 testEnvVar.py",
    "image": "test/DebianSingularity",
    "resources": {
        "core_count": 2,
        "gpu_count": null,
        "memory": 4,
        "gpu": null,
        "scratch": null
    },
    "max_run_time": 259200,
    "input_files": [
        {
            "id": 11258,
            "name": "testEnvVar.py",
            "source_type": "file",
            "source": "/home/rap/bin/testEnvVar.py",
            "cache": false,
            "upload_complete": true,
            "chunks_received": [
                [
                    1,
                    38
                ]
            ]
        }
    ],
    "created_at": "2022-05-19T20:49:22.903Z",
    "updated_at": "2022-05-19T20:49:22.903Z"
}

# Let's take a look at the output
$ lcli job output get --view --file "stdout.txt" 48283
'hello'