The Job Working Directory

The Job Working Directory

Access to the Job Working Directly(JWD) allows users to interact with running jobs for a variety of purposes such as checking the current output of a job and uploading necessary input files for a job mid-run. Users can access the JWD of a running job using the lcli job workdir sub-command. These are the flags available within each of the sub-commands under workdir:

  • add - Used to upload data from local storage/ persistent storage/ url to the JWD.

    • --path - (REQUIRED) Path in the Job Working Directory to upload data to.
    • --file - Input file from local storage.
    • --file-cached - Cached job input file.
    • --data - Input file from the user’s Lancium Persistent Storage Area.
    • --data-cached - Cached job input data.
    • --data-recursive - Input folder from the user’s Lancium Persistent Storage Area.
    • --url - URL to get input data from.
    • --url-cached - Cached job input URL.
    • --force - Overwrite an existing file if one already exists at this path.
  • get - Used to get the contents of a file within the JWD.

    • --path - (REQUIRED) Path in the Job Working Directory to retrieve data to.
    • --save - Path to download the file locally. Cannot be used in conjunction with the --tail flag.
    • --lines - Number of lines to tail. Cannot be used without the --tail flag. Defaults to 20.
    • --follow - Live tail a file. Cannot be used without the --tail flag. Must press control + c to kill command.
    • --view - Prints the result to the terminal.
    • --tail - Tail the file.
  • show - Used to show the contents of the JWD similar to the lcli data show command.

    • --path - If specified, allows users to take a closer look at the metadata for the file or directory path specified in the JWD.

Set-Up For Examples

For our examples, let’s run a simple job so we can access its JWD. First let’s create a bash script:

$ cd /tmp
$ touch bashscript.sh
$ nano bashscript.sh

#!/bin/bash

while true
do
echo "hello"
sleep 1
done

Let’s go ahead and run a job using this script:

$ lcli job run --name jwd_example_job --image lancium/ubuntu --file /tmp/bashscript.sh --command 'bash bashscript.sh' --max-run-time 7200

{
    "id": 1027,
    "name": "jwd_example_job",
    "status": "created",
    "qos": "high",
    "command_line": "bash bashscript.sh",
    "image": "ubuntu",
    "resources": {
        "node_exclusive": false,
        "core_count": 2,
        "gpu_count": null,
        "memory": 4,
        "gpu": null,
        "scratch": null,
        "mpi": false
    },
    "max_run_time": 7200,
    "input_files": [
        {
            "id": 2078,
            "name": "bashscript.sh",
            "source_type": "file",
            "source": "/tmp/bashscript.sh",
            "cache": false,
            "upload_complete": true,
            "chunks_received": [
                [
                    1,
                    55
                ]
            ]
        }
    ],
    "created_at": "2022-09-09T18:13:24.674Z",
    "updated_at": "2022-09-09T18:13:24.674Z"
}

Example: Show a File in the JWD

Let’s first take a look at the current contents of our JWD:

$ lcli job workdir show 1027

[
    {
        "name": "JOBNAME",
        "is_directory": false,
        "size": "16",
        "last_modified": "2022-09-09T18:34:42.697+00:00",
        "created": "2022-09-09T18:34:17.000+00:00"
    },
    {
        "name": "rusage-running.json",
        "is_directory": false,
        "size": "247",
        "last_modified": "2022-09-09T18:34:42.856+00:00",
        "created": "2022-09-09T18:34:20.000+00:00"
    },
    {
        "name": "stdout.txt",
        "is_directory": false,
        "size": "1266",
        "last_modified": "2022-09-09T18:34:42.981+00:00",
        "created": "2022-09-09T18:34:42.000+00:00"
    },
    {
        "name": "stderr.txt",
        "is_directory": false,
        "size": "438",
        "last_modified": "2022-09-09T18:34:43.103+00:00",
        "created": "2022-09-09T18:34:20.000+00:00"
    },
    {
        "name": ".singularityEnv",
        "is_directory": false,
        "size": "147",
        "last_modified": "2022-09-09T18:34:43.236+00:00",
        "created": "2022-09-09T18:34:20.000+00:00"
    },
    {
        "name": "hosts.txt",
        "is_directory": false,
        "size": "12",
        "last_modified": "2022-09-09T18:34:43.357+00:00",
        "created": "2022-09-09T18:34:20.000+00:00"
    },
    {
        "name": ".bes-info",
        "is_directory": false,
        "size": "54",
        "last_modified": "2022-09-09T18:34:43.475+00:00",
        "created": "2022-09-09T18:34:14.000+00:00"
    },
    {
        "name": ".genesisII-bes-state",
        "is_directory": true,
        "size": null,
        "last_modified": null,
        "created": null
    },
    {
        "name": "bashscript.sh",
        "is_directory": false,
        "size": "55",
        "last_modified": "2022-09-09T18:34:43.568+00:00",
        "created": "2022-09-09T18:34:15.000+00:00"
    },
    {
        "name": "qsub3471405012890072883.sh",
        "is_directory": false,
        "size": "1585",
        "last_modified": "2022-09-09T18:34:43.654+00:00",
        "created": "2022-09-09T18:34:19.000+00:00"
    }
]

Now let’s take a look at what happens if we include a path to the stderr.txt file in the above command:

$ lcli job workdir show --path stderr.txt 1027

{
    "length": "438",
    "last-modified": "Fri, 9 Sep 2022 18:36:11 +0000",
    "date-created": "Fri, 9 Sep 2022 18:34:20 +0000"
}

Example: Live Tailing a File

Now let’s try to live tail our stdout.txt file (Note: you will have to press control + c to kill this command):

$ lcli job workdir get --tail --follow --path stdout.txt 1027

'hello'
'hello'
'hello'
...
'hello'
'hello'
'hello'
...

Example: Save A File From The JWD To Local Storage

$ lcli job workdir get --save /tmp --path stderr.txt 1027

$ cat /tmp/stderr.txt
'
Parsing Env Var GENII_USER_DIR=.genesisII-bes-state, with value .genesisII-bes-state
Parsing Env Var GENII_INSTALL_DIR=/home/luser/GenesisII, with value /home/luser/GenesisII
Parsing Env Var JAVA_HOME=/home/luser/GenesisII/jre, with value /home/luser/GenesisII/jre
Parsing Env Var GENII_DEPLOYMENT_NAME=current_grid, with value current_grid
INFO:    Could not find any nv files on this host!
GID: readonly variable
UID: readonly variable
'

Example: Adding An Input File to the JWD

Let’s go ahead and add a file from local storage to our JWD (this can be any file):

$ lcli job workdir add --file /tmp/cosmoBench.py --path cosmo 1027

$ lcli job workdir show --path cosmo 1027

{
    "length": "913",
    "last-modified": "Fri, 9 Sep 2022 18:38:29 +0000",
    "date-created": "Fri, 9 Sep 2022 18:37:34 +0000"
}

Example: Adding An Input Folder to the JWD from the Persistent Storage

Let’s first take a look at a test folder in the data area named fred1 (this is a throwaway folder for example purposes):

$ lcli data show fred1
[
    {
        "name": "file1",
        "is_directory": false,
        "size": "0",
        "last_modified": "2022-09-09T18:40:15.503+00:00",
        "created": "2022-07-08T17:05:17.000+00:00"
    },
    {
        "name": "file2",
        "is_directory": false,
        "size": "0",
        "last_modified": "2022-09-09T18:40:15.673+00:00",
        "created": "2022-07-08T17:05:17.000+00:00"
    },
    {
        "name": "file3",
        "is_directory": false,
        "size": "0",
        "last_modified": "2022-09-09T18:40:15.913+00:00",
        "created": "2022-07-08T17:05:16.000+00:00"
    },
    {
        "name": "fred1",
        "is_directory": true,
        "size": null,
        "last_modified": null,
        "created": null
    }
]

Now let’s go ahead and add it to our JWD:

$ lcli job workdir add --data-recursive fred1 --path fred 1027

$ lcli job workdir show --path fred 1027
[
    {
        "name": "file1",
        "is_directory": false,
        "size": "0",
        "last_modified": "2022-09-09T18:41:35.668+00:00",
        "created": "2022-09-09T18:41:20.000+00:00"
    },
    {
        "name": "file2",
        "is_directory": false,
        "size": "0",
        "last_modified": "2022-09-09T18:41:35.849+00:00",
        "created": "2022-09-09T18:41:22.000+00:00"
    },
    {
        "name": "fred1",
        "is_directory": true,
        "size": null,
        "last_modified": null,
        "created": null
    },
    {
        "name": "file3",
        "is_directory": false,
        "size": "0",
        "last_modified": "2022-09-09T18:41:36.033+00:00",
        "created": "2022-09-09T18:41:23.000+00:00"
    }
]

Lesson 7: Running an MPI Job