The Job Working Directory
Access to the Job Working Directly(JWD) allows users to interact with running jobs for a variety of purposes such as checking the current output of a job and uploading necessary input files for a job mid-run. Users can access the JWD of a running job using the lcli job workdir sub-command. These are the flags available within each of the sub-commands under workdir:
-
add- Used to upload data from local storage/ persistent storage/ url to the JWD.--path- (REQUIRED) Path in the Job Working Directory to upload data to.--file- Input file from local storage.--file-cached- Cached job input file.--data- Input file from the user’s Lancium Persistent Storage Area.--data-cached- Cached job input data.--data-recursive- Input folder from the user’s Lancium Persistent Storage Area.--url- URL to get input data from.--url-cached- Cached job input URL.--force- Overwrite an existing file if one already exists at this path.
-
get- Used to get the contents of a file within the JWD.--path- (REQUIRED) Path in the Job Working Directory to retrieve data to.--save- Path to download the file locally. Cannot be used in conjunction with the--tailflag.--lines- Number of lines to tail. Cannot be used without the--tailflag. Defaults to 20.--follow- Live tail a file. Cannot be used without the--tailflag. Must presscontrol + cto kill command.--view- Prints the result to the terminal.--tail- Tail the file.
-
show- Used to show the contents of the JWD similar to thelcli data showcommand.--path- If specified, allows users to take a closer look at the metadata for the file or directory path specified in the JWD.
Set-Up For Examples
For our examples, let’s run a simple job so we can access its JWD. First let’s create a bash script:
$ cd /tmp
$ touch bashscript.sh
$ nano bashscript.sh
#!/bin/bash
while true
do
echo "hello"
sleep 1
done
Let’s go ahead and run a job using this script:
$ lcli job run --name jwd_example_job --image lancium/ubuntu --file /tmp/bashscript.sh --command 'bash bashscript.sh' --max-run-time 7200
{
"id": 1027,
"name": "jwd_example_job",
"status": "created",
"qos": "high",
"command_line": "bash bashscript.sh",
"image": "ubuntu",
"resources": {
"node_exclusive": false,
"core_count": 2,
"gpu_count": null,
"memory": 4,
"gpu": null,
"scratch": null,
"mpi": false
},
"max_run_time": 7200,
"input_files": [
{
"id": 2078,
"name": "bashscript.sh",
"source_type": "file",
"source": "/tmp/bashscript.sh",
"cache": false,
"upload_complete": true,
"chunks_received": [
[
1,
55
]
]
}
],
"created_at": "2022-09-09T18:13:24.674Z",
"updated_at": "2022-09-09T18:13:24.674Z"
}
Example: Show a File in the JWD
Let’s first take a look at the current contents of our JWD:
$ lcli job workdir show 1027
[
{
"name": "JOBNAME",
"is_directory": false,
"size": "16",
"last_modified": "2022-09-09T18:34:42.697+00:00",
"created": "2022-09-09T18:34:17.000+00:00"
},
{
"name": "rusage-running.json",
"is_directory": false,
"size": "247",
"last_modified": "2022-09-09T18:34:42.856+00:00",
"created": "2022-09-09T18:34:20.000+00:00"
},
{
"name": "stdout.txt",
"is_directory": false,
"size": "1266",
"last_modified": "2022-09-09T18:34:42.981+00:00",
"created": "2022-09-09T18:34:42.000+00:00"
},
{
"name": "stderr.txt",
"is_directory": false,
"size": "438",
"last_modified": "2022-09-09T18:34:43.103+00:00",
"created": "2022-09-09T18:34:20.000+00:00"
},
{
"name": ".singularityEnv",
"is_directory": false,
"size": "147",
"last_modified": "2022-09-09T18:34:43.236+00:00",
"created": "2022-09-09T18:34:20.000+00:00"
},
{
"name": "hosts.txt",
"is_directory": false,
"size": "12",
"last_modified": "2022-09-09T18:34:43.357+00:00",
"created": "2022-09-09T18:34:20.000+00:00"
},
{
"name": ".bes-info",
"is_directory": false,
"size": "54",
"last_modified": "2022-09-09T18:34:43.475+00:00",
"created": "2022-09-09T18:34:14.000+00:00"
},
{
"name": ".genesisII-bes-state",
"is_directory": true,
"size": null,
"last_modified": null,
"created": null
},
{
"name": "bashscript.sh",
"is_directory": false,
"size": "55",
"last_modified": "2022-09-09T18:34:43.568+00:00",
"created": "2022-09-09T18:34:15.000+00:00"
},
{
"name": "qsub3471405012890072883.sh",
"is_directory": false,
"size": "1585",
"last_modified": "2022-09-09T18:34:43.654+00:00",
"created": "2022-09-09T18:34:19.000+00:00"
}
]
Now let’s take a look at what happens if we include a path to the stderr.txt file in the above command:
$ lcli job workdir show --path stderr.txt 1027
{
"length": "438",
"last-modified": "Fri, 9 Sep 2022 18:36:11 +0000",
"date-created": "Fri, 9 Sep 2022 18:34:20 +0000"
}
Example: Live Tailing a File
Now let’s try to live tail our stdout.txt file (Note: you will have to press control + c to kill this command):
$ lcli job workdir get --tail --follow --path stdout.txt 1027
'hello'
'hello'
'hello'
...
'hello'
'hello'
'hello'
...
Example: Save A File From The JWD To Local Storage
$ lcli job workdir get --save /tmp --path stderr.txt 1027
$ cat /tmp/stderr.txt
'
Parsing Env Var GENII_USER_DIR=.genesisII-bes-state, with value .genesisII-bes-state
Parsing Env Var GENII_INSTALL_DIR=/home/luser/GenesisII, with value /home/luser/GenesisII
Parsing Env Var JAVA_HOME=/home/luser/GenesisII/jre, with value /home/luser/GenesisII/jre
Parsing Env Var GENII_DEPLOYMENT_NAME=current_grid, with value current_grid
INFO: Could not find any nv files on this host!
GID: readonly variable
UID: readonly variable
'
Example: Adding An Input File to the JWD
Let’s go ahead and add a file from local storage to our JWD (this can be any file):
$ lcli job workdir add --file /tmp/cosmoBench.py --path cosmo 1027
$ lcli job workdir show --path cosmo 1027
{
"length": "913",
"last-modified": "Fri, 9 Sep 2022 18:38:29 +0000",
"date-created": "Fri, 9 Sep 2022 18:37:34 +0000"
}
Example: Adding An Input Folder to the JWD from the Persistent Storage
Let’s first take a look at a test folder in the data area named fred1 (this is a throwaway folder for example purposes):
$ lcli data show fred1
[
{
"name": "file1",
"is_directory": false,
"size": "0",
"last_modified": "2022-09-09T18:40:15.503+00:00",
"created": "2022-07-08T17:05:17.000+00:00"
},
{
"name": "file2",
"is_directory": false,
"size": "0",
"last_modified": "2022-09-09T18:40:15.673+00:00",
"created": "2022-07-08T17:05:17.000+00:00"
},
{
"name": "file3",
"is_directory": false,
"size": "0",
"last_modified": "2022-09-09T18:40:15.913+00:00",
"created": "2022-07-08T17:05:16.000+00:00"
},
{
"name": "fred1",
"is_directory": true,
"size": null,
"last_modified": null,
"created": null
}
]
Now let’s go ahead and add it to our JWD:
$ lcli job workdir add --data-recursive fred1 --path fred 1027
$ lcli job workdir show --path fred 1027
[
{
"name": "file1",
"is_directory": false,
"size": "0",
"last_modified": "2022-09-09T18:41:35.668+00:00",
"created": "2022-09-09T18:41:20.000+00:00"
},
{
"name": "file2",
"is_directory": false,
"size": "0",
"last_modified": "2022-09-09T18:41:35.849+00:00",
"created": "2022-09-09T18:41:22.000+00:00"
},
{
"name": "fred1",
"is_directory": true,
"size": null,
"last_modified": null,
"created": null
},
{
"name": "file3",
"is_directory": false,
"size": "0",
"last_modified": "2022-09-09T18:41:36.033+00:00",
"created": "2022-09-09T18:41:23.000+00:00"
}
]