Working with the Persistant Storage Area

Persistant Storage Basic Commands

add : Add a file to the persistant storage area.
copy : Copies a file from a location in the persistant storage area to another area within the persisant storage area.
delete : Deletes a file / folder within the persistant storage area.
get : View or save a file from the persistant storage area to the local disk.
makedir: Create a directory within the persistant storage area.
move : Move a folder / directory within the persistant storage area to another area within the persistant storage area.
show : Shows the available files / directories within the persistant storage area.

Ex 1) Adding A File And Using It In a Job

Let’s add a file to the persistant storage area and use that file as an input-file for a job.

# First let's take a look at the file we are adding.
$cat ~/bin/hello2.py
print('hello world!')

# Let's add the file.
$lcli data add --file ~/bin/hello2.py

# Let's take a look at the file added.
$lcli data show hello2.py
{
    "length": "22",
    "last-modified": "Mon, 23 May 2022 20:48:35 +0000",
    "date-created": "Mon, 23 May 2022 20:46:15 +0000"
}

# Let's run a job with this file.
$lcli job run -n "Job with persistant storage file" --command "python3 hello2.py" --input-data hello2.py --image test/pythonEnv5
{
    "id": 653,
    "name": "Job with persistant storage file",
    "status": "created",
    "qos": "high",
    "command_line": "python3 hello2.py",
    "image": "test/pythonEnv5",
    "resources": {
        "core_count": 2,
        "gpu_count": null,
        "memory": 4,
        "gpu": null,
        "scratch": null,
        "mpi": false
    },
    "max_run_time": 259200,
    "input_files": [
        {
            "id": 1411,
            "name": "hello2.py",
            "source_type": "data",
            "source": "hello2.py",
            "cache": false,
            "upload_complete": false,
            "chunks_received": []
        }
    ],
    "created_at": "2022-05-23T20:50:03.793Z",
    "updated_at": "2022-05-23T20:50:03.793Z"
}

# Finally let's take a look at the output
$ lcli job output get --view --file stdout.txt 653
hello world!

Ex 2) Storing Output File from Job into Persistant Storage

Let’s take a look at an example where the output file from a finished job is stored within the persistant storage area. There are many reasons why this could be useful; for example, the output file of one job could be used as an input file for the next job.

# First let's take a look at the file we are adding.
$cat ~/bin/hello2.py
print('hello world!')

# Let's run a job and direct the output of that job to be stored within the persistant storage area as: "helloWorld.txt"
$lcli job run -n "output to persistant storage" --image test/pythonDockerImage --command "python3 hello2.py" --input-file ~/bin/hello2.py --output stdout.txt:helloWorld.txt
{
    "id": 48435,
    "name": "output to persistant storage",
    "status": "created",
    "qos": "high",
    "command_line": "python3 hello2.py",
    "image": "test/pythonDockerImage",
    "resources": {
        "core_count": 2,
        "gpu_count": null,
        "memory": 4,
        "gpu": null,
        "scratch": null
    },
    "max_run_time": 259200,
    "input_files": [
        {
            "id": 11322,
            "name": "hello2.py",
            "source_type": "file",
            "source": "/home/rap/bin/hello2.py",
            "cache": false,
            "upload_complete": true,
            "chunks_received": [
                [
                    1,
                    22
                ]
            ]
        }
    ],
    "output_files": [
        {
            "name": "stdout.txt",
            "size": null,
            "available": false
        }
    ],
    "created_at": "2022-05-23T20:59:42.141Z",
    "updated_at": "2022-05-23T20:59:42.141Z"
}

# After the job has finished, we can take a look at that file from the persistant storage area -- either to save it locally or to simply view it. Let's view it.
$lcli data get --view --file helloWorld.txt
hello world!