Robbie Persistent Disk

Robbie CLI commands allow you to manipulate files in the user's persistent-disk before or after you run a job. The persistent-disk will be automatically mounted at /home/job-user/persistent-disk on the remote machine when you run a job.

Supported commands:

• ls - list files and folders

• cp - copy files to and from the local machine to the persistent-disk or between locations on the persistent-disk

• mv - move files to and from the local machine to the persistent-disk or between locations on the persistent-disk

• mkdir - create directories/folders on the persistent-disk.

• rm - remove files or folders on the persistent-disk

Differences with corresponding Unix/Linux commands

While these commands may look similar to Unix/Linux there are differences in the syntax and behavior. Where there are behavior differences, it will be noted as a reference. Additionally, the Robbie CLI does not have the concept of a current working directory (e.g.`cd` command) on the persistent-disk All commands that reference (using pd://) the persistent-disk files and folders require absolute path names. Relative paths are not supported. When running these commands, you identify files and folders on the persistent disk using the pd:// prefix. Files or folders without the pd://prefix are assumed to be on your local machine and DO support absolute and relative path names.

Underlying implementation

The current persistent-disk is implemented on top of Amazon S3. The Robbie CLI commands are meant to hide the complexity of S3 and implement permissions so users can only view their files. In terms of the Robbie system, your persistent disk will be located at: s3://positron-dev-workspaces/{email}/persistent-job-disk.

All commands automatically insert the correct s3 path into commands such that the persistent-disk root (s3://) is always: s3://positron-dev-workspaces/{email}/persistent-job-disk

List files/folders

List all files/folders in root directory

robbie pd ls

You should see all files and folders in the user's /home/job-user/persistent-disk directory.

Names denoted with a PRE are folders.

List all files/folders in the test directory (note the following "/", that is required)

robbie pd ls pd://test/

List all files/folders in the test directory and below recursively (note the following "/", that is required)

robbie pd ls pd://test/ --recursive

List all files/folders in that match the pattern test directory (note there is no "/")

robbie pd ls pd://test

Try to list a non-existing folder

robbie pd ls pd://xyz

Expected Result: Nothing will be displayed

Folders

Create a folder

robbie pd mkdir pd://test

You should see something like this:

{ "ETag": ""d41d8cd98f00b204e9800998ecf8427e"", "ChecksumCRC32": "AAAAAA==", "ChecksumType": "FULL_OBJECT", "ServerSideEncryption": "AES256" }

If you attempt to create an existing folder, it just returns the above.

Copying files from local machine to persistent-disk

Copy a local file to the persistent-disk root directory

robbie pd cp test12.py pd://test12.py

— or --

Copy a local file to the persistent-disk root directory

robbie pd cp testing.py pd://

Recursively copy a local file to the persistent-disk

robbie pd cp myDir pd://mybucket/ --recursive 

example, if you had a local directory called images filled with .jpg file, you could run

robbie pd cp ./images pd://images --recursive and it would create the images folder on the pd and upload all the images

Copying files from persistent-disk to local machine

Copy a file in the persistent-disk root directory to the local CWD

robbie pd cp pd://testing.py ./test456.py

-- other option --

Copy a file in the persistent-disk root directory to the local CWD

robbie pd cp pd://testing.py .

Recursive copy files from bucket/prefix to local machine

robbie pd cp pd://mybucket . --recursive

copying files between folders

robbie pd cp pd://images pd://test --recursive

Moving files same semantic as cp

Recursively move files between folders on pd

robbie pd mv pd://images pd://test --recursive

Deleting files/folders

Delete a file in a folder

robbie pd rm pd://images/Abyssinian_1.jpg

Recursively removing files in a folder and the folder (equivalent to rm -rf <folder>)

robbie pd rm pd://test --recursive

Recursively removing files in a folder and the folder (equivalent to rm -rf <folder>) but exclude ".mat" files

robbie pd rm pd://images --recursive --exclude "*.mat"