Robbie Persistent Disk
Robbie CLI commands allow you to manipulate files in the user's persistent-disk
before or after you run a job. The persistent-disk
will be automatically mounted at /home/job-user/persistent-disk
on the remote machine when you run a job.
Supported commands:
ls
- list files and folderscp
- copy files to and from the local machine to thepersistent-disk
or between locations on thepersistent-disk
mv
- move files to and from the local machine to thepersistent-disk
or between locations on thepersistent-disk
mkdir
- create directories/folders on thepersistent-disk
.rm
- remove files or folders on thepersistent-disk
Differences with corresponding Unix/Linux commands
While these commands may look similar to Unix/Linux there are differences in the syntax and behavior. Where there are behavior differences, it will be noted as a reference. Additionally, the Robbie CLI does not have the concept of a current working directory (e.g.`cd`
command) on the persistent-disk
All commands that reference (using pd://
) the persistent-disk
files and folders require absolute path names. Relative paths are not supported. When running these commands, you identify files and folders on the persistent disk using the pd://
prefix. Files or folders without the pd://
prefix are assumed to be on your local machine and DO support absolute and relative path names.
Underlying implementation
The current persistent-disk
is implemented on top of Amazon S3. The Robbie CLI commands are meant to hide the complexity of S3 and implement permissions so users can only view their files. In terms of the Robbie system, your persistent disk will be located at: s3://positron-dev-workspaces/{email}/persistent-job-disk
.
All commands automatically insert the correct s3 path into commands such that the persistent-disk
root (s3://) is always: s3://positron-dev-workspaces/{email}/persistent-job-disk
List files/folders
List all files/folders in root directory
robbie pd ls
You should see all files and folders in the user's /home/job-user/persistent-disk
directory.
Names denoted with a PRE
are folders.
List all files/folders in the test
directory (note the following "/", that is required)
test
directory (note the following "/", that is required)robbie pd ls pd://test/
List all files/folders in the test
directory and below recursively (note the following "/", that is required)
test
directory and below recursively (note the following "/", that is required)robbie pd ls pd://test/ --recursive
List all files/folders in that match the pattern test
directory (note there is no "/")
test
directory (note there is no "/")robbie pd ls pd://test
Try to list a non-existing folder
robbie pd ls pd://xyz
Expected Result: Nothing will be displayed
Folders
Create a folder
robbie pd mkdir pd://test
You should see something like this:
{ "ETag": ""d41d8cd98f00b204e9800998ecf8427e"", "ChecksumCRC32": "AAAAAA==", "ChecksumType": "FULL_OBJECT", "ServerSideEncryption": "AES256" }
If you attempt to create an existing folder, it just returns the above.
Copying files from local machine to persistent-disk
persistent-disk
Copy a local file to the persistent-disk
root directory
persistent-disk
root directoryrobbie pd cp test12.py pd://test12.py
— or --
Copy a local file to the persistent-disk
root directory
persistent-disk
root directoryrobbie pd cp testing.py pd://
Recursively copy a local file to the persistent-disk
persistent-disk
robbie pd cp myDir pd://mybucket/ --recursive
example, if you had a local directory called images
filled with .jpg file, you could run
robbie pd cp ./images pd://images --recursive
and it would create the images folder on the pd and upload all the images
Copying files from persistent-disk
to local machine
persistent-disk
to local machineCopy a file in the persistent-disk
root directory to the local CWD
persistent-disk
root directory to the local CWDrobbie pd cp pd://testing.py ./test456.py
-- other option --
Copy a file in the persistent-disk
root directory to the local CWD
persistent-disk
root directory to the local CWDrobbie pd cp pd://testing.py .
Recursive copy files from bucket/prefix to local machine
robbie pd cp pd://mybucket . --recursive
copying files between folders
robbie pd cp pd://images pd://test --recursive
Moving files same semantic as cp
cp
Recursively move files between folders on pd
pd
robbie pd mv pd://images pd://test --recursive
Deleting files/folders
Delete a file in a folder
robbie pd rm pd://images/Abyssinian_1.jpg
Recursively removing files in a folder and the folder (equivalent to rm -rf <folder>
)
rm -rf <folder>
)robbie pd rm pd://test --recursive
Recursively removing files in a folder and the folder (equivalent to rm -rf <folder>
) but exclude ".mat" files
rm -rf <folder>
) but exclude ".mat" filesrobbie pd rm pd://images --recursive --exclude "*.mat"
Last updated
Was this helpful?