UNIX is an operating system originally developed at Bell Laboratories in the 1960s. Linux is a version of Unix; MacOS is derived from another version.
Linux is the dominate operating system for high-performance computing.
Graphical User Interfaces have been developed for Unix
Rivanna is the university’s primary resource for high-performance computation. It provides a platform for computationally-intensive research across disciplines.
Logging into a remote UNIX based system requires a client
Your options for the client depends on your operating system.
Command line access is based on the “SSH” or Secure Shell protocol
When off Grounds, you must be connected to the VPN
Access through Netbadge (does not require VPN) at the portal
From the Dashboard go to the menu Clusters->Rivanna
FastX is a remote desktop application.
Provides a MATE desktop environment.
Available from a Web browser.
We recommend MobaXterm. Download the Home Edition, Installer Edition.
From the Sessions menu, start an SSH session. Using one of the particular names for a rivanna frontend (rivanna1.hpc.virginia.edu, etc.). as the host name.
If you prefer to type an ssh command, start a local terminal session and enter the same command as is used in the Mac OS or Linux terminal (the -Y option is the default).
MobaXterm includes an X server for graphics, a simple editor, and other useful tools in addition to ssh and sftp/scp functions.
Mac OSX and Linux ship with a Terminal app.
Open Finder and Go to Applications
ssh -Y email@example.com
mst3k with your user ID.
Direct interaction with Unix is through a shell.
The shell is a program that interprets commands issued to the operating system.
bash(the Bourne Again Shell).
zshas its default.
You will have a prompt which indicates that the shell is ready to accept commands.
bash-4.2$(the number is the version)
cmd -o --opts <filename>
Let’s run our first command
Know where you are!
string<tab>causes bash to expand
stringas far as it has a unique name.
A file stores some form of information.
Directories are collections of files
Both files and directories contain metadata
In UNIX all files and directories have a “path”
A path that starts with
/ is an absolute path.
/ /home/mst3k /home/mst3k/rivanna-cl /home/mst3k/rivanna-cl/basic_commands
An absolute path will always lead to the location.
~ stands for
/home so alternatives to the above are
The current directory can be represented by a period (.) Thus if we are in
/home/mst3k/rivanna-cl/basic_commands we can type
gedit ./hello_world.sh ./hello_world.sh
The parent directory is symbolized by two periods (..).
Relative paths are defined by their relationship to the current directory. Be sure you are starting in the correct location!
list files in current directory
ls has many.
cp file1 file2Copy
-rcopy files and subdirectories recursively.
-n“noclobber”. Do not overwrite an existing file.
-iask for confirmation of overwriting.
-nare present, the last one is used.
cp -r /share/resources/tutorials/rivanna-cl ./
Be sure to include a space between
mv file1 file2
-iask for confirmation before overwriting
-umove only if file1 is newer than file2
mv“moves” a file.
$pwd $/home/mst3k/rivanna-cl/basic_commands $mv SP_R1.list list_of_reads.txt
If not already in your basic_commands subdirectory, navigate there. Type after your prompt
cat >mynewfile # This is a file.
Use an editor of your choice and type another line or two of text.
more mynewfile ls mv mynewfile the_file cp the_file old_file ls -l
rm file(r_ e _m_ ove)
-rremove files and subdirectories recursively.
-iask for confirmation
-fdo not ask (overrides -i if it comes after it).
Change to a tutorial directory. List the files that are in it.
cd rivanna-cl/basic_commands ls -lh
You should see something like
total 245M -rw-r--r-- 1 mst3k users 2.1K Jan 29 13:20intro_basic-unix.txt -rwxr-xr-x 1 mst3k users 197M Jan 29 13:20list_of_reads.txt -rw-r--r-- 1 mst3k users 46M Jan 29 13:20sequences.fasta -rw-r--r-- 1 mst3k users 1.6M Jan 29 13:20 SP_R1.fastq -rw-r--r-- 1 mst3k users 1.6M Jan 29 13:20 SP_R2.fastq
cd rivanna-cl/basic_commands mkdir newdir $ ls -lhtr
You can use absolute paths or relative paths for the new directory. Let’s do some examples! Please type these into your shell directly.
mkdir newcode mkdir ~/newcode/build cd newcode mkdir ../oldcode mkdir src mkdir ../oldcode/src
rmcan also remove a directory
rm -d somedirremove the directory (must be empty)
rm -rf somedirremove directory, files, and subdirectories. Don’t ask for permission for each entry.
rm -d somedir
testdirin your home directory.
Displays file contents on the screen with paging.
In most implementations you can search a page in the forward direction with
Search for next occurrence of
Strings of characters may be matched with wildcards.
BE CAREFUL when using wildcards with
rm! Gone is gone! On some systems there may be backups, or maybe not, and if you are using a personal system you would need to set up backups and learn to retrieve files.
grep is used to filter a file or group of files, line by line, against a pattern (e.g., to print each line of a file which contains a match for
-iignore the case of letters
-vinvert; show non-matching lines
-rsearch recursively through current directory (omit filename)
The PATTERN that grep matches may be a simple string, or it may be a special type of pattern called a regular expression. Regular expressions (or “regexes”) are beyond our scope here, but to use one with
grep add the
-e option, for extended, and enclose the regex in quotes.
A very common regex pattern is
^ which stands for “beginning of the line.” So we could search for the letter
C at the beginning of the line with
grep -e "^C" *.f
See this site for an introduction to regular expressions.
how would you count the number of sequences it contains?
The basic documentation for a command can be read from the shell with theman(manual) command
Each executable has associated with it three I/O streams: standard input , standard error , and standard output . They are often represented as stdin , stderr , and stdout .
Normally these streams are attached to your console (your interactive shell). Most Unix commands read from standard input and/or write to standard output.
You can redirect standard input with <
mycode < params.txt
Redirect standard output with >
ls –l > filelist
Redirection overwrites anything already in the target file. Append with »
cat file1 >>bigfile
Redirection of standard error depends on the shell
make >& make.out
Redirects both stdout and stderr to make.out
|is above the backslash on most US keyboards.
grep "@H-148:116" SP_R1.fastq | head
grep searches for the pattern in the file and
head displays first 10 lines of the output.
Bash is a scripting language and as such has variables. Shell variables can be defined and then referenced with a leading
Important: when setting shell variables with
=, do not put any spaces around the
The value of a variable can be printed with
Some variables are preset by the system.
An environment variable is a shell variable that describes some aspect of the user’s environment. These variables are conventionally written in all capitals.
Several environment variables are predefined.
Environment variables are still shell variables and can be printed with
echo. But a special command
printenv is available. It does not use the leading
echo $SHELL printenv SHELL printenv HOME printenv USER ls $HOME
printenv PATH module load gcc/9.2.0 printenv PATH
Environment variables can be set with the
export command. This differs from
= in that
export sets the variable for the current shell and any “child” shells started by the current shell.
Example: In most cases, current working directory (.) is not in your default search path. To add it, type (for bash)
In this case it is essential to add the first $PATH or you will lose the default path set by the system.
If you forget to include the base $PATH when you modify it, you can find most Linux utilities in /usr/bin or /usr.
export PATH=. ls
Any changes to PATH will disappear when you exit the shell in which you made them.
You will run some form of executable on the system. Executables are often called binaries, especially by Unix types and computer programmers. The terms are synonymous in most cases.
If the executable is in your search path you can simply type its name at the prompt.
gedit is the name of the binary. Its actual location is
/usr/bin is in the default search path.
If it is not in your search path you must type the path to the executable (can be absolute or relative)
Usually the current directory is not in your default search path for security reasons.
You now have the tools to manage your jobs from the command line and write your own scripts for your SLURM jobs. Please see our SLURM tutorial for more information.