UNIX is an operating system originally developed at Bell Laboratories in the 1960s. Linux is a version of Unix; MacOS is derived from another version.
Linux is the dominate operating system for high-performance computing.
Graphical User Interfaces have been developed for Unix
Rivanna is the university’s primary resource for high-performance computation. It provides a platform for computationally-intensive research across disciplines.
Logging into a remote UNIX based system requires a client
Your options for the client depends on your operating system.
Command line access is based on the “SSH” or Secure Shell protocol
When off Grounds, you must be connected to the VPN
https://www.rc.virginia.edu/userinfo/rivanna/ood/overview/
Access through Netbadge (does not require VPN) at the portal
From the Dashboard go to the menu Clusters->Rivanna
FastX is a remote desktop application.
Provides a MATE desktop environment.
Available from a Web browser.
Menu
We recommend MobaXterm. Download the Home Edition, Installer Edition.
From the Sessions menu, start an SSH session. Using one of the particular names for a rivanna frontend (rivanna1.hpc.virginia.edu, etc.). as the host name.
If you prefer to type an ssh command, start a local terminal session and enter the same command as is used in the Mac OS or Linux terminal (the -Y option is the default).
MobaXterm includes an X server for graphics, a simple editor, and other useful tools in addition to ssh and sftp/scp functions.
Mac OSX and Linux ship with a Terminal app.
Open Finder and Go to Applications
Connect using
ssh -Y mst3k@rivanna.hpc.virginia.edu
Replace mst3k with your user ID.
Direct interaction with Unix is through a shell.
The shell is a program that interprets commands issued to the operating system.
bash (the Bourne Again Shell).zsh as its default.You will have a prompt which indicates that the shell is ready to accept commands.
bash-4.2$ (the number is the version)$
echo $shell
ls and LS are different.cmd -o --opts <filename>Let’s run our first command
pwd
/home/mst3k
Know where you are!
string<tab> causes bash to expand string as far as it has a unique name.clear or control-lA file stores some form of information.
Directories are collections of files
Both files and directories contain metadata
In UNIX all files and directories have a “path”
/.A path that starts with / is an absolute path.
Examples:
/
/home/mst3k
/home/mst3k/rivanna-cl
/home/mst3k/rivanna-cl/basic_commands
An absolute path will always lead to the location.
A tilde ~ stands for /home so alternatives to the above are
~mst3k
~mst3k/rivanna-cl
The current directory can be represented by a period (.) Thus if we are in
/home/mst3k/rivanna-cl/basic_commands we can type
gedit ./hello_world.sh
./hello_world.sh
The parent directory is symbolized by two periods (..).
gedit ../scripts/script.sh
Relative paths are defined by their relationship to the current directory. Be sure you are starting in the correct location!
ls
list files in current directory
Options. ls has many.
cp file1 file2
Copy file1 into file2
file2 exists-r copy files and subdirectories recursively.-n “noclobber”. Do not overwrite an existing file.-i ask for confirmation of overwriting.-p preserve metadata.-i and -n are present, the last one is used.
cp -r /share/resources/tutorials/rivanna-cl ./
Be sure to include a space between rivanna-cl and ./.
mv file1 file2
-i ask for confirmation before overwriting-n noclobber-u move only if file1 is newer than file2mv “moves” a file.
$pwd
$/home/mst3k/rivanna-cl/basic_commands
$mv SP_R1.list list_of_reads.txt
If not already in your basic_commands subdirectory, navigate there. Type after your prompt
cat >mynewfile
# This is a file.
Use an editor of your choice and type another line or two of text.
more mynewfile
ls
mv mynewfile the_file
cp the_file old_file
ls -l
rm file
(r_ e _m_ ove)
-r remove files and subdirectories recursively.-i ask for confirmation-f do not ask (overrides -i if it comes after it).Change to a tutorial directory. List the files that are in it.
cd rivanna-cl/basic_commands
ls -lh
You should see something like
total 245M
-rw-r--r-- 1 mst3k users 2.1K Jan 29 13:20intro_basic-unix.txt
-rwxr-xr-x 1 mst3k users 197M Jan 29 13:20list_of_reads.txt
-rw-r--r-- 1 mst3k users 46M Jan 29 13:20sequences.fasta
-rw-r--r-- 1 mst3k users 1.6M Jan 29 13:20 SP_R1.fastq
-rw-r--r-- 1 mst3k users 1.6M Jan 29 13:20 SP_R2.fastq
mkdir newdircd rivanna-cl/basic_commands
mkdir newdir
$ ls -lhtr
You can use absolute paths or relative paths for the new directory. Let’s do some examples! Please type these into your shell directly.
mkdir newcode
mkdir ~/newcode/build
cd newcode
mkdir ../oldcode
mkdir src
mkdir ../oldcode/src
rm can also remove a directory
rm -d somedir remove the directory (must be empty)rm -rf somedir remove directory, files, and subdirectories. Don’t ask for permission for each entry.rmdir somedir
rm -d somedirtestdir in your home directory.more filename
Displays file contents on the screen with paging.
In most implementations you can search a page in the forward direction with
/<pattern>
Search for next occurrence of
n
Strings of characters may be matched with wildcards.
BE CAREFUL when using wildcards with rm! Gone is gone! On some systems there may be backups, or maybe not, and if you are using a personal system you would need to set up backups and learn to retrieve files.
grep is used to filter a file or group of files, line by line, against a pattern (e.g., to print each line of a file which contains a match for
-i ignore the case of letters-v invert; show non-matching lines-r search recursively through current directory (omit filename)-c count linesThe PATTERN that grep matches may be a simple string, or it may be a special type of pattern called a regular expression. Regular expressions (or “regexes”) are beyond our scope here, but to use one with grep add the -e option, for extended, and enclose the regex in quotes.
A very common regex pattern is ^ which stands for “beginning of the line.” So we could search for the letter C at the beginning of the line with
grep -e "^C" *.f
See this site for an introduction to regular expressions.
~/rivanna-cl/basic_commands/sequences.fasta
how would you count the number of sequences it contains?
The basic documentation for a command can be read from the shell with theman(manual) command
man ls
Each executable has associated with it three I/O streams: standard input , standard error , and standard output . They are often represented as stdin , stderr , and stdout .
Normally these streams are attached to your console (your interactive shell). Most Unix commands read from standard input and/or write to standard output.
You can redirect standard input with <
mycode < params.txt
Redirect standard output with >
ls –l > filelist
Redirection overwrites anything already in the target file. Append with »
cat file1 >>bigfile
Redirection of standard error depends on the shell
Bash:
make >& make.out
Redirects both stdout and stderr to make.out
| is above the backslash on most US keyboards.Example
grep "@H-148:116" SP_R1.fastq | head
grep searches for the pattern in the file and head displays first 10 lines of the output.
Bash is a scripting language and as such has variables. Shell variables can be defined and then referenced with a leading $.
Important: when setting shell variables with =, do not put any spaces around the = symbol.
The value of a variable can be printed with echo.
Some variables are preset by the system.
Examples
echo $shell
An environment variable is a shell variable that describes some aspect of the user’s environment. These variables are conventionally written in all capitals.
Several environment variables are predefined.
Environment variables are still shell variables and can be printed with echo. But a special command printenv is available. It does not use the leading $.
echo $SHELL
printenv SHELL
printenv HOME
printenv USER
ls $HOME
printenv PATH
printenv PATH
module load gcc/9.2.0
printenv PATH
Environment variables can be set with the export command. This differs from = in that export sets the variable for the current shell and any “child” shells started by the current shell.
Example: In most cases, current working directory (.) is not in your default search path. To add it, type (for bash)
export PATH=$PATH:.
In this case it is essential to add the first $PATH or you will lose the default path set by the system.
If you forget to include the base $PATH when you modify it, you can find most Linux utilities in /usr/bin or /usr.
export PATH=.
ls
OOPS!
/usr/bin/ls
Any changes to PATH will disappear when you exit the shell in which you made them.
You will run some form of executable on the system. Executables are often called binaries, especially by Unix types and computer programmers. The terms are synonymous in most cases.
If the executable is in your search path you can simply type its name at the prompt.
gedit hello_world.sh
here gedit is the name of the binary. Its actual location is /usr/bin/gedit,but /usr/bin is in the default search path.
If it is not in your search path you must type the path to the executable (can be absolute or relative)
./hello_world.sh
Usually the current directory is not in your default search path for security reasons.
You now have the tools to manage your jobs from the command line and write your own scripts for your SLURM jobs. Please see our SLURM tutorial for more information.