For first time login, only the ssh protocol is supported. Other protocols such as X11 Forwading over ssh, sftp, and VNC will be discussed later in other tutorials.
The (S)ecure (SH)ell protocol opens up a pseudo terminal connection, which will give you an interactive shell in order to communicate with the cluster. There are several clients available:
(Note - this is by no means a full list of every client that you can use)
For the purposes of this tutorial we will be using a terminal emulator such as [OSX:Terminal Terminal], [Gnome Terminal], or [XTerm]. Begin by opening up your terminal and typing:
rlyon@deepthought:~$ ssh email@example.com
You will substitute your username for the fictitious one that is shown.
We also have a web-based solution, ondemand.
Once a connection is made you will receive a prompt to enter your password. Enter the password which has been given to you and if all is well, you will see this:
_ _ _ o-----------\____(_) | | | o------\__/ | | |__ ___ ___| |_ o----/ | | '_ \ / _ \/ __| __| o-----\____| | |_) | __/\__ \ |_ o---------/ |_|_.__/ \___||___/\__| WARNING: To protect the system from unauthorized use and to ensure that the system is functioning properly, activities on this system are monitored recorded and subject to audit. Use of this system is expressed consent to such monitoring and recording. Any unauthorized access or use of this system is prohibited and subject to criminal and civil penalties. +-----------------------------------------------+ | Current Standalone System Usage | +-----------------------------------------------+ | SERVER MEM Usage(%) CPU Usage(%) | | colin 10.96 70.92 | | ford 8.90 37.00 | | marvin 9.11 39.99 | | petunia 3.65 0.01 | | seraph 7.41 66.70 | | slartibartfast 3.84 2.08 | | hactar 40.38 67.53 | | trillian 11.84 37.47 | | arthur 2.93 0.00 | | whale 7.82 68.85 | | zaphod 10.28 68.67 | +-----------------------------------------------+ SYSTEM: zaphod.ibest.uidaho.edu CORES: 16 MEMORY: 32167 MB SUMMARY: (collected Mon Nov 7 09:15:01 PST 2016) * CPU Usage (total average) = 0.00% * Memory used (real) = 1823 MB * Memory free (cache) = 31111 MB * Swap in use = 52 MB * Load average = 0.00 0.02 0.00 QUESTIONS: Submit all questions, requests, and system issues to: firstname.lastname@example.org
You have now logged in.
We have two main types of servers, standalone servers and cluster servers. The difference between these two is how you go about running your jobs.
On a standalone server, you simply log in and can run your jobs directly. For example:
benji@trillian ~ $ cd /mnt/lfs2/benji benji@trillian ~ $ module load R benji@trillian ~ $ R R version 3.1.3 (2015-03-09) -- "Smooth Sidewalk" Copyright (C) 2015 The R Foundation for Statistical Computing Platform: x86_64-unknown-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > rnorm(20)  -0.31292512 -0.04088035 -0.45084361 -0.15509451 -1.31036346 1.04192160  -1.54716041 0.75207358 0.37357678 -0.37222896 1.31458656 -1.46211410  0.64556641 0.61382511 -1.24606528 -0.35138105 -1.55991190 -0.31917710  -1.00967966 -0.86483148 > quit(save="no") benji@trillian ~ $
We currently maintain 12 generally available standalone servers. The servers you have access to will depend on the type of account you have:
|Standard||(Used for workshops/classes)|
For general compute needs the servers marvin, trillian, and zaphod are where you should start - the login screen will tell you which is the least busy. In the above login example, note that marvin is pretty busy and would not be a good server to start more jobs on. If your project requires lots of memory (RAM), slartibartfast and ford may be used. Arthur and colin are newer servers, and have yet more RAM. Whale has an obscene amount of RAM (1TB), please use only for your jobs that really need that kind of brute force. Note that your account type may limit which servers you have access to. For a more details on server capabilities see this page. For a general walk-through see this workshop documentation.
The CRC maintains a ~2500+ core compute cluster, currently managed by fortyfour.ibest.uidaho.edu. Note: Free account users can submit cluster jobs from the standalone servers. If your jobs can run in replicates or be parallelized (most can), the cluster will serve you well. When running your jobs on the cluster, you must use the scheduler to add your jobs to the cluster queue. For example:
benji@fortyfour ~ $ cd /mnt/lfs2/benji benji@fortyfour ~ $ cat my_job_script.slurm #!/bin/bash cd $SLURM_SUBMIT_DIR echo "running on " hostname source /usr/modules/init/bash module load R Rscript -e "rnorm(10)" echo "finished" benji@fortyfour ~ $ sbatch my_job_script.slurm Submitted batch job 726 benji@fortyfour ~ $ cat slurm-726.out running on n090  0.2058277 1.3586386 0.2363083 0.1044323 0.3514225 1.9576543 0.1503541  0.4841014 0.6014489 1.0225770 finished benji@fortyfour ~ $
I put the commands I wanted to run (generating random numbers) into a simple script named my_job_script.slurm, and then submitted that script the the queue using the
sbatch command. The output of the job (what would normally print to screen) is returned in a file which is by default named slurm-XXX.out. There are many examples of various submission script options here, and you may also find the cluster computing workshop examples helpful.