UNIX
Getting Started with Unix and the Command Line
The Command Line is a way of interacting with your computer using text-based commands. These trainings will orient you to the Unix command line language, which is the language that is easiest to use for conducting research with FreeSurfer and the BYU Supercomputer.
How to Access the Command Line on Your Machine
For Mac users:
- Use Command+Space to search your computer
- Type "Terminal" and press enter
- The app that begins running is called Terminal (affectionately referred to as 'the Terminal,' whether you are on a Mac or PC). You can execute Linux commands in this Terminal
For PC users:
- Go into your computer settings and search for "Turn Windows features on or off"
- In the settings window that opens, make sure that "Windows Subsystem for Linux" is checked. If it is not, check it, then click "OK"
- You may need to reboot your computer at this point
- From the Microsoft Store, download the app "Ubuntu"
- Launch Ubuntu, and this terminal is The Terminal! You can now execute Linux commands when running this app
Unix Training Modules
Unix Additional Resources
Unix cheat sheet
Learn Enough Unix tutorial
Korflab Tutorial
Read the first 8 chapters of "Unix for Dummies" (located in the lab)
Some Other Helpful UNIX Stuff
Keeping a process running after ending an ssh session
- ssh into the remote machine
- start tmux by typing tmux into the shell
- start the process you want inside the started tmux session
- leave/detach the tmux session by typing Ctrl+b and then d
If you want to have multiple sessions running side-by-side, you should name each session using Ctrl+b and $. You can get a list of the currently running sessions using tmux list-sessions, now attach to a running session with command tmux attach-session -t
How does 'scp' differ from 'rsync'?
The major difference between these tools is how they copy files.
scp basically reads the source file and writes it to the destination. It performs a plain linear copy, locally, or over a network.
rsync also copies files locally or over a network. But it employs a special delta transfer algorithm and a few optimizations to make the operation a lot faster. Consider the call.
rsync A host:B
- rsync will check files sizes and modification timestamps of both A and B, and skip any further processing if they match.
- If the destination file B already exists, the delta transfer algorithm will make sure only differences between A and B are sent over the wire.
- rsync will write data to a temporary file T, and then replace the destination file B with T to make the update look “atomic” to processes that might be using B.
Another difference between them concerns invocation. rsync has a plethora of command line options, allowing the user to fine tune its behavior. It supports complex filter rules, runs in batch mode, daemon mode, etc. scp has only a few switches.
In summary, use scp for your day to day tasks. Commands that you type once in a while on your interactive shell. It’s simpler to use, and in those cases rsync optimizations won’t help much.
For recurring tasks, like cron jobs, use rsync. As mentioned, on multiple invocations it will take advantage of data already transferred, performing very quickly and saving on resources. It is an excellent tool to keep two directories synchronized over a network.
Also, when dealing with large files, use rsync with the -P option. If the transfer is interrupted, you can resume it where it stopped by reissuing the command.
see more at https://linux.die.net/man/1/rsync
tar tips- To see tar progress with one line command: tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '{print $1}') | gzip > big-files.tar.gz