---
title: "The command line"
subtitle: "Controlling local and remote machines"
---

# Shell

* A shell exposes an operating system to a human user or other programs
* On most Linux systems, the default shell is _Bash_ (_PowerShell_ on Windows; _Zsh_ on MacOs)
* Different shells come with different syntax and characteristics

## Built-in commands

* Allow to operate a computer in a text-only way
* Built-in commands like `cd`, `ls`, `cp`, `mv`, and `rm` provide all features of a graphical file manager (e.g. Finder)
* Type `info <command>` for more information

## Coreutils - GNU core utilities

* Basic file, shell and text manipulation utilities that extend the shell-builtins
* Some useful commands are [`sed`](https://www.gnu.org/savannah-checkouts/gnu/sed/manual/sed.html), [`grep`](https://www.gnu.org/savannah-checkouts/gnu/grep/manual/grep.html), [`awk`](https://www.gnu.org/savannah-checkouts/gnu/gawk/manual/gawk.html), [`head`](https://www.gnu.org/savannah-checkouts/gnu/coreutils/manual/html_node/head-invocation.html), [`tail`](https://www.gnu.org/savannah-checkouts/gnu/coreutils/manual/html_node/tail-invocation.html), ...
* [Standard output](https://www.man7.org/linux/man-pages/man3/stdin.3.html) can be _piped_ (`|`) to another command
  ```{raw}
  ps aux | grep $USER
  ```
* Type `man <command>` to read the manual for a command

## Text editors

* Editors (e.g. [vim](https://www.vim.org/), [emacs](https://www.gnu.org/software/emacs/), or [nano](https://duckduckgo.com/?q=nano+editor&t=osx)) allow you to modify text files from the command line
  ```{.sh}
  vim test.txt
  ```
* Basic `vim` usage: hit {{< kbd i >}} to activate _insert_ mode. Type something and hit {{< kbd Esc >}} to fall back into _normal_ mode. Save and close the file using {{< kbd :wq >}}.

## Scripts

* You can bundle commands in a shell script, e.g.:
  ```{.sh filename=script.sh}
  hostname
  date
  ```
  Running the script will execute the commands sequentially:
  ```{raw}
  bash script.sh
  ```

  ```{raw}
  levante4.lvt.dkrz.de
  Mon Mar 25 16:49:06 CET 2024
  ```

## Take home messages

* The command line is a text interface for interacting with a computer
* Built-in commands and core utilities are available on (almost) all machines
* Sequences of commands can be stored and executed as shell scripts

# Hands-on session {.handson}

1. Open the command line (_Terminal_^[More difficult for Windows users])
1. Create a shell script using an editor of your choice
1. Run the script and check the output

# Levante
![[Levante](https://docs.dkrz.de/doc/levante/index.html) is the current _supercomputer_ at DKRZ (since 2022)](static/levante-as-box.jpg)

## Configuration

High-performance computers consist of various components

:::: {.columns}

::: {.column width="60%"}
* Login (login nodes)
* Computing (CPU & GPU partitions)
* Storage (disks, tape, object store)
* Services (GitLab, JupyterHub, ...)
:::

::: {.column width="40%"}
![](static/levante-in-hall.jpg)
:::

::::

## Working on Levante

Different ways to work on a cluster depending on the task:

* Housekeeping/compiling (login nodes)
* Interactive sessions (compute nodes, `salloc`)
* Batch scripts (compute nodes, `sbatch`)
* External services (e.g. _Jupyterhub_)

## SSH

* Connect to the login nodes via secure shell (`ssh`)
  ```{raw}
  ssh <YOUR_USERNAME>@levante.dkrz.de
  ```
* Now you have access to the command line **on levante**

## Public keys

![](static/public-key.png)

## Generating SSH keys

* Generate a private / public SSH key pair on your local system
  ```bash
  ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_levante
  ```
* Upload the public (`.pub`) key to your DKRZ profile ([how to](https://docs.dkrz.de/doc/levante/access-and-environment.html#public-key-authentication))
* Use your private SSH key to authenticate when connecting to levante
  ```bash
  ssh -i ~/.ssh/id_ed25519_levante <YOUR_USERNAME>@levante.dkrz.de
  ```

## Configuring SSH

* Create a config file for convenience
  ```{.sshconfig filename=~/.ssh/config}
  Host levante # the name you use in the shell
     Hostname levante.dkrz.de # real host name of the system
     User <YOUR_USERNAME> # your user id
     IdentityFile ~/.ssh/id_ed25519_levante # your private key
  ```
  (call `man ssh_config` for more info)

* The connection to levante simplifies to
  ```{raw}
  ssh levante
  ```

* The configuration is used by all tools using an ssh connection _(more than you think)_

## Remote file transfer {auto-animate=true}

* There are various tools to transfer data between your local machine and a remote server (e.g. [`scp`](https://linux.die.net/man/1/scp), [`rsync`](https://linux.die.net/man/1/rsync), [`uftp`](https://uftp-multicast.sourceforge.net))
* `rsync` is a powerful option available on most machines
  ```{raw}
  rsync <YOUR_USERNAME>@levante.dkrz.de:file_in_home .
  ```

## Remote file transfer {auto-animate=true visibility="uncounted"}

* There are various tools to transfer data between your local machine and a remote server (e.g. [`scp`](https://linux.die.net/man/1/scp), [`rsync`](https://linux.die.net/man/1/rsync), [`uftp`](https://uftp-multicast.sourceforge.net))
* `rsync` is a powerful option available on most machines
  ```{raw}
  rsync levante:file_in_home .
  ```

## Three parts of the file system

| `/home` | `/work` | `/scratch` |
| --- | --- | --- |
|keep scripts | store output | temporary stuff |
| small | big | big |
| SSD | HDD | HDD |
|backup | no backup | deleted after 2 weeks |

::: {.fragment .fade-up}
> Anything not saved will be lost --- _Nintendo_
:::

## Compute nodes

* Jobs that require more resources or run longer can be submitted to the job scheduler ([SLURM](https://docs.dkrz.de/doc/levante/running-jobs/index.html))
  ```{raw}
  sbatch --account=<PROJ_NUMBER> --partition=compute script.sh
  ```
* There are dedicated [partitions](https://docs.dkrz.de/doc/levante/running-jobs/partitions-and-limits.html) for different use cases (e.g., `compute`, `shared`, `interactive`, ...)
* Try to use the smallest amount of resources (e.g. `shared` with `--mem=50G`)

## Take home messages

* High-performance computers like Levante are technically a cluster of numerous _normal_ computers that share the same hard disks
* Using a secure shell (`ssh`) you can log into Levante and (mostly) use it like any other machine
* Computationally demanding or long-lasting jobs have to be submitted to the job queue

# Hands-on session {.handson}

1. Create an SSH key pair on your local machine
1. Upload the **public** key to luv.dkrz.de
1. Set up an entry for levante in your SSH config file
1. Copy & run your shell script to levante (using bash and sbatch)

# Shotgun buffet

. . .

> I am just gonna throw a bunch of stuff at you. Take what you might find interesting. --- _Scott Chacon_

## IDEs

* **I**ntegrated **D**evelopment **E**nvironments are text editors on steroids
* Specific functionality for your programming language (tab completion, jump to definition, scope-aware renaming, debugger, auto documentation, ...)
* VSCode, PyCharm, Jupyterhub (in a way), vim + emacs (with plugins), ...

## tmux

`tmux` enables to create and attach to terminal sessions

* Crate a new session
  ```{raw}
  tmux
  ```
* Attach to to running sessions
  ```{raw}
  tmux attach -t 0
  ```
* You need to login to the same login node (e.g., `levante3`^[SSH config])

## X-forwarding

forward graphical user interfaces (GUIs) from the server

* Install an X-server on your local machine (e.g. XQuartz)
* Pass the `-X`^[Mac users might want to use `-Y`, or MacOS will quit the forwarding after about 20 min.] option to your `ssh` command^[or check the SSH config option]
  ```{raw}
  ssh -X levante
  ```

## YubiKey

* SSH keys at DKRZ expire after a month
* When using a hardware authenticators (e.g. YubiKey) the key is valid for a year

# References

* [Software Carpentry: The Unix Shell](https://swcarpentry.github.io/shell-novice/)
* [Software Carpentry Incubator: Unix Shell Extras (incl. ssh)](https://carpentries-incubator.github.io/shell-extras/)