Skip to content
Snippets Groups Projects

Update "command line" lecture

Merged Lukas Kluft requested to merge command-line into main
All threads resolved!
1 file
+ 23
38
Compare changes
  • Side-by-side
  • Inline
---
---
title: "The command line"
title: "The command line"
subtitle: "Controlling local and remote machines"
subtitle: "Controlling local and remote machines"
author: "Lukas Kluft and Florian Ziemen"
---
---
# Shell
# Shell
* A shell exposes an operating system to a human user or other programs
* A shell exposes an operating system to a human user or other programs
* On most Linux systems the default shell is _Bash_ (_PowerShell_ on Windows; _Zsh_ on MacOs)
* On most Linux systems, the default shell is _Bash_ (_PowerShell_ on Windows; _Zsh_ on MacOs)
* Different shells come with different syntax and characteristics
* Different shells come with different syntax and characteristics
## Built-in commands
## Built-in commands
@@ -32,7+31,7 @@
@@ -32,7+31,7 @@
```{.sh}
```{.sh}
vim test.txt
vim test.txt
```
```
* Basic usage: hit {{< kbd i >}} to activate _insert_ mode. Type something and hit {{< kbd Esc >}} to fall back into _normal_ mode. Save and close the file using {{< kbd :wq >}}.
* Basic usage: hit {{< kbd i >}} to activate _insert_ mode. Type something and hit {{< kbd Esc >}} to fall back into _normal_ mode. Save and close the file using {{< kbd :wq >}}.
## Scripts
## Scripts
@@ -53,7+52,7 @@
@@ -53,7+52,7 @@
## Take home messages
## Take home messages
* The command line is a text interface to interact with a computer
* The command line is a text interface for interacting with a computer
* Built-in commands and core utilities are available on (almost) all machines
* Built-in commands and core utilities are available on (almost) all machines
* Sequences of commands can be stored and executed as shell scripts
* Sequences of commands can be stored and executed as shell scripts
# Hands-on session {.handson}
# Hands-on session {.handson}
1. Open the command line (_Terminal_)
1. Open the command line (_Terminal_^[More difficult for Windows users])
1. Create a shell script using an editor of your choice
1. Create a shell script using an editor of your choice
1. Run the script and check the output
1. Run the script and check the output
@@ -68,16 +67,7 @@ author: "Lukas Kluft and Florian Ziemen"
@@ -68,16 +67,7 @@ author: "Lukas Kluft and Florian Ziemen"
## Configuration
## Configuration
::: {.r-stack}
::: {.fragment .fade-out fragment-index=1}
Supercomputers consist of various components
:::
::: {.fragment .fade-in fragment-index=1}
High-performance computers consist of various components
High-performance computers consist of various components
:::
:::
. . .
:::: {.columns}
:::: {.columns}
@@ -99,7 +89,8 @@ High-performance computers consist of various components
@@ -99,7 +89,8 @@ High-performance computers consist of various components
Different ways to work on a cluster depending on the task:
Different ways to work on a cluster depending on the task:
* Housekeeping/compiling (login nodes)
* Housekeeping/compiling (login nodes)
* Interactive sessions/scripts on a compute node (`salloc`/`sbatch`)
* Interactive sessions (compute nodes, `salloc`)
 
* Batch scriptsscripts (compute nodes, `sbatch`)
* External services (e.g. _Jupyterhub_)
* External services (e.g. _Jupyterhub_)
## SSH
## SSH
@@ -108,8 +99,7 @@ Different ways to work on a cluster depending on the task:
@@ -108,8 +99,7 @@ Different ways to work on a cluster depending on the task:
```{raw}
```{raw}
ssh <YOUR_USERNAME>@levante.dkrz.de
ssh <YOUR_USERNAME>@levante.dkrz.de
```
```
(use your user id instead of a123456)
* Now you have access to the command line **on levante**
* Now you have access to the command line on levante
## Public keys
## Public keys
@@ -117,12 +107,12 @@ Different ways to work on a cluster depending on the task:
@@ -117,12 +107,12 @@ Different ways to work on a cluster depending on the task:
## Generating SSH keys
## Generating SSH keys
* Generate a private / public ssh key pair for authentication on your local system
* Generate a private / public SSH key pair on your local system
```bash
```bash
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_levante
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_levante
```
```
* Upload the public (`.pub`) key to your [DKRZ profile](https://luv.dkrz.de) ([instructions](https://docs.dkrz.de/doc/levante/access-and-environment.html)), so levante can check your connection attempts
* Upload the public (`.pub`) key to your DKRZ profile ([how to](https://docs.dkrz.de/doc/levante/access-and-environment.html#public-key-authentication))
* Use your private ssh key instead of your password when connecting to levante
* Use your private SSH key to authenticate when connecting to levante
```bash
```bash
ssh -i ~/.ssh/id_ed25519_levante <YOUR_USERNAME>@levante.dkrz.de
ssh -i ~/.ssh/id_ed25519_levante <YOUR_USERNAME>@levante.dkrz.de
```
```
@@ -132,8 +122,8 @@ Different ways to work on a cluster depending on the task:
@@ -132,8 +122,8 @@ Different ways to work on a cluster depending on the task:
* Create a config file for convenience
* Create a config file for convenience
```{.sshconfig filename=~/.ssh/config}
```{.sshconfig filename=~/.ssh/config}
Host levante # the name you use in the shell
Host levante # the name you use in the shell
Hostname levante.dkrz.de # the official name of the system
Hostname levante.dkrz.de # real host name of the system
User <YOUR_USERNAME> # add your user id
User <YOUR_USERNAME> # your user id
IdentityFile ~/.ssh/id_ed25519_levante # your private key
IdentityFile ~/.ssh/id_ed25519_levante # your private key
```
```
(call `man ssh_config` for more info)
(call `man ssh_config` for more info)
@@ -147,23 +137,16 @@ Different ways to work on a cluster depending on the task:
@@ -147,23 +137,16 @@ Different ways to work on a cluster depending on the task:
## Remote file transfer {auto-animate=true}
## Remote file transfer {auto-animate=true}
* There are different tools to transfer between your local machine and a remote server
* There are various tools to transfer data between your local machine and a remote server (e.g. [`scp`](https://linux.die.net/man/1/scp), [`rsync`](https://linux.die.net/man/1/rsync), [`uftp`](https://uftp-multicast.sourceforge.net))
* For single files, a simple `scp` is sufficient
* `rsync` is a powerful option available on most machines
```{raw}
scp <YOUR_USERNAME>@levante.dkrz.de:file_in_home .
```
* `rsync` is a more powerful alternative
```{raw}
```{raw}
rsync <YOUR_USERNAME>@levante.dkrz.de:file_in_home .
rsync <YOUR_USERNAME>@levante.dkrz.de:file_in_home .
```
```
## Remote file transfer {auto-animate=true visibility="uncounted"}
## Remote file transfer {auto-animate=true visibility="uncounted"}
* There are different tools to transfer between your local machine and a remote server
* For single files, a simple `scp` is sufficient
* There are various tools to transfer data between your local machine and a remote server (e.g. [`scp`](https://linux.die.net/man/1/scp), [`rsync`](https://linux.die.net/man/1/rsync), [`uftp`](https://uftp-multicast.sourceforge.net))
```{raw}
* `rsync` is a powerful option available on most machines
scp levante:file_in_home .
```
* `rsync` is a more powerful alternative with more options
```{raw}
```{raw}
rsync levante:file_in_home .
rsync levante:file_in_home .
```
```
@@ -175,9 +158,11 @@ Different ways to work on a cluster depending on the task:
@@ -175,9 +158,11 @@ Different ways to work on a cluster depending on the task:
|keep scripts | store output | temporary stuff |
|keep scripts | store output | temporary stuff |
| small | big | big |
| small | big | big |
| SSD | HDD | HDD |
| SSD | HDD | HDD |
|backuped | no backup | deleted after 2 weeks |
|backup | no backup | deleted after 2 weeks |
We all lost data to `/scratch`, many killed something on `/work`
::: {.fragment .fade-up}
 
> Anything not saved will be lost --- _Nintendo_
 
:::
## Compute nodes
## Compute nodes
@@ -191,7 +176,7 @@ We all lost data to `/scratch`, many killed something on `/work`
@@ -191,7 +176,7 @@ We all lost data to `/scratch`, many killed something on `/work`
## Take home messages
## Take home messages
* High-performance computers like Levante are technically a cluster of numerous _normal_ computers that share the same hard disks
* High-performance computers like Levante are technically a cluster of numerous _normal_ computers that share the same hard disks
* Using a secure shell (`ssh`) one can login to Levante and use it as any other machine
* Using a secure shell (`ssh`) you can log into Levante and (mostly) use it like any other machine
* Computationally demanding or long-lasting jobs have to be submitted to the job queue
* Computationally demanding or long-lasting jobs have to be submitted to the job queue
# Hands-on session {.handson}
# Hands-on session {.handson}
@@ -215,7 +200,7 @@ We all lost data to `/scratch`, many killed something on `/work`
@@ -215,7 +200,7 @@ We all lost data to `/scratch`, many killed something on `/work`
## tmux
## tmux
`tmux` enables to create and (re)attach to terminal sessions
`tmux` enables to create and attach to terminal sessions
* Crate a new session
* Crate a new session
```{raw}
```{raw}
@@ -229,7 +214,7 @@ We all lost data to `/scratch`, many killed something on `/work`
@@ -229,7 +214,7 @@ We all lost data to `/scratch`, many killed something on `/work`
## X-forwarding
## X-forwarding
You can use X-forwarding to forward graphical user interfaces (GUIs) from the server
forward graphical user interfaces (GUIs) from the server
* Install an X-server on your local machine (e.g. XQuartz)
* Install an X-server on your local machine (e.g. XQuartz)
* Pass the `-X` option to your `ssh` command^[or check the SSH config option]
* Pass the `-X` option to your `ssh` command^[or check the SSH config option]
Loading