From 0c4a4216a4eacaa9e009ec33d913b61aa267de33 Mon Sep 17 00:00:00 2001
From: Manuel Reis <reis@dkrz.de>
Date: Mon, 31 Mar 2025 18:19:12 +0200
Subject: [PATCH] Complete examples and handson
---
lectures/tooling-ci/slides.qmd | 376 ++++++++++++++++++++++++++-------
1 file changed, 296 insertions(+), 80 deletions(-)
diff --git a/lectures/tooling-ci/slides.qmd b/lectures/tooling-ci/slides.qmd
index e334be6..f2b354f 100644
--- a/lectures/tooling-ci/slides.qmd
+++ b/lectures/tooling-ci/slides.qmd
@@ -1,83 +1,232 @@
---
title: "Tooling and Continuous Integration"
subtitle: ""
-author: "Lukas Kluft, Tobias Kölling, (Flo)"
+author: ""
---
# *Software Development* Tooling
-## Software to create software 😵
-* Validate code
-* Test functionality
+::: {.incremental}
+
+- Software to develop software 😵
+
+:::
+
+## Tools aim at:
+* Simplifying
+ - Abstractions
+ - Encapsulation
+* Validate & Test
+* Deploy / install
* Speedup development
-* Make things easier
+
+:::{.notes}
+[Extended definition](https://en.wikipedia.org/wiki/Programming_tool)
+:::
+
+## _Toolchain_ example
+
+
+```{mermaid}
+flowchart LR
+ classDef tool stroke:#f96,shape:stadium-shaped
+ A[Source Code]
+ E([Editor]):::tool
+ C([Compiler]):::tool
+ I([Interpreter]):::tool
+ OS([Op. System]):::tool
+ B[Executable File]
+ O[["Output"]]
+ A --> E --> A
+ A --> C --> B
+ A --> I --> O
+ B --> OS --> O
+```
+
+```{mermaid}
+flowchart LR
+ subgraph Legend
+ f[File]
+ t([Tool])
+ style t stroke:#f96
+ o[[Result]]
+ end
+```
+. . .
+
+:::{.smaller}
+In python, compilation and execution are done in the same process.
+:::
+
+## Example
+
+```
+$ vim example.c
+```
+```C
+#include <stdio.h>
+int main() {
+ printf("Hello world\n");
+ return 0;
+}
+```
+```
+$ gcc example.c
+$ ./a.out
+Hello world!
+```
+
+. . .
+
+Modify the souce `file`, `compile` and `execute`:
+
+. . .
+
+```
+$ sed -i 's/world/there/g' example.c
+$ gcc example.c
+$ ./a.out
+Hello there!
+```
+
+. . .
+
+We use two programs/**tools** ( `sed` and `gcc`) to achieve this.
+
+
+## The devious way...
+
+Change output of executable from `Hello world!` to `Hello there!` **without** (re)compiling!
+
+:::{.smaller}
+Directly maniputale the binary executable. Refer to the [ASCII table](https://en.wikipedia.org/wiki/ASCII#Character_set)
+:::
+
+. . .
+```
+$ sed -i 's/\x77\x6f\x72\x6c\x64/\x74\x68\x65\x72\x65/g' a.out
+$ ./a.out
+Hello there!
+```
+
+. . .
+
+This is to show how important compilers and interpreters are!
+
+. . .
+
+For this example it did not need to be so cumbersome:
+```
+$ sed -i 's/there/world/g' a.out
+$ ./a.out
+Hello world!
+```
+
+. . .
+
+:::{.smaller}
+For completion (`hexdump` or `xxd` can help decoding binary contents)
+```
+$ xxd a.out | grep 'Hello world!'
+00002000: 0100 0200 4865 6c6c 6f20 776f 726c 6421 ....Hello world!
+$ python3 -c 'print(b"Hello world!".hex(" ",2))'
+4865 6c6c 6f20 776f 726c 6421
+```
+:::
## Making things easier
:::: {.columns}
::: {.column width="50%"}
+::: {.smaller}
Python
```python
def say_hi():
print("Hello World")
-
```
-
-
-:::{.justify}
-Python **interpreter** is implemented in C, which is **compiled** into Machine Code.
+Disassembled bytecode
+
+```python
+import dis
+dis.dis(say_hi)
+ 2 0 LOAD_GLOBAL 0 (print)
+ 2 LOAD_CONST 1 ('Hello World')
+ 4 CALL_FUNCTION 1
+ 6 POP_TOP
+ 8 LOAD_CONST 0 (None)
+ 10 RETURN_VALUE
-This abstraction makes the process of developing a program, **easier** and **faster**.
+```
:::
:::
+
::: {.column width="50%"}
-Assembly
-```asm
- 0 RESUME 0
- 1 LOAD_CONST 0 (<code object say_hi at 0x5f19d84931b0, file "example.py", line 1>)
- MAKE_FUNCTION
- STORE_NAME 0 (say_hi)
- RETURN_CONST 1 (None)
+The python _interpreter_ **generates** and **executes** bytecode from the source files
-Disassembly of <code object say_hi at 0x5f19d84931b0, file "example.py", line 1>:
- 1 RESUME 0
+::: {.incremental}
+- The **interpreter** is written in C and **compiled** into Machine Code.
+:::
- 2 LOAD_GLOBAL 1 (print + NULL)
- LOAD_CONST 1 ('Hello world')
- CALL 1
- POP_TOP
- RETURN_CONST 0 (None)
-```
:::
:::
+. . .
+
+::: {.smaller}
+This abstraction makes it **easier** and **faster** to develop a program.
+:::
+
+# Categories
+
+Lets check on types of tools
+
+
## Are programming languages tools?🤷
+. . .
+
+Yes & No!
+
+::: {.notes}
+Languages are an abstract concept but they play the fundamental role of programming.
+:::
+
+:::{.incremental}
* Compiler/Interpreter is a tool
- - `/usr/bin/python3` follows the [Python Language Reference](https://docs.python.org/3/reference/index.html)
+ - [`CPython`](https://www.python.org/), [`PyPy`](https://pypy.org/) and [`Pythran`](https://pythran.readthedocs.io/en/latest/) follow the [Python Language Reference](https://docs.python.org/3/reference/index.html)
* Libraries / modules (often) come with tools
- `pdb` - python debugger
- `pip` - famous package installer
+:::
+
+
+## What about the Operating System?
+
+It's a software that manages running programs. So ... **yes!**
+
+. . .
-## What about the Operating System?
+Generally the execution of a program can only happen by interacting with the operating system.
-It's a software that manages running programs. So yes!
+. . .
-It often supplies users with software development tools:
+:::{.smaller}
+There are bundles of development tools for each operating system:
-* [🪟](https://developer.microsoft.com/en-us/windows/dev-tools/) `dev-tools`
-* [ðŸŽ](https://developer.apple.com/xcode/) `xcode`
-* [ðŸ§](https://itsfoss.com/build-essential-ubuntu/) `build-essential`
+- [🪟](https://developer.microsoft.com/en-us/windows/dev-tools/) `dev-tools`
+- [ðŸŽ](https://developer.apple.com/xcode/) `xcode`
+- [ðŸ§](https://itsfoss.com/build-essential-ubuntu/) `build-essential`
+:::
## Tools in the development environment
- Package managers (`conda`, `pyenv`, `uv`)
- Version control systems (`git`)
-- Editors, IDEs - Integrate Development Environments
+- Editors, IDEs - Integrated Development Environments
- Linters, formatters, type-checkers (`flake8`, `black`, `ruff`)
- Compilers and/or Interpreters (e.g. `gcc`, `python`)
- Testing tools (`pytest`,[`unittest`](https://docs.python.org/3/library/unittest.html))
@@ -95,7 +244,8 @@ It often supplies users with software development tools:
* All languages follow a syntax.
* Easy for us to make mistakes/typos
-* Hard to understand where the problem is
+* Spot where the problem are.
+
::: {.columns}
@@ -106,7 +256,7 @@ def helloworld():
print('world')
helloworld()
- ```
+```
:::
:::{.column}
@@ -118,13 +268,11 @@ $ python3 helloworld.py
IndentationError: unindent does not match any outer indentation level
```
:::
-
:::
## Linters & formatters
::: {.columns}
-
::: {.column}
Popular linters
```
@@ -138,7 +286,6 @@ helloworld.py:3:4: E0001: Parsing failed: 'unexpected indent (helloworld, line 3
```
:::
-
::: {.column}
Formatters
```
@@ -156,9 +303,7 @@ def helloworld():
helloworld()
```
-
:::
-
:::
```
@@ -170,27 +315,98 @@ Found 2 errors.
## Hands-on Session {.handson}
-`git precommit hook` exercise?
-- It could be linting .gitlab.yaml
--
-- Apply provided faulty patch, check if pushing code
+Simple `git precommit hook` exercise.
+
+- Clone this repo: [ci-example](https://gitlab.dkrz.de/generic-software-skills/ci-example)
+- Use a linter to check if the file `.gitlab.yml` is complying to the Gitlab [YAML](https://docs.gitlab.com/ci/yaml/) syntax refence
+
+. . .
+
+:::{.smaller}
+```bash
+module load git python3
+python3 -m pip install yamllint
+[[ ! $PATH =~ "${USER}/.local/bin" ]] && PATH=${PATH}:/home/$(echo $USER | cut -c1)/$USER/.local/
+git clone https://gitlab.dkrz.de/generic-software-skills/ci-example
+cd ci-example
+yamllint .gitlab.yml
+```
+:::
+
+:::aside
+Does it succeed or fail?
+:::
+
+## Pre-commit hook 🪠{.handson}
+
+:::{.smaller .incremental}
+- Create a shell script named `.git/hooks/pre-commit` so that:
+ - The yaml linter is called on the `yml` file
+ - Returns the exit code of the linter (0 if sucess else error)
+- Fix the yaml file to fix warnings and erros:
+ - Add (stage) and commit the changes to automatically trigger the linter.
+:::
+
+. . .
+
+:::{.smaller}
+- Bonus: do that only when `.gitlab-ci.yml` is stagged (you can use `git diff --cached --name-only` to check that)
+:::
+
+::: aside
+Useful git _Fu_:
+
+- `git commit -am 'Fix yml syntax' # Adds and commits modification`
+- `git reset HEAD~1 # Undo last commit!`
+:::
+::: notes
+```
+$ git add .gitlab-ci.yml
+$ git diff --cached --name-only
+.gitlab-ci.yml
+$ git restore --staged .gitlab-ci.yml
+$ git diff --cached --name-only`
+```
+:::
+
+
+## Solution {.handson visibility="hidden"}
+
+Content of `.git/hooks/pre-commit`
+
+```bash
+#!/bin/sh
+code=0
+if git diff --cached --name-only | grep -q '.gitlab-ci.yml'; then
+ echo Running linter on .gitlab-ci.yaml
+ yamllint .gitlab-ci.yml
+ code=$?
+ echo Result: $code
+fi
+exit $code
+```
+
+# Continuous Integration (CI) {.smaller}
-[//]:... The idea is that when introducing a change to a code-base, it should be checked **automatically**.
-# Continuous Integration (CI)
- * Automated builds and tests with each code change
- * Rapid feedback on build and test status
- * Consistent, reproducible testing environment
- * Frequent merging to reduce integration conflicts
- * Improved collaboration across development teams
+* Automated builds and tests with each code change
+* Rapid feedback on build and test status
+* Consistent, reproducible testing environment
+* Frequent merging to reduce integration conflicts
+* Improved collaboration across development teams
+
+::: notes
+... The idea is that when introducing a change to a code-base, it should be checked **automatically**.
+:::
## Where?
* Git repositoris are the **perfect place for CI**!
-* The CI assesses if a commit is up to the expectations ("Passes/Fails the CI").
+* The CI assesses if a commit is up to the expectations
+ - "Passes/Fails the CI"
* Provided on popular git hosting systems
- - [GitLab CI](https://docs.gitlab.com/ee/ci/)
- - [GitHub Actions](https://docs.github.com/en/actions/about-github-actions/understanding-github-actions).
+ - [GitLab CI](https://docs.gitlab.com/ee/ci/) as pipelines
+ - [GitHub Actions](https://docs.github.com/en/actions/about-github-actions/understanding-github-actions).
## What to expect?
@@ -209,15 +425,8 @@ Found 2 errors.
- One can have many variants of the same stage
* Tests to be done in each of them (to be continued...)
-#
-[//]::(https://gitlab.dkrz.de/generic-software-skills/lecture-materials/-/pipelines/96666)
-[//]::(https://github.com/faster-cpython/cpython/actions/runs/13454935981)
-
-## Example
-
-:::: {.columns}
+## Example of `.gitlab-ci.yml` {.smaller}
-::: {.column width="50%"}
```yml
stages:
- lint
@@ -226,29 +435,36 @@ stages:
linter:
stage: lint
- image: python # container with basic python tools
+ image: python # container image with basic python tools
script:
- pip install flake8
- flake8 . --count --exit-zero --statistics
-```
-:::
-::: {.column width="50%"}
-```yml
+
type-check:
- stage: build
- image: python
- script:
- - pip install mypy
- - mypy .
+ stage: build
+ image: python
+ script:
+ - pip install mypy
+ - mypy .
```
-:::
-::::
+
+## Real case examples
+
+- [GitHub Actions]( https://github.com/faster-cpython/cpython/actions/runs/13454935981) workflow for CPython
+- Course lectures on [Gitlab CI](https://gitlab.dkrz.de/generic-software-skills/lecture-materials/-/pipelines/96666)
+ - Gitlab CI syntax: [docs.gitlab.com/ci/yaml](https://docs.gitlab.com/ci/yaml/)
+
## Hands-on Session {.handson}
-* Create a gitlab CI pipeline for [repo](https://gitlab.dkrz.de/generic-software-skills/ci-example) that:
-1. Install tools to lints your changes
-2. Makes sure environment has required dedependecies
-3. Runs the program (no testing yet)
-4. Bonus: creates an executable?
+Change the Gitlab CI pipeline for [repo](https://gitlab.dkrz.de/generic-software-skills/ci-example) so that:
+
+- Install tools to lint your source code
+- Runs the program and check if the output is "Hello world!"
+- Bonus: Setup [ruff-pre-commit](https://docs.astral.sh/ruff/integrations/#pre-commit) and [Gitlab CI](https://docs.astral.sh/ruff/integrations/#gitlab-cicd)
+. . .
+
+```
+pip install pre-commit
+```
--
GitLab