Skip to content
Snippets Groups Projects
Commit a0301867 authored by Martin Bergemann's avatar Martin Bergemann :speech_balloon:
Browse files

Update docs

parent b6fc0abf
No related branches found
No related tags found
1 merge request!43Update docs
Hide whitespace changes
Inline Side-by-side
README.md
+ 21
− 14
View file @ a0301867
......@@ -55,29 +55,28 @@ We also recommend installing a command line based text editor like vim, nano, et
After installing the above listed packages via cygwin you can clone and
install the freva deployment:
```bash
pip install freva-deployment --extra-index-url https://gitlab.dkrz.de/api/v4/projects/1732/packages/pypi/simple
```console
python3 -m pip install -U freva-deployment
```
Add the `--user` flag if you don't have sufficient rights.
## Installation on \*nix systems or wsl.
If you're using Linux, OsX or a Windows subsystem for Linux (WSL) it should
be sufficient to issues the following commands:
```bash
pip install (--user) ansible
pip install (--user) git+https://gitlab.dkrz.de/freva/deployment.git
```console
python3 pip install -U freva-deployment
```
This command installs ansible and all required python packages.
> **_Note:_** On *CentOS* python SELinux libraries need to be installed.
> You will need to install libselinux for your CentOS version.
```bash
python3 -m pip install (--user) libselinux-python3
```console
python3 -m pip install libselinux-python3
```
## Commands after installation:
The `pip` install command will create *four* different commands:
The `pip install` command will create *four* different commands:
- `deploy-freva-map`: To setup a service that keeps track of all deployed
freva instances and their services.
- `deploy-freva`: Text user interface to configure and run the deployment.
......@@ -85,6 +84,14 @@ The `pip` install command will create *four* different commands:
- `freva-service`: Start|Stop|Restart|Check services of freva instances.
- `freva-migrate`: Command line interface to manage project migration from
old freva systems to new ones.
If you can't find the commands mentioned above pip was probably installing
them in the user space. In that case you need to append your `PATH`
environment variable by `~/.local/bin`.If you use bash for example, add
the following command to your local `.bashrc`:
```console
export PATH=$PATH:$HOME/.local/bin
```
## Installing docker/podman and sudo access to the service servers
......@@ -105,7 +112,7 @@ We have created a service that keeps track of the locations of all services for
a certain freva instance. Although not strictly needed we recommend you to setup
this special server mapping service. To do so use the following command:
```bash
```console
deploy-freva-map --help
usage: deploy-freva-map [-h] [--port PORT] [--wipe] [--user USER] [--python-path PYTHON_PATH] [-v] [-V] servername
......@@ -218,7 +225,7 @@ If the target machine where the services (solr, mariadb, web) were deployed
is a Linux machine you will have a `systemd` unit service was created.
You can control the service via the `freva-service` command:
```bash
```console
freva-service --help
usage: freva-service [-h] [--server-map SERVER_MAP] [--services {web,db,solr} [{web,db,solr} ...]] [--user USER] [-v] [-V]
{start,stop,restart,status} [project_name]
......@@ -241,9 +248,9 @@ options:
-v, --verbose Verbosity level (default: 0)
-V, --version show program's version number and exit
```
The following command restarts `web` server for the `xces` project at dkrz:
```bash
freva-service restart xces --services web --user k12345 --domain dkrz
The following command restarts `web` server for the `xces`:
```console
freva-service restart xces --services web --user k12345
```
All services (`db`, `web` and `solr`) will be selected if the `--services` option
is omitted.
......@@ -279,7 +286,7 @@ you can figure out the `db_docker_name` using the following command:
docker container ls
```
### Git related unit tests in backend playbook fail
### Git related unit tests in core playbook fail
Git pull and push commands tend to fail if you haven't configured git. In this case change into the /tmp/evaluation_system directory of the host that runs the playbook
then manually trigger the unit tests by
......
docs/Configure.md
+ 6
− 6
View file @ a0301867
......@@ -46,7 +46,7 @@ Please refer to the [Appendix I](TuiHowto.html) on tui usage instructions.
If you already have a configuration saved in a toml inventory file you can
issue the `deploy-freva-cmd` command:
```bash
```console
deploy-freva-cmd --help
usage: deploy-freva-cmd [-h] [--server-map SERVER_MAP] [--config CONFIG] [--steps {web,core,db,solr} [{web,core,db,solr} ...]]
[--ask-pass] [-v] [-V]
......@@ -80,16 +80,16 @@ The following activation scripts are located in the `<root_dir>/freva` folder:
* *loadfreva.module*: Modules file
The source scripts can either be copied where they are automatically sourced
or users can be instructed to use the source command for their shell. If
`modules` is available on the host system you can copy the *loadfreva.modules* file
into the `MODULEPATH` location.
`modules` is available on the host system you can copy the *loadfreva.modules*
file into the `MODULEPATH` location.
### Systemd units for all services:
If the target machine where the services (solr, mariadb, web) were deployed
are Linux machines you will have access to a `systemd` unit of the created
service. In general the services can be accessed by `<project_name>-<service_name>.service`
If for example the `project_name` key was set to `clex-ces` then the following services
are created:
service. In general the services can be accessed by
`<project_name>-<service_name>.service` If for example the `project_name`
key was set to `clex-ces` then the following services are created:
- database: `clex-ces-db.service`, `clex-ces-vault.serice`
- solr: `clex-ces.solr.service`
......
docs/Installation.md
+ 26
− 7
View file @ a0301867
......@@ -42,25 +42,44 @@ We also recommend installing a command line based text editor like vim, nano, et
After installing the above listed packages via cygwin you can clone and install the freva-deployment:
```bash
pip install freva-deployment --extra-index-url https://gitlab.dkrz.de/api/v4/projects/1732/packages/pypi/simple
```console
python3 pip install -U freva-deployment
```
### Installation on \*nix systems or wsl.
If you're using Linux, OsX or a Windows subsystem for Linux (WSL) it should be
sufficient to issues the following commands:
```bash
pip install ansible
pip install freva-deployment --extra-index-url https://gitlab.dkrz.de/api/v4/projects/1732/packages/pypi/simple
```console
python3 -m pip install -U freva-deployment
```
This command installs ansible and all required python packages.
> **_Note:_** On *CentOS* python SELinux libraries need to be installed.
> You will need to install libselinux for your CentOS version.
```bash
```console
python3 -m pip install libselinux-python3
```
## Commands after installation:
The `pip install` command will create *four* different commands:
- `deploy-freva-map`: To setup a service that keeps track of all deployed
freva instances and their services.
- `deploy-freva`: Text user interface to configure and run the deployment.
- `deploy-freva-cmd`: Run already configured deployment.
- `freva-service`: Start|Stop|Restart|Check services of freva instances.
- `freva-migrate`: Command line interface to manage project migration from
old freva systems to new ones.
If you can't find the commands mentioned above pip was probably installing
them in the user space. In that case you need to append your `PATH`
environment variable by `~/.local/bin`. If you use bash for example, add
the following command to your local `.bashrc`:
```console
export PATH=$PATH:$HOME/.local/bin
```
## Installing docker/podman and sudo access to the service servers
Since the services of MariaDB, Apache Solr and Apache httpd will be deployed on
docker container images, docker needs to be available on the target servers.
......@@ -80,7 +99,7 @@ for *all* Freva instance within your.
Although not strictly needed we recommend you to setup this special server
mapping service. To do so use the following command:
```bash
```console
deploy-freva-map --help
usage: deploy-freva-map [-h] [--port PORT] [--wipe] [--user USER] [--python-path PYTHON_PATH] [-v] [-V] servername
......
docs/Transition.md
+ 99
− 12
View file @ a0301867
......@@ -47,7 +47,7 @@ the previous Freva instance will be ready for use.
## Transition to new DRS Config
In the old version the DRS (Date Reference Syntax) File configuration,
In the old version the DRS (Date Reference Syntax) File configuration,
that is the definitions of dataset metadata, was hard coded into the module
`evaluation_system.model.file`. In the new version this configuration
is saved in a designated [toml](https://toml.io/en/) file (drs_config.toml).
......@@ -79,24 +79,40 @@ These are the disadvantages of this method:
### Transitioning steps:
1. clone the repository of a plugin, change into the directory and create a new branch.
There are multiple ways of how you can get your old plugin back to the new
Freva system. We do recommend a deployment strategy involving conda.
While this strategy is not strictly necessary it offers the best as
possible reproducibility and is host system independent.
Meaning your plugins can be easily transitioned to other institutions and
are more likely to work after major system updates on your host system.
Here we briefly cover the steps that are needed to bring your old plugin back
to life in the new Freva system. We will also discuss alternatives
to using conda. Regardless of choice on using conda or not the first two
steps will be necessary.
2. export the user plugin variable:
#### Common steps:
```bash
1. clone the repository of a plugin, change into the directory and create a new branch.
2. export the user plugin variable:
```console
export EVALUATION_SYSTEM_PLUGINS=$PWD/path_to_wrapper_file,class_name
freva plugin -l
```
Most certainly, the plugin manager will output a warning that the plugin could
not be loaded. If it does, change the plugin accordingly to make the
warning messages go away.
Most certainly, the plugin manager will output a warning that the plugin could not be loaded.
If it does, change the plugins accordingly to make the warning messages go away.
#### a. Using conda:
As mentioned above this step has the advantage that you increase the reproducibility
of your plugin. Transitioning your plugin to other institutions is also easy
because all libraries are encapsulated from the host system and hence independent.
While not strictly necessary it is a good idea to familiarise yourself with
[anaconda](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html).
3. Download the [conda environment file template](https://swift.dkrz.de/v1/dkrz_3d3c7abc-1681-4012-b656-3cc1058c52a9/k204230/freva-transition/plugin-env.yml) and the [Makefile template](https://swift.dkrz.de/v1/dkrz_3d3c7abc-1681-4012-b656-3cc1058c52a9/k204230/freva-transition/Makefile)
4. Add all dependencies to the `plugin-environment.yml` file. In doubt search [anaconda.org](https://anaconda.org) for dependencies.
4. Add all dependencies to the `plugin-env.yml` file. In doubt search [anaconda.org](https://anaconda.org) for dependencies.
5. If needed adjust the `build` step in the `Makefile` for compiling plugin dependencies, e.g. fortran dependencies.
......@@ -106,12 +122,25 @@ If it does, change the plugins accordingly to make the warning messages go away.
8. Format the plugin using black: `black -t py310 path_to_plugin.py`
#### b. Using the environment of freva
If your plugin doesn't need many libraries you can simply try to use everything
the comes with freva. This is the easiest way as you don't have to do anything.
Simply try to execute all commands that come with your plugin and see what
happens.
#### c. Using software of the host system
You can also make use of the software installed on the host system. For
example via spack. Many HPC systems offer the `module` command. Using this
approach will result in a plugin that is tailored around the current host system
you are using. Future updates may break usage and you defiantly won't be able
to use your plugin at other institutions.
### Transitioning `python2` plugins
Python plugins (especially python2) need special care. The recommended strategy
is to convert the plugin content to python3. If this is not possible an anaconda
python2 environment should be created.
If in the original plugin the plugin code is directly executed in the `run_rool`
If in the original plugin the plugin code is directly executed in the `run_rool`
method (formerly named `runTool`) this code has to be split from the new `run_tool` method. The
transition strategy is gathering the essential information in a `json` file that
is passed to the actual core part of the plugin. The code below shows a simple
......@@ -145,7 +174,7 @@ code.
```python
def runTool(self, config_dict={}):
def run_tool(self, config_dict={}):
"""This is the wrapper API part of the plugin.
It gathers the plugin information and passes the needed information
to the actual plugin code that is split into another python file."""
......@@ -171,7 +200,7 @@ def runTool(self, config_dict={}):
with open(tf.name, "w"):
json.dump(**compute_kw, f, indent=3)
self.call(f"python2 {Path(__file__).parent / 'compute_python2'} {tf.name}")
return self.prepareOutput(config_dict["output_dir"])
return self.prepare_output(config_dict["output_dir"])
```
The below code demonstrates the usage of the above created `json` file.
......@@ -194,3 +223,61 @@ if __name__ == "__main__":
raise ValueError("Usage: {} path_to_json_file.json".format(sys.argv[0]))
calculate(config["variable"], config["files"], config["output_dir"])
```
If you want to use the json file in a bash script you must install the `jq`
json parser. Simply add `jq` to your `plugin-env.yml` file and read the
[docs of jq](https://stedolan.github.io/jq/tutorial/).
### After conda deployment: Increasing reproducibility of your plugin
If you have successfully deployed your plugin environment using conda you
can increase the reproducibility by "freezing" all packages that have
been installed by conda. This will increase the reproducibility of your package
because the versions of your packages will exactly match the one you are
using at this point in time. Every time you re-install the plugin environment
you will have the same dependency versions. To fix the package versions
execute the following command:
```
./plugin_env/bin/conda list --explicit > spec-file.txt
```
Add and commit the created `spec-file.txt` to your repository. Afterwards
you can replace
`conda env create --prefix ./plugin_env -f plugin-env.yml --force` in the `conda`
section of your `Makefile` by:
```
conda env create --prefix ./plugin_env -f spec-file.txt --force
```
And you're done.
### Problem: conda doesn't finish resolving dependencies
Sometimes conda is unable/won't finish to solve all dependencies. You have a
couple of options in that case. First you can try replacing the `conda` command
by `mamba` in your Makefile. `mamba` is written in C and comes with a
different dependency solver. Most of the time switching from `conda` to `mamba`
solves the problem. If the issues persists with `mamba` you can try identifying
the problematic package(s). Usually by guessing which package(s) might be the
offending ones and removing them from the `plugin-env.yml` file. Once the
package(s) have been circled you can create an additional conda environment.
For example by adding the following line into the `conda` section of the
`Makefile`. For example:
```
conda create -c conda-forge -p ./plugin_env2 cdo netcdf-fortran
``
If you want to use resources from this environment in your plugin you need
to modify the environment in your API wrapper. Following the example
above, this modification could look like this:
```python
env = os.environ.copy()
this_dir = os.path.dirname(os.path.abspath(__file__))
env["PATH"] = env["PATH"] + ":" + os.path.join(this_dir, "plugin_env2", "bin")
env["LD_LIBRARY_PATH"] = os.path.join(this_dir, "plugin_env2", "lib")
self.call(your_command_here, env=env)
```
If you need to compile additional libraries you probably also want to adjust
the `PATH` and `LD_LIBRARY_PATH` variable in the file that executes the compile
step to be able to pick up the `lib` and `bin` folders in the `plugin_env2`
conda environment.
docs/TuiHowto.md
+ 44
− 31
View file @ a0301867
......@@ -70,13 +70,13 @@ essential setup steps:
for multi-user write access (2775) on the target machine. This assures that
multiple admins can adjust the Freva configuration. You can leave this
variable blank.
11. If you need to change to a common user (via sudo -i) that is able to install
Freva on the HPC system you can set this common user name here. Note that this
method should only be used if freva needs to be set up with a shared user
account *and* you cannot log on to the HPC system with this shared user name.
Most of the time it should not be necessary to use this option, either because
you don't need a shared user account, or you can instruct ansible to directly
log on as the shared user.
11. If you need to change to a common user (for example via `sudo -i`) that is
able to install Freva on the HPC system you can set this common user name
here. Note that this method should only be used if freva needs to be set up
with a shared user account *and* you cannot log on to the HPC system with
this shared user name. Most of the time it should not be necessary to use
this option, either because you don't need a shared user account, or you
can instruct ansible to directly log on as the shared user.
12. If you have an anaconda binary already installed on the target system you
can set the path to the `conda` or `mamba` executable here. Leaving this variable
blank (default) will cause the deployment to temporary download and create
......@@ -117,60 +117,73 @@ essential setup steps:
7. A short text about the Freva admin team/group. This text will go to the
header of the web page.
8. Contact email address of the admin(s).
9. Address of the institute, will be appearing in the footer of the website.
10. Detailed overview over the project, this text will go into the left central
9. Smtp mail server host name. This mail server will be used to send emails
from the web ui.
10. Address of the institute, will be appearing in the footer of the website.
11. Detailed overview over the project, this text will go into the left central
box in the Freva main page. It should describe the purpose of this project
11. The header for the detailed overview mentioned in 9.
12. Host name(s) - usually HPC login node(s) - the web backend code submits
12. The header for the detailed overview mentioned in 9.
13. Host name(s) - usually HPC login node(s) - the web backend code submits
plugin jobs from. The backend will logon to this host name and submit a
plugin job.
13. String of ldap server uri(s) (comma separated). The ldap server uris are
14. String of ldap server uri(s) (comma separated). The ldap server uris are
used to make authentication requests via the web side.
14. Tick this box (`<SPACE>`) to enable TLS encrypted connection from the django
15. Tick this box (`<SPACE>`) to enable TLS encrypted connection from the django
web application to the ldap server.
[See also django ldap tls config](https://django-auth-ldap.readthedocs.io/en/latest/authentication.html#notes)
15. Set the user group name that is allowed to logon to the web ui.
16. Set the ldap user base filters. See also the
16. Set the user group name that is allowed to logon to the web ui.
17. Set the ldap user base filters. See also the
[djanog ldap authentication for user](https://django-auth-ldap.readthedocs.io/en/latest/users.html)
17. Set the ldap group base filters. See also the
18. Set the ldap group base filters. See also the
[django ldap authentication for groups](https://django-auth-ldap.readthedocs.io/en/latest/groups.html#finding-groups)
18. Set the distinguished name of the ldap user. This setting is used to make
19. Set the distinguished name of the ldap user. This setting is used to make
queries to the ldap server. TODO: [this might not be necessary](https://django-auth-ldap.readthedocs.io/en/latest/authentication.html#search-bind)
19. Set the password for the ldap user.
20. Set the ldap search key for finding the first name entries
21. Set the ldap search key for finding the last name entries
22. Set the ldap search key for finding email addresses
23. Set the type of ldap object class name
24. Set the type of ldap group type (nested or posix). Note: This is a
20. Set the password for the ldap user.
21. Set the ldap search key for finding the first name entries
22. Set the ldap search key for finding the last name entries
23. Set the ldap search key for finding email addresses
24. Set the type of ldap object class name
25. Set the type of ldap group type (nested or posix). Note: This is a
dropdown menu, hit enter to select the options.
25. Set the evaluation_system ldap class that handles the ldap connections
26. Set the path to the `python3.4+` binary, this should be set if the python
26. Set the evaluation_system ldap class that handles the ldap connections
27. Set the path to the `python3.4+` binary, this should be set if the python
binary is not part of the `$PATH` variable.
27. Set the login user name for the remote machine.
28. Set the login user name for the remote machine.
### Notes on the web ui setup
Currently there are ties between the core running on the HPC machine and
the web ui. These are a common configuration `evaluation_system.conf` and
a common preview folder. Plugins running on the HPC will store visual content
the web ui. These are:
- a common configuration `evaluation_system.conf` and
- a common preview folder. Plugins running on the HPC will store visual content
like plots into the plugin folder, which can be displayed by the web ui.
This interconnection is usually fulfilled creating a network mount from the
HPC system to the machine running the web ui. The deployment routine is expects
HPC system to the machine running the web ui. The deployment routine expects
the preview folder and `evaluation_system.conf` to be present (mounted) during
on the host machine during deployment time. The docker container expects the
mounts to be in the same path as on the HPC system. For example, if
an instance of Freva has been deployed on `/work/clex/freva-clex` then
this directory should be available via the same path on the machine running
the django web application. The same applies to all paths defined in the
the web ui application. The same applies to all paths defined in the
plugin section in the `evaluation_system.conf`. Please also refer to the
[Architecture section](Architecture.html#web-based-user-interface) for more
information.
The django web application service needs two more services, that are
The web ui application service needs two more services, that are
automatically set up. Those are a redis service, which is used as a database
query cache and an apache httpd service that is used as reverse proxy.
The httpd configuration for the reverse proxy will be saved
`/opt/freva/<project_name>/web_service/`.
During the deployment process of the web ui you will be ask your login
credentials for the smtp email server. The credentials are needed to connect
to the email server and send emails. Your credentials won't be saved anywhere
on disk but securely stored in the vault service that gets deployed when setting
up the database.
## Solr server setup
The third screen configures the setup apache solr serch server. At the top of
the screen you'll see a tick box, which indicates whether or not this step is
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

Imprint | Privacy Policy