Merge branch 'master' of https://gitlab.dkrz.de/freva/deployment into use_conda_freva

.gitlab-ci.yml

@@ -46,10 +46,9 @@ release_job:
         - chmod +x /usr/local/bin/release-cli
     script:
         - python setup.py sdist bdist_wheel
-        - TWINE_PASSWORD=${CI_JOB_TOKEN}
-          TWINE_USERNAME=gitlab-ci-token
-          python -m twine upload --skip-existing --verbose --repository-url
-          ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
+        - TWINE_PASSWORD=${PIPY_TOKEN}
+          TWINE_USERNAME=__token__
+          python -m twine upload --skip-existing --verbose dist/*
     release:
         tag_name: '$CI_COMMIT_TAG'
         description: '$CI_COMMIT_TAG'

docs/AfterDeployment.md

+# After successful deployment
+## After deployment of the core
+### Activation scripts and module files for users
+The core deployment will create activation scripts for various shell flavours.
+The following activation scripts are located in the `<root_dir>/freva` folder:
+* *activate_sh*: To be sourced by shell flavours like shell, bash, zsh etc.
+* *activate_csh*: To be sourced by c-shell flavours like csh, tcsh
+* *activate_fish*: To be source by fish
+* *loadfreva.module*: Modules file
+The source scripts can either be copied where they are automatically sourced
+or users can be instructed to use the source command for their shell. If
+`modules` is available on the host system you can copy the *loadfreva.modules*
+file into the `MODULEPATH` location.
+
+
+## Solr, database and web services:
+If the target machine where the services (solr, mariadb, web) were deployed
+are Linux machines you will have access to a `systemd` unit of the created
+service. In general the services can be accessed by
+`<project_name>-<service_name>.service` If for example the `project_name`
+key was set to `clex-ces` then the following services are created:
+
+- database: `clex-ces-db.service`, `clex-ces-vault.serice`
+- solr: `clex-ces.solr.service`
+- web ui: `clex-ces-web.service`, `clex-ces-redis.service`, `clex-ces-httpd.service`
+
+### Access of service data on the host machine
+The data of the services, like the database or solr cores are stored "outside"
+the docker containers on the host machine and can be accessed at
+`/opt/freva/<project_name>/<service_name>_service/` for example
+`/opt/freva/clex-ces/db_service`
+
+
+### Simple backup scripts:
+The `db` and `solr` services offer a very simple backup script that can
+be run from outside the container. To issue a backup command simply call the
+following command `docker/podman exec -it <project_name>-solr/db /usr/local/bin/daily_backup`.
+For example:
+
+```
+podman exec -it clex-ces-solr /usr/local/bin/daily_backup
+```
+
+If you have `anacron` set up on your host machine then a cron script to
+automatically backup databases and solr cores is applied nightly.
+By default the script keeps the last 7 backups. Backup data can be found in:
+
+- `db`:`/opt/freva/<project_name>/db_service>/backup`
+- `solr`: `/opt/freva/<project_name>/solr_service/<core_name>/data/snapshot.YYYYMMDDHHMMSSMS`
+
+This is only a rudimentary backup solution, ideally you should transfer those
+backups regularly to a different location. You can also disable this
+rudimentary backup strategy by deleting the backup scripts in `/etc/cron.daily`
+and replace it by a more sophisticated backup mechanism.
+
+## The web UI admin Panel
+
+The web user interface is manged by django which offers a powerful admin
+panel to manage users, user history and add additional links as so called
+flat pages. More information on the django admin panel can be found
+[online](https://www.tutorialspoint.com/django/django_admin_interface.htm).
+To access the admin panel you will first have to login as user `freva-admin`
+using the *master* password you've created during deployment. Once logged in
+as freva-admin you should navigate to `/admin` of your homepage. For example
+`https://www.clex.nci.org.au/admin` assuming the web page url is
+`https://www.clex.nci.org.au`. Here you can transfer admin rights to other
+users, like your user name. Or create
+[new flat-pages](https://docs.djangoproject.com/en/4.0/ref/contrib/flatpages/).
+Flat-pages are useful to create additional project specific information that
+can be displayed on the website. This information can be links to plugin
+documentation or legal notes (see below).
+
+
+### Adding privacy notes and terms of conditions to the Web UI
+For some incitations it might be necessary to add legal statements on privacy
+and terms of usage. Those statements can be added as flat-pages in the django
+*admin* panel. All flat-pages starting with the url **/legal** for example
+`/legal/privacy/` will be added as a link to the footer of home page. Examples
+of the content of such flat-pages can be found in the [Appendix III](LegalNotes.html).

docs/Configure.md

@@ -69,62 +69,6 @@ optional arguments:
 
 The `--steps` flags can be used if not all services should be deployed.
 
-## After successful deployment
-
-## Activation scripts for users
-The core deployment will create activation scripts for various shell flavours.
-The following activation scripts are located in the `<root_dir>/freva` folder:
-* *activate_sh*: To be sourced by shell flavours like shell, bash, zsh etc.
-* *activate_csh*: To be sourced by c-shell flavours like csh, tcsh
-* *activate_fish*: To be source by fish
-* *loadfreva.module*: Modules file
-The source scripts can either be copied where they are automatically sourced
-or users can be instructed to use the source command for their shell. If
-`modules` is available on the host system you can copy the *loadfreva.modules*
-file into the `MODULEPATH` location.
-
-
-### Systemd units for all services:
-If the target machine where the services (solr, mariadb, web) were deployed
-are Linux machines you will have access to a `systemd` unit of the created
-service. In general the services can be accessed by
-`<project_name>-<service_name>.service` If for example the `project_name`
-key was set to `clex-ces` then the following services are created:
-
-- database: `clex-ces-db.service`, `clex-ces-vault.serice`
-- solr: `clex-ces.solr.service`
-- web ui: `clex-ces-web.service`, `clex-ces-redis.service`, `clex-ces-httpd.service`
-
-### Access of service data on the host machine
-The data of the services, like the database or solr cores are stored "outside"
-the docker containers on the host machine and can be accessed at
-`/opt/freva/<project_name>/<service_name>_service/` for example
-`/opt/freva/clex-ces/db_service`
-
-
-### Simple backup scripts:
-The `db` and `solr` services offer a very simple backup script that can
-be run from outside the container. To issue a backup command simply call the
-following command `docker/podman exec -it <project_name>-solr/db /usr/local/bin/daily_backup`.
-For example:
-
-```
-podman exec -it clex-ces-solr /usr/local/bin/daily_backup
-```
-
-If you have `anacron` set up on your host machine then a cron script to
-automatically backup databases and solr cores is applied nightly.
-By default the script keeps the last 7 backups. Backup data can be found in:
-
-- `db`:`/opt/freva/<project_name>/db_service>/backup`
-- `solr`: `/opt/freva/<project_name>/solr_service/<core_name>/data/snapshot.YYYYMMDDHHMMSSMS`
-
-This is only a rudimentary backup solution, ideally you should transfer those
-backups regularly to a different location. You can also disable this
-rudimentary backup strategy by deleting the backup scripts in `/etc/cron.daily`
-and replace it by a more sophisticated backup mechanism.
-
-
 ## Known Issues:
 Below are possible solutions to some known issues:
 

docs/LegalNotes.md

+# Appendix III: Legal notes as flat-pages
+
+This appendix gives examples of flat pages containing privacy and terms of
+conditions. Such statements should be added as flat-pages in the [django admin
+panel](AfterDeployment.html#the-web-ui-admin-panel). Any flat-page url starting with `/legal/` will
+be added as a link to the footer. For example `/legal/privacy/` which could
+be a flat-page for a privacy statement. Below you'll find examples of a
+privacy statement and terms of use.
+
+### Privacy statement
+
+```html
+
+<h1 id="privacy-policy">Privacy Policy</h1>
+<p>The internet sites under the current main website page (hereinafter: "Website") are operated by Deutsches Klimarechenzentrum GmbH (hereinafter: "DKRZ"), Bundesstraße 45a, 20146 Hamburg. The careful and legitimate handling of your data is an important concern for us. The following Privacy Policy explain which personal data are gathered on our website and how and for which purposes these are used.</p>
+<p>Please, read the Privacy Policy completely before you transmit data to us or use our website in other manner and inform yourself about updates of these Data protection notices on a regular basis. The following Privacy Policy is valid for the use of our website in its respective current version.</p>
+<h3 id="automatic-collection-of-data">Automatic Collection of data</h3>
+<p>When you visit our website, the following data are automatically gathered exclusively for the creation of anonymous statistics as well as for the purposes of data protection monitoring, data security and to ensure the proper operation of our data processing systems:</p>
+<ul>
+<li><p>The IP address of your computer</p></li>
+<li><p>Date and time of the visit</p></li>
+<li><p>Operating system and the browser software of your computer</p></li>
+<li><p>Volume of data transmitted</p></li>
+<li><p>Internet address of the website from which you have reached this website</p></li>
+<li><p>Domain and Host, from which you are using the Internet</p></li>
+</ul>
+<h3 id="other-collection-processing-and-use-of-personal-data">Other collection, processing and use of personal data</h3>
+<p>As a rule, you can visit our website without us requiring, collecting, processing or using your personal data. You will remain anonymous as an individual user or at least pseudonymous.</p>
+<p>Personal data are only collected if you provide them to us voluntarily, in particular during registration and use of the current instance of the Free Evaluation System Framework (Freva) website.</p>
+<p>Your information will not be disclosed to third parties and is used exclusively for the execution of these services and for processing your request and remain stored only up to the final processing of the matter of the case and in compliance with statutory retention periods.</p>
+<h3 id="freva-registration-and-use">Freva: Registration and use</h3>
+<p>To be able to use the current instance of Freva, you must register as a user via <a href="https://luv.dkrz.de/">DKRZ's LDAP system</a>. On this occasion, we also collect personal data like given name (First Name), surname, e-mail address, user name as well as, optionally: the second given name (Middle Name), Organisation, City (State -) country (State), country (Country). Besides, you must select a separate password; this is stored encoded, of course. The thus collected information is stored on a server of the DKRZ.</p>
+<p>The thus collected data are stored only on our servers for the purpose and for the duration of the use of the Freva server operated by us and are not disclosed to any third party.</p>
+<h3 id="withdrawal-of-consent-for-the-collection-processing-and-use-of-your-data">Withdrawal of consent for the collection, processing and use of your data</h3>
+<p>You have the option to revoke your consent for the collection, processing and use of personal data at any time with effect for the future. A revocation must be sent to:</p>
+<p><br />
+Deutsches Klimarechenzentrum GmbH<br />
+Bundesstraße 45a<br />
+D-20146 Hamburg<br />
+e-mail: <a href="mailto:data@dkrz.de" class="email">data@dkrz.de</a><br />
+Telephone: +49404600940<br />
+Telefax: +4940460094270</p>
+<h3 id="right-to-access-correction-blocking-and-deletion-of-your-data">Right to access, correction, blocking and deletion of your data</h3>
+<p>You are entitled to receive, free of charge, information about the personal data stored about you. On request, we will share with you in writing, in accordance with the applicable law, whether and which personal data about you are stored with us.</p>
+<p>In addition, in accordance with the legal provisions, you have a right to correction, blocking and deletion of your personal data.</p>
+<h3 id="changes-to-the-contents-of-this-data-protection-notice">Changes to the contents of this Data protection notice</h3>
+<p>DKRZ reserves the right to change the contents of these Privacy Policy every now and then in accordance with the legal data protection regulations.</p>
+```
+
+
+### Terms of use
+
+
+```html
+<h1 id="terms-of-use">Terms of Use</h1>
+<p>Deutsches Klimarechenzentrum GmbH (hereinafter: "DKRZ"), Bundesstraße 45a, 20146 Hamburg, provides an Internet platform under this website and all the sites under it, with which the users are given access to the web version of the current Free Evaluation System Framework instance (hereinafter: Freva). Freva is a standardized data and evaluation system for high performance computing that provides access to a vast model and observational database as well as to tools to analyze them.</p>
+<h3 id="availability-of-the-website">Availability of the Website</h3>
+<p>DKRZ makes every effort to ensure the constant availability of the website. However, it is not impossible that - for technical reasons or for maintenance work purposes - this website will be temporarily unavailable from time to time, and this without prior notice. A right of the user to a continuous accessibility of the content of the website does not exist.</p>
+<h3 id="use-of-freva">Use of Freva</h3>
+<p>The use of Freva via our website is only allowed under protection of the copyrights and other rights of third parties. DKRZ is not obliged to verify or to monitor User Content in terms of completeness, accuracy, timeliness, non-infringement and as to its compatibility with legal provisions or other mandatory rules.</p>
+<p>The user is not permitted to upload contents that violate the rights of third party parties. With content upload, the user declares to have the necessary rights to use the content in question and not to violate any laws or other mandatory rules with the contents. In case of infringement of the law, he is obliged to indemnify DKRZ against third party claims; this includes the costs for a legal defense through DKRZ.</p>
+<p>Data Portal users may create individual copying (e.g. through download and printing out) for private use of the retrievable and downloadable contents there. In addition, copies are particularly permissible for scientific use if and as far as the reproduction is necessary for that purpose and serves no commercial purposes.</p>
+<p>Further rights are not granted.</p>
+<h3 id="liability-principles">Liability principles</h3>
+<p>The visit and the use of the website is done on own responsibility of the user. A liability of the DKRZ for slightly negligent breaches of obligations is excluded, provided that this does not concern damage arising from injury to life, body or health or claims according to the product liability law. This shall furthermore not affect the liability for the breach of obligations whose fulfilment makes the proper execution of the contract generally possible in the first place and the observance of which the users may usually rely on ("cardinal contractual obligations"). However, in the case of violation of cardinal contractual obligations, compensation claims of the user are limited to compensation for contract typical, predictable damage. The same liability restrictions are valid for breaches of obligations by the legal representatives and vicarious agents of DKRZ.</p>
+<p>DKRZ assumes as part of the above restrictions in particular no liability for unauthorized access by third parties to the contents of the website or manipulation by third parties as well as for a possible transmission of computer viruses, computer worms, Trojans or other malicious programs. The user is obliged to undertake own precautions for the protection of his own computer against such malicious programs, in particular to install anti-virus software and to use the respectively current antivirus definitions.</p>
+<h3 id="liability-for-third-party-content">Liability for Third Party Content</h3>
+<p>DKRZ shall only be liable for contents published by users or, otherwise, by a third party ("Third Party Content") according to the principles named in Clause 3. This applies in particular for all climatology data retrievable via this website, no matter whether these are stored on our own or external servers. DKRZ is not obliged to monitor third-party content as to the completeness, accuracy, timeliness, non-infringement and for compliance with statutory provisions, other mandatory rules or for compliance with professional ethics. Accordingly, DKRZ also takes over no guarantee or responsibility for the completeness, accuracy, timeliness or legality of third party contents.</p>
+<h3 id="liability-for-links">Liability for Links</h3>
+<p>The website contains references ("Links") to contents of external websites. The placing of such links is done exclusively for information purposes and as an additional service for the users of the website, without DKRZ thereby wanting to express a recommendation, approval or endorsement with respect to the linked content or without wanting to adopt the linked content as its own. At the time of DKRZ placing the links, there were no illegal contents recognizable on the internet sites linked to. However, DKRZ has influence neither on the creation nor the contents of the linked websites. Rather, their respective operator is exclusively responsible for the creation and contents of the linked websites. Hence, DKRZ assumes no guarantee or responsibility for the existence, the accuracy, completeness, timeliness or legality of the design and content of linked websites.</p>
+<h3 id="copyright">Copyright</h3>
+<p>As far as the contents on this website were not created by DKRZ, the copyrights of third parties have been observed. Concerning the contents of the current Freva website instance, Clause 2 of these Terms of Use is valid in addition. Should you become aware of copyright infringement, we ask you to inform us. Upon becoming aware of an infringement, we will undertake all reasonable steps in order to prevent this.</p>
+<h3 id="data-protectionpersonal-information">Data protection/Personal information</h3>
+<p>Information about the collection and processing of your personal data during the use of the website can be found in the Privacy Policy.</p>
+<h3 id="applicable-law-and-place-of-jurisdiction">Applicable law and Place of Jurisdiction</h3>
+<p>The law of the Federal Republic of Germany shall apply. If you are using the services on this website as a consumer and have your residence in another country at the time, the application of mandatory law of that country remains unaffected by the choice of law made in sentence 1.</p>
+<p>Place of performance and Place of Jurisdiction for all disputes from and in connection with the offer of this website is Hamburg for the use by merchants, legal entities under public law or public law special funds. For the rest, the applicable legal provisions are valid for the local and international jurisdiction.</p>
+
+```

docs/index.rst

@@ -49,9 +49,11 @@ basic deployment setup and servicing Freva will be introduced.
    Folders
    Installation
    Configure
+   AfterDeployment
    Services
    TuiHowto
    Transition
+   LegalNotes
    modules