broken job: cannot get access to the /tmp directory to download from swift
Description
There's an issue with the from_zipfile
function in datacontainer.py
that cannot get the /tmp
directory.
Configuration Used
freva-plugin climpact shape_file=https://nextcloud.dkrz.de/s/HPC7GBaFZn3LDyc/download?path=%2FR2K-Klim&files=Region_R2K-Klim-plus_Microlevel.geojson project=nukleus product=ceu-3 experiment=ssp370-gwl3k models=mpi-m-mpi-esm1-2-hr-clmcom-btu-icon-2-6-5-rc-nukleus-x2yn2-v1 time_frequency=1hr variable=sfcwind output_units=true output_file_type=nc mask_type=none mask_method=centres plot_map=true pre_process_function=pre_proc
Relevant Logs and Files
- Log URL: Job Information
-
Jupyter Notebook:
/work/ch1187/regiklim-work/k202187/regiklim-ces/output/climpact/4049/PostPorcessing.ipynb
- Shape File: Attached file Region_R2K-Klim-plus_Microlevel.geojson
Observed Error
FileNotFoundError Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 dc = DataContainer.from_zipfile(data_file)
2 dc
File /work/ch1187/freva-regiklim/plugins/climpact/src/climpact/datacontainer.py:453, in DataContainer.from_zipfile(cls, *zipfile)
450 zarr_files += [f for f in Path(".").rglob(str(z_file))]
452 datasets = []
--> 453 for dataset in [cls._read_zipfile(zf) for zf in zip_files]:
454 datasets += dataset
455 for dataset in [cls._read_zarr(zf) for zf in zarr_files]:
File /work/ch1187/freva-regiklim/plugins/climpact/src/climpact/datacontainer.py:453, in <listcomp>(.0)
450 zarr_files += [f for f in Path(".").rglob(str(z_file))]
452 datasets = []
--> 453 for dataset in [cls._read_zipfile(zf) for zf in zip_files]:
454 datasets += dataset
455 for dataset in [cls._read_zarr(zf) for zf in zarr_files]:
File /work/ch1187/freva-regiklim/plugins/climpact/src/climpact/datacontainer.py:341, in DataContainer._read_zipfile(cls, z_file)
337 """Read the content of a zipfile."""
338 datasets = []
339 with TemporaryDirectory(
340 dir=os.environ.get("TEMP_DIR")
...
--> 368 _os.mkdir(file, 0o700)
369 except FileExistsError:
370 continue # try again
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpa9ycbo85/scratch/k/k202187'
Possible Cause
The issue seems to be related to a finding the expected /tmp
directory.
Expected Behavior
The data extracted from the shape file should be temporarily downloaded from swift cloud and be stored in the /tmp
directory and be accessible for further processing steps in the workflow.
Steps to Reproduce
- Easy way to reproduce the error would be using the freva-web to re-lunch the job, but the other way would be running the configuration in
levante.dkrz.de
Addition
After migrating to the plugin template, all functions in this plugin need to be added to the unit-test to prevent getting these generic errors.