broken job: Data Extraction Failure shape files more than 100 features and more than one variable
Description
There's an issue with the data extraction process in the climate impact plugin, specifically relating to the handling of shape files and the temporary storage of extracted data. Despite the Jupyter notebook processing step In[3]
completing successfully, indicating that the data was extracted well, the subsequent steps fail to locate the extracted data in the /tmp
directory.
Configuration Used
freva-plugin climpact shape_file=/work/ch1187/regiklim-work/b381987/lme_rechengitter_lcc_rhein_epsg3034_10006-11142.geojson split_by=CELL_CODE project=nukleus product=ceu-3 experiment=ssp370-gwl2k models=mpi-m-mpi-esm1-2-hr-clmcom-btu-icon-2-6-5-rc-nukleus-x2yn2-v1 time_frequency=day variable=hurs output_units=true output_file_type=nc mask_type=none mask_method=centres plot_map=true pre_process_function=pre_proc
Relevant Logs and Files
- Log URL: Job Information
-
Jupyter Notebook:
/work/ch1187/regiklim-work/b381987/regiklim-ces/output/climpact/3950/PostProcessing.ipynb
- Shape File: Attached file lme_rechengitter_lcc_rhein_epsg3034_10006-11142.geojson
Observed Error
File src/netCDF4/_netCDF4.pyx:2307, in netCDF4._netCDF4.Dataset.__init__()
File src/netCDF4/_netCDF4.pyx:1925, in netCDF4._netCDF4._ensure_nc_success()
FileNotFoundError: [Errno 2] No such file or directory: b'/tmp/tmp8mpuzr2x/Time_series_ceu-3_mpi-m-mpi-esm1-2-hr-clmcom-btu-icon-2-6-5-rc-nukleus-x2yn2-v1_r1i1p1f1_ssp370-gwl2k_day_3677526725.nc'
Possible Cause
The issue seems to be related to a storage limitation problem, preventing the extracted data from being found in the /tmp
directory.
Expected Behavior
The data extracted from the shape file should be temporarily stored in the /tmp
directory and be accessible for further processing steps in the workflow.
Steps to Reproduce
- Easy way to reproduce the error would be using the freva-web to re-lunch the job, but the other way would be running the configuration in
levante.dkrz.de
Addition
After resolving this issue, the fix should be incorporated into the unit tests to prevent future occurrences of similar problems.