No big deal. My CI was breaking and I dropped now support for python 3.8 in slkspec.
The syntax used in https://gitlab.dkrz.de/hsm-tools/pyslk/-/blob/master/pyslk/constants.py#L29 is failing with python 3.8. This syntax has only been introduced in python 3.9 (https://docs.python.org/3/whatsnew/3.9.html#type-hinting-generics-in-standard-collections).
I agree, let's wait for the slk
fix.
Sry, I need to correct myself. The behavior is now as expected. The file is no longer saved in a sub-folder structure, but in the base directory. However, the sub-folder structure is still created.
So after executing
import pyslk
search_id = pyslk.core.search('{"$and":[{"path":{"$gte":"/arch/mh0010/m300408/showcase/dataset.zarr","$max_depth":1}},{"resources.name":{"$regex":".zmetadata"}}]}')
pyslk.slk_retrieve(search_id, "/scratch/m/m300408/testretrieval/")
I get the following folder structure:
ls -aR /scratch/m/m300408/testretrieval/
/scratch/m/m300408/testretrieval/:
. .. arch .zmetadata
/scratch/m/m300408/testretrieval/arch:
. .. mh0010
/scratch/m/m300408/testretrieval/arch/mh0010:
. .. m300408
/scratch/m/m300408/testretrieval/arch/mh0010/m300408:
. .. showcase
/scratch/m/m300408/testretrieval/arch/mh0010/m300408/showcase:
. .. dataset.zarr
/scratch/m/m300408/testretrieval/arch/mh0010/m300408/showcase/dataset.zarr:
. ..
So .zmetadata
is at the right place. But the /arch
-folder shouldn't have been created.
The pyslk.slk_retrieve(SEARCH_ID, TARGET_PATH)
continues to behave inconsistently with pyslk.slk_retrieve(SOURCE_PATH, TARGET_PATH)
and creates the entire sub-folder structure, although preserve_path
is set False.
@k204221, I tested this with pyslk v1.8.0 now, but the behavior remains the same.
import pyslk
pyslk.slk_retrieve("/arch/mh0010/m300408/showcase/dataset.zarr/.zmetadata", "/scratch/m/m300408/testretrieval/")
downloads the file .zmetadata
to /scratch/m/m300408/testretrieval/
. Because the preserve_path
option is by default False
, this is the expected behavior.
However, in case I request the file not via the path but the search id, like so:
import pyslk
search_id = pyslk.slk_search_int('{"$and":[{"path":{"$gte":"/arch/mh0010/m300408/showcase/dataset.zarr","$max_depth":1}},{"resources.name":{"$regex":".zmetadata"}}]}')
pyslk.slk_retrieve(search_id, "/scratch/m/m300408/testretrieval/")
the file is downloaded to /scratch/m/m300408/testretrieval/arch/mh0010/m300408/showcase/dataset.zarr/.zmetadata
. This was unexpected because preserve_path
is still False
by default.
The latter seems to only occur since a recent update of pyslk/slk
. I observed this as it currently breaks slkspec
. I can easily adjust it, but want to make sure that the adjustment would last.
>>> pyslk.__version__
'1.6.2'
>>> module load slk
Autoloading openjdk/17.0.0_35-gcc-11.2.0
Autoloading slk_helpers/1.7.3
Loading slk/3.3.83
Loading requirement: openjdk/17.0.0_35-gcc-11.2.0 slk_helpers/1.7.3
Thanks @k204221 ! The issue is fixed for me now!
@m221078 is there a way I can link my own clone of pyslk
to uncompresm until this issue is resolved?
I also encountered this issue with unpackems
. However, here I do not know how to fix it:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ STARTED UNPACKEMS ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Oops: StrongLink login token not valid. message: 'pyslk: slkh: command not found'. Please refresh/create your login token via 'slk login'.
CC: @m221078 My slk login token is valid. I assume unpackems just interprets the raised error incorrectly.
Hi, I think I just ran into this issue.
Example
module load slk
from pyslk import pyslk
pyslk.slk_gen_file_query(["/arch/mh0010/m300408/experiments/EUREC4A/highCCN/output/DOM02/surface/EUREC4A_ICON-LES_highCCN_DOM02_surface_native.clct.001.tar", "/arch/mh0010/m300408/experiments/EUREC4A/highCCN/output/DOM02/surface/EUREC4A_ICON-LES_highCCN_DOM02_surface_native.coords.001.tar", "/arch/mh0010/m300408/experiments/EUREC4A/highCCN/output/DOM02/surface/EUREC4A_ICON-LES_highCCN_DOM02_surface_native.lhfl_s.001.tar"])
returned
File ~/.conda/envs/slkspec/lib/python3.10/site-packages/pyslk/pyslk.py:960, in slk_gen_file_query(resources, recursive)
957 funName = inspect.stack()[0][3]
959 if shutil.which(SLK_HELPERS) is None:
--> 960 raise PySlkException("pyslk: "+SLK_HELPERS+": command not found")
962 slk_call = [SLK_HELPERS, "gen_file_query"]
963 if (recursive):
PySlkException: pyslk: slkh: command not found
Thanks to the solution that @k204210 suggested (pyslk.SLK_HELPERS="slk_helpers
), I made it working. Would be great if this could be fixed in the package itself.
Keep up the great work!
Cheers, Hauke