diff --git a/notebooks/tzis-tutorial.ipynb b/notebooks/tzis-tutorial.ipynb index 125309b8ed0f55a69060fd9db1620125ef9594cf..e0c479d621cb9309c0b36b9afd506cc6d5c73099 100644 --- a/notebooks/tzis-tutorial.ipynb +++ b/notebooks/tzis-tutorial.ipynb @@ -384,6 +384,20 @@ "metadata": { "tags": [] }, + "source": [ + "## Parallelization\n", + "\n", + "Dask is used to parallelize the `write_zarr` function. Tzis tries to get a running client with dask's `get_client` function. If that fails, it opens a local client with half of the maximum available CPU and memory resources. \n", + "\n", + "Dask will process a default batch of 70 chunks at once. Most of the time, tzis resizes the batch so that it is most likely that the memory is not overloaded." + ] + }, + { + "cell_type": "markdown", + "id": "4ec0cf68-ebb9-455a-b215-f58b4ff24a06", + "metadata": { + "tags": [] + }, "source": [ "## Chunking - best practices\n", "\n", @@ -617,7 +631,13 @@ "xarry.open_zarr(zarrinput, consolidated=True, decode_times=True)\n", "```\n", "\n", - "This is possible if the container is *public*.\n", + "This is possible if the container is *public* (see *how to make a container public*).\n", + "\n", + "Tzis automatically generate an index file which contains URLs for all prefixes of the container. The location is \n", + "\n", + "```python\n", + "'/'.join([OS_STORAGE_URL,os_container,\"INDEX.csv\"])\n", + "```\n", "\n", "1. If your container is *private*, you have to use tokens and fsspec. If you create environment variables for OS_AUTH_TOKEN and OS_STORAGE_URL, you can get a fsspec mapper which than can be used for xarray:\n", "\n", @@ -779,7 +799,7 @@ "tzis.toggle_public(outstore)\n", "```\n", "\n", - "This will either make the container of the outstore *public* if it was not or it will make it *private* by removing all access control lists if it was public. Note that only container as a whole can be made public or private.\n", + "This will either make the container of the outstore *public* if it was not public or it will make it *private* by removing all access control lists if it was public. Note that only container as a whole can be made public or private.\n", "\n", "- With hand:\n", "\n", diff --git a/tests/test_validations.ipynb b/tests/test_validations.ipynb index 6abb290ce074719a524e93933091e427e501d0fb..cda0016e82c050befe432e9f88cb3708cd551aa3 100644 --- a/tests/test_validations.ipynb +++ b/tests/test_validations.ipynb @@ -58,7 +58,7 @@ "\n", "def get_encoding(zarr_dset):\n", " encoding = {var:zarr_dset[var].encoding \n", - " for var in list(ds.variables.keys())}\n", + " for var in list(zarr_dset.variables.keys())}\n", " for var in zarr_dset.data_vars:\n", " encoding[var].update(dict(zlib=True, complevel=1))\n", " for var in zarr_dset.coords:\n",