From e968f69c3a39bd6c1786ebc788e183e736a4839a Mon Sep 17 00:00:00 2001
From: wachsylon <fabian.wachsmann90@gmx.de>
Date: Thu, 8 Dec 2022 09:14:52 +0100
Subject: [PATCH] Updated tzis
---
notebooks/tzis-tutorial.ipynb | 24 ++++++++++++++++++++++--
tests/test_validations.ipynb | 2 +-
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/notebooks/tzis-tutorial.ipynb b/notebooks/tzis-tutorial.ipynb
index 125309b..e0c479d 100644
--- a/notebooks/tzis-tutorial.ipynb
+++ b/notebooks/tzis-tutorial.ipynb
@@ -384,6 +384,20 @@
"metadata": {
"tags": []
},
+ "source": [
+ "## Parallelization\n",
+ "\n",
+ "Dask is used to parallelize the `write_zarr` function. Tzis tries to get a running client with dask's `get_client` function. If that fails, it opens a local client with half of the maximum available CPU and memory resources. \n",
+ "\n",
+ "Dask will process a default batch of 70 chunks at once. Most of the time, tzis resizes the batch so that it is most likely that the memory is not overloaded."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4ec0cf68-ebb9-455a-b215-f58b4ff24a06",
+ "metadata": {
+ "tags": []
+ },
"source": [
"## Chunking - best practices\n",
"\n",
@@ -617,7 +631,13 @@
"xarry.open_zarr(zarrinput, consolidated=True, decode_times=True)\n",
"```\n",
"\n",
- "This is possible if the container is *public*.\n",
+ "This is possible if the container is *public* (see *how to make a container public*).\n",
+ "\n",
+ "Tzis automatically generate an index file which contains URLs for all prefixes of the container. The location is \n",
+ "\n",
+ "```python\n",
+ "'/'.join([OS_STORAGE_URL,os_container,\"INDEX.csv\"])\n",
+ "```\n",
"\n",
"1. If your container is *private*, you have to use tokens and fsspec. If you create environment variables for OS_AUTH_TOKEN and OS_STORAGE_URL, you can get a fsspec mapper which than can be used for xarray:\n",
"\n",
@@ -779,7 +799,7 @@
"tzis.toggle_public(outstore)\n",
"```\n",
"\n",
- "This will either make the container of the outstore *public* if it was not or it will make it *private* by removing all access control lists if it was public. Note that only container as a whole can be made public or private.\n",
+ "This will either make the container of the outstore *public* if it was not public or it will make it *private* by removing all access control lists if it was public. Note that only container as a whole can be made public or private.\n",
"\n",
"- With hand:\n",
"\n",
diff --git a/tests/test_validations.ipynb b/tests/test_validations.ipynb
index 6abb290..cda0016 100644
--- a/tests/test_validations.ipynb
+++ b/tests/test_validations.ipynb
@@ -58,7 +58,7 @@
"\n",
"def get_encoding(zarr_dset):\n",
" encoding = {var:zarr_dset[var].encoding \n",
- " for var in list(ds.variables.keys())}\n",
+ " for var in list(zarr_dset.variables.keys())}\n",
" for var in zarr_dset.data_vars:\n",
" encoding[var].update(dict(zlib=True, complevel=1))\n",
" for var in zarr_dset.coords:\n",
--
GitLab