Commit bab16516 authored by Marco Kulüke's avatar Marco Kulüke
Browse files

weekend push

parent 2a368e2d
......@@ -12,7 +12,7 @@
" 1. Parallelism\n",
" 1. Overview\n",
" 1. `dask.delayed`\n",
"2. Which data can be processed with dask?\n",
"2. Process climate data with dask?\n",
"3. Common mistakes"
]
},
......@@ -76,39 +76,6 @@
"import time"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"@delayed\n",
"def add(x, y):\n",
" result = x +y\n",
" return result"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"output = []\n",
"for i in range(0,11):\n",
" output.append(i)\n",
"result_delayed = delayed(sum)(output)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"result_delayed.visualize()"
]
},
{
"cell_type": "markdown",
"metadata": {},
......@@ -136,7 +103,7 @@
" time.sleep(0.5)\n",
" return x + y\n",
"\n",
"data = list(range(10))\n",
"data = list(range(4))\n",
"\n",
"output = []\n",
"for x in data:\n",
......@@ -172,12 +139,12 @@
" time.sleep(0.5)\n",
" return 2 * x\n",
"\n",
"@delayed\n",
"#@delayed\n",
"def add(x, y):\n",
" time.sleep(0.5)\n",
" return x + y\n",
"\n",
"data = list(range(10))\n",
"data = list(range(4))\n",
"\n",
"output = []\n",
"for x in data:\n",
......@@ -186,8 +153,9 @@
" c = add(a, b)\n",
" output.append(c)\n",
"\n",
"total_delayed = delayed(sum)(output) #also delay sum because it is a function\n",
"%time total_delayed.compute()"
"total_delayed = sum(output) #also delay sum because it is a function\n",
"#%time total_delayed.compute()\n",
"total_delayed.visualize()"
]
},
{
......@@ -205,7 +173,24 @@
"metadata": {},
"outputs": [],
"source": [
"result_delayed.compute()"
"client.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Looks good in theory, right? Now, let us apply our knowldege to climate model data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Process Climate Data with Dask\n",
"- load with intake\n",
"- think about processing\n",
"- compare conventional with dask"
]
},
{
......@@ -213,9 +198,7 @@
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"client.close()"
]
"source": []
},
{
"cell_type": "markdown",
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment