{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Example - Reading and Writing with Dask" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import multiprocessing\n", "# Linux/OSX:\n", "import multiprocessing.popen_spawn_posix\n", "# Windows:\n", "# import multiprocessing.popen_spawn_win32\n", "import threading\n", "\n", "from dask.distributed import Client, LocalCluster, Lock\n", "\n", "import rioxarray" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tips for using dask locks:\n", "- Be careful about what lock you use for your process. It is required to have a lock for each worker, so the more fine-grained the better.\n", "- The reading and writing processes need the same type of lock. They don't have to share the same lock, but they do nead a lock of the same type.\n", "\n", "See docs for:\n", "\n", "- [Reading COGs in Parallel](read-locks.ipynb)\n", "- [rioxarray.open_rasterio](../rioxarray.rst#rioxarray-open-rasterio)\n", "- DataArray: [rio.to_raster()](../rioxarray.rst#rioxarray.raster_array.RasterArray.to_raster)\n", "- Dataset: [rio.to_raster()](../rioxarray.rst#rioxarray.raster_dataset.RasterDataset.to_raster)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### No distributed computing example\n", "Note: Without a lock provided, `to_raster` does not use dask to write to disk." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "xds = rioxarray.open_rasterio(\n", " \"../../test/test_data/compare/small_dem_3m_merged.tif\",\n", " chunks=True,\n", ")\n", "xds.rio.to_raster(\"simple_write.tif\", tiled=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multithreaded example" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "xds = rioxarray.open_rasterio(\n", " \"../../test/test_data/compare/small_dem_3m_merged.tif\",\n", " chunks=True,\n", " lock=False,\n", " # lock=threading.Lock(), # when too many file handles open\n", "xds.rio.to_raster(\n", " \"dask_thread.tif\", tiled=True, lock=threading.Lock(),\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Multiple worker example" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "with LocalCluster() as cluster, Client(cluster) as client:\n", " xds = rioxarray.open_rasterio(\n", " \"../../test/test_data/compare/small_dem_3m_merged.tif\",\n", " chunks=True,\n", " lock=False,\n", " # lock=Lock(\"rio-read\", client=client), # when too many file handles open\n", " )\n", " xds.rio.to_raster(\n", " \"dask_multiworker.tif\",\n", " tiled=True,\n", " lock=Lock(\"rio\", client=client),\n", " )" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.1" } }, "nbformat": 4, "nbformat_minor": 4 }