[hiopy] buffer to disk
Motivation
We want to enable hiopy to write larger time chunks. Currently this is limited by memory consumption.
Task description
Instead of collection the timesteps for each variable in the memory, we could store the data on some hard drive (/fastdata
?). At the timestep where the time chunk is full, hiopy reads all timesteps from the hard drive again and stores it into the zarr dataset (variable by variable). This would require less memory.
-
Extend the
DataHandler
with an additional buffer type. -
Add a "configuration attribute" to the variables (
hiopy::buffer_to_disk
) that contains the path for storing the data. (If it is not set the old behavior is used) - Add documentation
- Add a test to CI
- Test it with ICON on levante