Migrate to Zarr 3

Pre-work to #6

This MR has the following:

1. Migration to Zarr 3 API while removing the deprecated yet still supported Zarr 2 API usage.
2. Essential fixes for the main-branch resulting from the learnings out of the global-hackathon exploits (The non-essential ones will be fixed in a seperate branch).

added hiopy label

added 3 commits

• 224cd2cc - added argument to allow chunks-per-shard while configuring the dataset
• 6cb14f31 - added some documentation to the create-dataset helper functions
• 16fcef06 - bumped zarr required version to 3 and updated the scripts to use python3.11

Compare with previous version

added 6 commits

• 42fa6953 - fixed the right value to be passed for chunk-shape in zarr 3
• bf843f72 - fixed chunk-shape calculation
• 15f31938 - changed the order of imports in the worker as the numpy (built with different...
• fa991350 - zarr 3 fixes
• 9f19bda2 - fixed the py-linting and turned off alphabetical sorting of imports due to an...
• d5de118e - Ran pre-commit hook to fix issues from last changes

Compare with previous version

added 3 commits

• 8623e414 - bumped the python version to 3.11 with a local python installation referenced in scripts
• ea7d8e57 - added rich as a mandatory dependency
• c813e519 - fixing zarr version to v3.0.6 for essential bug fixes

Compare with previous version

added 1 commit

• f8fba528 - fixing zarr version to v3.0.6 for essential bug fixes

Compare with previous version

added 1 commit

• cf118b6b - Fixes for zarr-3 which slipped through

Compare with previous version

added 3 commits

• d8427fd6 - fix: copy_metadata
• 5bff601f - fix: keep all data_handlers for final flush
• 43685227 - perf: optimize zarr metadata acces further

Compare with previous version

added 4 commits

• 8ec2c15d - Revert "perf: optimize zarr metadata acces further"
• 583480f6 - fix: avoid reopen the zarr again and again
• 98b57fe5 - fix: datetime shift
• 17f9c2ea - reconsolidating zmetadata after pulling metadata for fields via yac

Compare with previous version

marked this merge request as ready

requested review from @k202160

changed the description

assigned to @k202160

added 1 commit

• bbde92e3 - reconsolidating zmetadata after pulling metadata for fields via yac

Compare with previous version

Why did you revert 43685227? This is contained in global_hackathon isn't it?

resolved all threads

added 3 commits

• 1c56757c - fix: avoid reopen the zarr again and again
• f8a0fc30 - fix: datetime shift
• b269ec2f - reconsolidating zmetadata after pulling metadata for fields via yac

Compare with previous version

enabled an automatic merge when all merge checks for b269ec2f pass

aborted the automatic merge because the source branch was updated. Learn more.

added 1 commit

• 9a8307db - ci: added python3.11 to project work accessible across developers (to be...

Compare with previous version

merged