D

Data

Modular satellite data pipeline built on satpy: staged file discovery, persistent Zarr caching, and transform-based processing, with PyTorch Dataset/DataLoader integration for ML training.

LimScaup turns large-scale filesystem metadata and S3 event logs into queryable Parquet data lakes, with fast CLI queries, historical snapshots, and usage rollups. Supports Lustre, VAST, and MinIO event logs.

This plugin servers as a concept study of how fair digital data objects can be accessed and processed solely using a DOI request.