Skip to content

Ingest different DRS formats

Brian Lewis requested to merge drs-specs into main

This PR is a significant rewrite of how we handle DRS file paths to allow us to ingest paths of a few different known DRS formats.

This involved:

  • creating a new crate to handle just loading the different DRS types
  • changing ingestion to load the different types and then convert them to our metadata format
  • fix some funky issues with data-dir not behaving how it should
    • I also made it optional so not having it means that all datasets are ingested

Merge request reports