JOANNE dataset

I don't know if this belongs here: when I did the UX homework and tried to follow this example, the command

cat = eurec4a.get_intake_catalog(use_ipfs="QmahMN2wgPauHYkkiTGoG2TpPBmj3p5FoYJAq9uE9iXT9N")

takes ages without throwing an exception or doing anything. However, if I switch it for

cat = eurec4a.get_intake_catalog() 

it works fine. Just in case anyone else had similar problems. Do you know why this could happen?

This is due to the use of IPFS. If no argument is passed, get_intake_catalog() defaults to using the current intake catalog hosted on the EUREC4A servers (see code), which is fast.

However, in the first example, a specific version of the catalog is requested based on its content id. Therefore, you need to find a participant in the IPFS network that hosts that version of the file. If I understand @m300827 correctly, this should be faster if you have a local version of the IPFS client running on your machine.

Yes, this is due to the use of IPFS. At some point in the past, we've switched over the howto EUREC4A to use IPFS by default. IPFS works best (i.e. often much faster than other methods of access), if you run a local IPFS node (as mentioned briefly on the intro page). Without a local IPFS node, you'll have to fall back to a public gateway, which provides no guarantees for performance and can indeed be incredibly slow. Unfortunately, apparently the public gateways became much slower since we've switched over to IPFS by default , so running a local node became even more important.

If you disable IPFS, you can access the "classic" intake catalog, which does not need a local IPFS node and thus is easier to set up. Some downsides of not using IPFS are: you won't have local caching, the code won't be as reproducible (as IPFS ensures that if you get data, it's exactly the same), you can't access the data when the server is down.

I can see, that from a UX standpoint, the current situation is not ideal. We could just remove the use_ipfs from the howto.eurec4a again, this would often lead to simpler access and ok-ish performance. Or we maybe have to more prominently announce the pro's and con's for IPFS and leave it open to the user. For the code on howto.eurec4a.eu IPFS does provide the advantage, that the code examples are reproducible and for that reason, we get fewer random CI failures.