Skip to content

Draft: Add batchmode

Martin Bergemann requested to merge add-batchmode into master

This adds the functionality to add support for workload managers. I've only added a base class and a class for slurm (in case we need more, for example, Kubernetes or whatnot). We could also discuss adding a non-workload manager Job class. That is spawning a job in the background.

I am not finished yet as I wanted your opinions first. I have only played with the pyslk_retreive method the rest of the functions would essentially work the same. I only need to know which methods should support the new batchmode option.

I've decided to create a blocking and a non-blocking mode. The blocking mode can be accessed via a context manager (with statement). Blocking mode means that the execution awaits the slurm job to be finished. Non-blocking doesn't wait for the job to be finished unless the run_until_completed method is called.

One problem I see is the fundamental difference of the pyslk method behaviour when using the batchmode. Without batchmode a string is returned. With batchmode a class. This needs to be documented properly - any suggestions? Maybe it would be better to always return a Job instances, I could easily implement a non workload manager class that basically uses something like nohub pyslk foo but that would probably kill backwards compatibility. Any thoughts?

@k202134 @k204210 @k204221

Merge request reports