Add better debugging feature.

This adds small debugging functionality. That is if -v flag is set then no parallel processing is applied.

requested review from @k202187

assigned to @k204230

removed review request for @k202187

requested review from @k202206

@k202206 could you please have a look at the changes of Martin here. Most of changes are formatting stuff via black. Take your time and do it whenever you had time:) thanks a lot

approved this merge request

The main point is that logger.getEffectiveLevel() <= logging.DEBUG determines the debug status based on the logging level in Freva. When a user applies the -vvv option in the freva-plugin, it clearly indicates that debug mode is activated. However, how can the user understand that Dask will be disabled in this mode? Where do you think is the best place to explain this to the user?

Additionally, there's another concern: if we don't define a specific parameter, users won't be able to use this debug in Freva-web. So, how would you suggest we approach this for users who aren't using the freva-plugin via the command line but still want to access detailed logs?

Thanks in advance for consideration ...

I am not sure what you mean by "Dask will be disabled"? Are you referring to the fact that we do not inform the users that in debug mode we do not create dask delayed objects that open the files in parallel? Let me put it that way: If we would add a dedicated debug option. We could create jobs like this :

freva-plugin dummy debug=false --debug

And I think this is quite confusing. Hence I decided to check whether or not the debug mode is turned on. Unfortunately the --debug flag has no effect whatsoever in most plugins, so we could just disable it. But since it can be useful for plugin developers we don't want to do that.

Therefore I would say that it's fair enough to go with this --debug approach here. Mainly because in practice you'll get an email saying the plugin isn't working, can you tell me what's wrong? Then, you'd ask the person to run this thing again by logging on to Levante and directly submitting the job in --debug mode. Then you'd inspect the output.

Exactly this would be the debug workflow. I think we should not encourage users to do debugging via a workload manager job. That is to fiddle with the setup/code, submit the job to the workload manager, queue for half an hour, let it fail and see what was wrong. When I was working on climate model development at previous appointments in the past I had to do this kind of workflow and it was extremely annoying. That's a waste of time and resources. Debugging should be done from a) the command line (or an interactive python session) and b) on the login node. If a user is not able to follow these steps:

ssh user@levante.dkrz.de
module load freva
freva-plugin climpact --debug

Then they probably won't figure out what's wrong anyway, even if they have a debug option from the web.

added 1 commit

• 86069614 - Add debug note

Compare with previous version

approved this merge request

resolved all threads

@k204230 lgtm and feel free to merge it whenever you want

mentioned in commit f332026f

merged