[Fortran] Unittests with FortUTF

changed target branch from master to cmake_for_tests

mentioned in merge request !5 (closed)

@m300488 I am responsible for almost all unittests in this MR. Any suggestion/extension wished from your side?

added 1 commit

• f3838dee - write table to file

Compare with previous version

mentioned in merge request !14 (merged)

deleted the cmake_for_tests branch. This merge request now targets the master branch

added 3 commits

• f3838dee...f3ea9970 - 2 commits from branch master
• a233ace8 - Merge branch 'master' of gitlab.dkrz.de:icon-libraries/libfortran-support into fortran_tests

Compare with previous version

added 1 commit

• a745f6cd - add folder with tests

Compare with previous version

@m300488 Any comments here?

I don't think PROJECT_NAME is a variable that is supposed to be set directly.

changed this line in version 8 of the diff

You are right, but the problem is the following:

/scratch/b/b381001/EXCLAIM/unittest/new/test/fortran/run_tests.f90:1:8:

    1 | PROGRAM TEST_fortran-support
      |        1
Error: Invalid form of PROGRAM statement at (1)

PROJECT_NAME is used as a program name for the test-exe. And - is not allowed for fortran-programs.

Therefore we overwrite it here.

Do you have a smart idea to fix that?

See here: https://github.com/artemis-beta/FortUTF/blob/9f70d03c5d92c4023b1eab312af18ff56790c00a/cmake/fortutf.cmake#L80C45-L80C45

I see. I have a couple of hints if you ever decide to get this fixed upstream:

1. It's questionable whether PROJECT_NAME has to appear in the generated source code, i.e. PROGRAM TEST does not seem any worse than PROGRAM TEST_fortran_support.
2. A much better way to generate a source file is to add a template file to the repo and then generate the required source file using the configure_file function.

Thank you for the hints.

I wonder what it will take to come up with a minimal reproducer of a problem identified with one of the tests (e.g. in order to submit it to a compiler vendor). There is a single executable for all tests if I am not mistaken. If one of the tests fails, the whole executable fails and the output of make test won't tell me much. I will have to dig deeper just to identify the failed test. Am I correct? I might be too conservative but I prefer having a separate executable for each individual test.

To be honest, I am not convinced that FortUTF is the way to go. Why don't we just extend the helpers.f90 with ASSERT_EQUAL and friends and just use it as a library in separate executables? All those GLOBs and REGEX MATCHALLs look fishy to me, to be honest.

You can run the test-exe from FortUTF. It dumps you the tests that fail, see https://pad.gwdg.de/WLGZz0LKSJmvqUrEbNkCug?view#FortUTF

That's the way FortUTF is designed!

About your general critisism: We documented our decision quite well here https://pad.gwdg.de/WLGZz0LKSJmvqUrEbNkCug?# and here https://gitlab.dkrz.de/icon/icon-c/-/issues/67

Your feedback is very welcome but a bit late.

Of course we could come up with our own solution, but do YOU have time to do that? I don't! Also I personally liked some features of FortUTF like the test-collection and the relatively easy CMake-integration. And would our solution really be much better then FortUTF? I doubt.

But as you see in the tests, not much infrastructure is needed by FortUTF. Basically only the asserts need to be adapted to another framework. And this portability was also a plus to choose fortutf. I.e. pfunit comes along with its own fortra-decorator and preprocessor, this makes a transition much harder.

You can run the test-exe from FortUTF. It dumps you the tests that fail, see https://pad.gwdg.de/WLGZz0LKSJmvqUrEbNkCug?view#FortUTF

So, an extra digging step if something goes wrong.

That's the way FortUTF is designed!

Yes, it's the design that I question. Or, at least, the applicability of the tool that is designed that way. I'm not saying that it's a show-stopper but just a drawback that I expect to be compensated with the benefits of the tool.

About your general critisism: We documented our decision quite well here https://pad.gwdg.de/WLGZz0LKSJmvqUrEbNkCug?# and here https://gitlab.dkrz.de/icon/icon-c/-/issues/67

Thank you for the links, I will go over them one more time. Hopefully, I'll find the answer there.

Your feedback is very welcome but a bit late.

Does it make it invalid?

Of course we could come up with our own solution, but do YOU have time to do that? I don't!

It's always about the cost/benefit ratio. The question is whether you have time to support the dependency that you want to introduce. Fix bugs, create PRs, make sure they get merged, etc. If not, you implicitly put this on other people who, most probably, will find it much easier for the code that is under the community's control.

Also I personally liked some features of FortUTF like the test-collection and the relatively easy CMake-integration.

And right away we have to circumvent the PROJECT_NAME issue.

And would our solution really be much better then FortUTF? I doubt.

I am sorry but at least the CMake part does not really look like a high bar to clear.

But as you see in the tests, not much infrastructure is needed by FortUTF. Basically only the asserts need to be adapted to another framework. And this portability was also a plus to choose fortutf. I.e. pfunit comes along with its own fortra-decorator and preprocessor, this makes a transition much harder.

I trust that FortUTF is the best tool for our needs out there. However, the best does not mean good but just that all the others are worse. Again, I'm not saying that you should discard all your work and close this MR, I'm just making you aware of certain potential issues that I have in mind.

Why not using ctest --output-on-failure instead of make test?

This makes an extra step obsolete.

Your feedback is valid of course. I appreciate that you take your time and really dig to the bottom of things. Your ideas help to improve things a lot.

We discussed the maintenance as well. Conclusion was, that FortUTF is still simple, so I think I could fix small things in this repo. For other frameworks this isn't the case.

I can't judge the CMake-scripting, still a bloody beginner in this domain.

The option for Fortran unittesting aren't great, I agree.

I think we have to make some experiences and see how things evolve. Maybe FortUTF turn into a nightmare, then we need to evaluate again.

added 1 commit

• 52169e2c - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

added 1 commit

• 01704bde - rename FortUTF to fortran, remove extra src-folder for tests

Compare with previous version