What’s the project’s policy on new Fortran code? Suppose there’s some feature which would be nice to have. There’s an externally developed, high-quality implementation, but it’s written in Fortran. My impression from #18566 is that we’re trying to get rid of Fortran code where possible.
In general, would we accept this?
Does it matter if the code is F77 or F90?
Does it matter if someone is offering to maintain the code if it’s added to SciPy?
(I originally asked about this on Slack, but I’m starting a discussion here so we can have a public record.)
replacing F77 with a modern implementation is welcome, particularly if it is actively maintained
There has been a huge effort to translate existing F77 code, mostly to C, and that has been worthwhile
The main point of disagreement is:
for new algorithms, should we accept any Fortran at all, even if it is actively maintained modern Fortran?
Arguments against accepting it:
it works against the goal of “no Fortran in SciPy”, an idea which was explored in Releasing (or not) 32-bit Windows wheels. This is compelling because of the build challenges which Fortran brings, and the lack of maintainers for existing Fortran in SciPy. (EDIT: see also the next comment)
actively maintained code can always become unmaintained in the future. At that point the responsibility would fall to SciPy to maintain or translate it. So there is risk involved.
Some new modern Fortran implementations are not mature and we should be careful to not start depending on projects in the “honeymoon phase”
perhaps (good) automatic translation will become possible in the future with LFortran
there is similar risk involved with code we vendor or depend on written in other languages. For some codebases, the risk (re: maintainability) with Fortran code can actually be not significantly worse than code like Boost.Math and SuperLU.
I don’t really have any opinions of my own to throw into the conversation yet, but perhaps others can expand on these points below.
I’d open with “we are slowly starting to discuss the same points”;
I think bunch of us need to, at least attempt to translate some Fortran code to understand what is what to have a more real experience.
Here are very few facts, or things objectively true about this discussion, that I can verify or provide evidence first hand after the current translation work
F77 is very bad. Zero lines of F77 should be in. F77 needs to die. Legacy or otherwise it does not matter. It has no place in modern code bases. Way too many footguns very little benefit. Too verbose due to lack of data structures and encourages you to write bad code. I don’t care about some folks who can’t move on. Not our problem.
Not every old code is not bug free. Not every old code is battle-tested.
The more we translate, the easier it gets. There is no Fortran specific benefit of code being in Fortran. Performance is identical. Fortran has array syntax superior instead of pointer math. That’s it.
Fortran code can’t be further optimized, offloaded to other hosts, or implemented with SIMD tricks (within our capacity).
Here are some things I learned about “modern fortran”
Modern fortran means nothing and multiple things. It’s an umbrella term for fortran folks to distance themselves from F77. It can be F95, 03, 08, 23 whatever. Each version has more weird syntax added and the language is driven by folks who are HPC oriented. Committee does not seek more adoption but more specialization.
There is nothing Fortran specific that makes Fortran code better. All languages converged on the same point. It’s the syntax and ergonomics make the difference.
Almost everything in modern fortran codebase is new including compilers. Flang is new, LFortran is new, gfortran stays new because of the standards. ifx is also new as opposed to ifort. This is not the case with C or C++ or Python and so on. So compiler stability is still a thing for now though it will get better over time.
We don’t need to discuss these points anymore. So let’s leave these at that. I am removing F77 slowly. So we will be left with LAPACK and PRIMA for fortran. For the remaining parts of this discussion, you can replace fortran with swift and all will be identical. Do we want swift in our codebase? It is relatively new but many people use it. It is maintained though and so on.
My thinking is the following; if PRIMA was not there, I would translate COBYLA in its buggy state. It would have been the same but at least we would not need to compile stuff in fortran. Now PRIMA is here, we are tempted to add more because it is free lunch. I would argue otherwise. It is future cost.
I don’t find this argument strong enough to repopulate SciPy with new fortran and miss this opportunity of getting fortran free. Because, then finally, we can work on what to do with BLAS and LAPACK as the standalone linear algebra issue. I am quite invested in that subject more so than SciPy F77. Because it is every language’s issue and not just Python’s.
We got PRIMA in and that’s fine. It’s not too much code (much less than what we already translated) and can be translated. We do track upstream codebases and it will not be the first one in case double code tracking is an issue.
I recommend folks to at least translate one function of their choosing from the fortran codebase to check how hard this work is or what it entails. Just even in their own time and not to commit to any codebase. Because there is so much mythical thinking involved until you actually do it. And it is not really different than regular Python work. It is just very annoying.
My view is somewhat aligned to @ilayn. I would much prefer to have no new Fortran additions to SciPy (modulo the replacement for the historic cobyla code).
I greatly appreciate the efforts that everyone in the Fortran community has done to contribute to scipy in toolchains and code to this point, and into the future. The BLAS/LAPACK libraries aren’t going to be shifting from this anytime soon.
My reasons have largely been enumerated:
there is little Fortran knowledge within the active scipy maintainer community
we therefore find it difficult/impossible to address bugs related to Fortran ourselves (especially with the ancient spaghetti code that is gradually being removed). It’s vital for projects to be able to make fixes themselves, otherwise one is always dependent on upstream/external devs with an uncertain lag time. It’s frustrating to have a whole load of bugs related to these kind of things and not be able to do anything about them. I’ve spent way too much time trying to decipher SLSQP/LBFGSB without getting anywhere.
with new code additions there’s no easy ability to review the code (lack of knowledge), so we have to take it on trust that it’s robust. I’m not sure if upstream project size is related to trust there.
if the new additions are from a small project, then there’s the bus factor to consider. We need to support code at least 5-10 years into the future.
That being said, I would like to see algorithms such as BOBYQA added. I just wish they were added in code I could understand and manipulate.
It feels like there’s a consensus building around a position of generally discouraging new Fortran submissions, but possibly making exceptions in cases of very compelling projects?
One thing that’s not super clear to me is this goal of getting rid of all Fortran. It’s repeatedly stated as a goal, but then people say things like “BLAS/LAPACK will stay around for a while.”
I get that as programmers it’s very compelling to eliminate entire classes of problems by appropriately choosing technology, but in this case is it really serving the larger project goals? Might they be better served by finding better ways to support that technology? I’m genuinely asking, I don’t know because I’m not intimately familiar with the build challenges SciPy has faced and is currently facing. From my perspective, I’ve been able to successfully install and use Fortran toolchains on various operating systems and platforms. Is it fun? Not really, but it’s a solvable challenge.
Turning to the concerns about lack of knowledge of Fortran within the community, I think that
a) This is a problem for any codebase in any language. There will always be complicated algorithms with complicated implementations. This is a scientific codebase after all.
b) In terms of concepts, Fortran and C are pretty similar. There are syntax differences and differences in default behaviors and other differences, but the point I’m making is that it’s not like the difference between Python and C or C and C++.
c) I think today’s AI tools can help tremendously in this avenue. The cost of figuring out “How do I do ‘abc’ in language X?” is dropping rapidly, and while this doesn’t eliminate the need to develop proficiency in a language it drops the barrier significantly.
I don’t mean to diminish the difficulty of maintaining this sort of code or the efforts that have been made to get all this work in a way that lets users abstract it all away with pip install scipy, but I do want to help focus the technical decisions on serving the project instead of becoming goals in and of themselves.
If we are fortran-free we can work on integrating OpenBLAS/MKL/… without forming a complete Fortran ABI and without a Fortran chain but by linking to it. So it becomes a true external dependency. That’s also not fun to work on, there are a lot of intricacies but if we still have fortran code it’s pointless.
I have been around for quite a while. It is not. Nobody maintains code they don’t understand and we didn’t.
Since the other discussion took over already in terms of distance. I hope I clarified the points and hopefully, I can go about my way
I mostly agree with @ilayn’s stance here. I’ll repost my reply from Slack:
Well, the end goal is to get rid of Fortran completely, so the only F90 code we’re considering (PRIMA) at the moment is okay because it replaced F77 code that’s a lot worse. I think we should avoid adding new F90 code at all unless there’s a super compelling reason, because it’s just going to have to be translated later on.
I think it is a valuable project goal. It’s not only about whether well-written modern Fortran is easier or harder to maintain for us than C/C++, but more about compiler toolchains. Fortran compilers are basically only mature on Linux/Unix, and a major time sink and tech debt hazard on Windows and macOS. We’ve had narrow escapes with Python 3.12 (xref the blog post from @h-vetinari that Lucas linked above), and also with the introduction of Apple’s M1 processor (updating gfortran for that took a while and was the work of 1-2 dedicated experts, we couldn’t have done that ourselves). Now this situation may improve or it may get worse, it’s hard to know. LFortran development has slowed down a lot; Flang currently looks promising with the upcoming 19.0 being able to build SciPy - but I’m very aware that Flang has been in “promising” status for many years, so I wouldn’t yet could on it being a silver bullet in the long term.
I disagree on this actually. Flang has been good enough to build Scipy since v16, and overall has matured enough that we’ll use it as the default Fortran compiler on windows in conda-forge as of v19. We haven’t rolled this out yet (it was just released this week, but I’ve been testing the rc’s), but that will happen very soon.
So I actually think the toolchain situation has shifted fairly dramatically compared to a couple of years ago. Flang is a very actively maintained and responsive project within LLVM, and is not more difficult to deploy & use (fundamentally) than clang.
While reducing toolchain surface is certainly an overall positive, I don’t find it to be a compelling argument anymore to completely rule out Fortran (though of course our language priority remains roughly: Python → Cython → C → C++ → Fortran).
I’ll change my opinion only once we’ve actually got something working reliably for a while. There are a lot of open questions, and remember that we don’t only need to support conda-forge - building wheels for PyPI is the harder and more time-consuming part, and AFAIK no one has even tried to build any redistributable wheels with Flang. It may or may not work at all, it may increase the binary size due to the extra vendored runtime, etc.
I’m afraid that this will never be true unless Microsoft and Apple start shipping a Fortran compiler or runtime. To give a very concrete example of the type of problem Fortran causes that C/C++ do not: right now we’re considering dropping support for macOS 12 wheels in our next release, because we’re using gfortran from Homebrew and the deployment target was bumped too high by Homebrew. For C/C++ you can support older macOS versions very simply, by setting the MACOSX_DEPLOYMENT_TARGET environment variable. For Fortran on the other hand it’s intrinsically hard, you have the choice between using a compiler someone else built and lose control over what you can support, or maintaining you own build of a compiler.
We have burned huge amounts of hours on stuff like that (hard to estimate how much exactly, but many hundreds of hours at least by SciPy/Numpy maintainers, possibly 1000+), and that’s effort that could be better spent in other ways.
a) Accept the updated COBYLA implementation in PRIMA with thanks.
b) Resolve not to accept new Fortran implementations without a compelling justification, for now.
c) Defer wrapping the other solvers in PRIMA for now
d) Resolving to review b) and c) in the context of current Fortran toolchain maturity, perhaps every year or so.
This sounds good to me given the discussion so far.
If any of us are feeling particularly courteous, we could perhaps offer help to @nbelakovski in getting PRIMA vendored in an alternative library such as Optimagic as thanks for his offer to maintain PRIMA on the SciPy side. Then the other algorithms from PRIMA can still be made available to the Python community with a SciPy-like API, without putting any more work on @ilayn ‘s plate towards Fortran-free SciPy.
If you’ll allow it, let me make one more argument for adding these extra solvers:
From reading the article about building for 3.12, it sounds like there was a major problem for 3.12 specifically that required a lot of effort, but it has been solved, and things work now. Are there other potential 3.12-like catastrophes on the horizon?
If things are stable now, then I think adding the solvers in the Fortran form would actually be preferable even if they are to be translated later, because by adding them they get usage and as developers we get more familiar with the algorithms by getting user reports. If there are bugs in it, well they’d need to be fixed at some point and it doesn’t really matter if they’re in Fortran or Python (although if people are reporting problems that have more to do with the compilation than the algorithms then I think those problems can be closed as Won’t Fix).
The idea here is not to open the door to everyone who wants to add a new Fortran function to SciPy (is there anyone else besides PRIMA?), but adding one more exception in addition to the exception already granted for PRIMA/COBYLA, and doing so in the service of users since these are unique solvers.
This would add 4 more solvers to the list of things to translate, but unless there’s a looming deadline of some sort, would it hurt SciPy meaningfully to accept this delay to the Fortran free future?
Nothing as large as that one, but as I pointed out there are ongoing problems. It’s very uncomfortable to for example be forced to drop macOS 12 wheels for the next release because the compiler we use dropped support. In this particular case it’s not that bad, because this is about arm64 and there aren’t yet users who cannot upgrade to macOS 13 due to old hardware.
Another one that is always a potential problem is hitting a new numpy.f2py bug that will force us to drop support for old numpy versions if we cannot work around whatever the bug is. We haven’t encountered this yet, but we did have a close call recently where we had to add some f2py-internal define to work around such a bug.
That’s hard to know, since most of the potential issues aren’t predictable.
I’d prefer a different solution here if possible. Are the solvers already exposed in a PRIMA package with Python bindings? If so, could we make that an optional dependency (enable use via minimize if the package is installed)? If not, could it be in a separate package?
We started from around 100K SLOC of F77 in SciPy. We are close to 70% currently.
Out of 60K SLOC, 30K is ARPACK (11k) and PROPACK (19k) which is very close to finish. And FITPACK is another 10K that I don’t know what to do yet. It would be nice to know who will be involved in the translation efforts of that extra 14k.
Not yet but I hope we’ll be able to do that soon. What would the user interface/documentation look like for that? Would we just list the solvers in the documentation as if they were available, and then when someone goes to run them we check if the prima package is available and if so use it, otherwise print a helpful message and exit?
I think that can work. Is the idea here that if someone reports an issue with any of these solvers you would punt it over to PRIMA, since it’s an optional package? To be clear, there’s nothing wrong with that, I just want to make sure we’re all on the same page.
Yes indeed. The SciPy bug report may stay open until a fix is available, but the bug report should be forwarded to the repo where the problem in the source code is located.
