After a longer pause due to various other priorities, I finally managed to become more active in Scipy recently… I also wanted to see how coding assistants can help with the development, so I took up an idea I had a longer time ago: run a check how accurate the methods of distributions in scipy.stats are for various domain and parameter values.
I set up an agentic workflow for this (more on this below) and ran it for almost all continuous distributions which used up about ~2 weekly limits of my Claude Code Pro subscription with Opus 5.5 (~30 minutes per distributions on my laptop where it runs code to analyze the methods, some take quite a bit longer, some are pretty fast). The output is a collection of reports for each distribution with detailed findings (more details below). I only looked at a few of them so far but I thought it would be good to share my experience and start a discussion around this approach.
Overall, I was really impressed how well the agents analyze the code and identify edge cases, trace down the root cause (e.g., due to bugs in special functions, see BUG: inaccuracy in special.gammainc for certain large parameters · Issue #26471 · scipy/scipy · GitHub and BUG: scipy.stats.beta.ppf returns wrong values or nan for small q · Issue #26531 · scipy/scipy · GitHub ) and propose fixes. Watching the discovery of bugs or ways to improve the code was quite an experience. Of course you read a lot about this in the press recently, say related to cyber security or math research.
I assume a similar approach can help us improve the code in many other areas of SciPy and I already noticed recent activity in scipy/xsf. Are there plans to run code reviews with coding assistants?
The next topic is how to deal with the findings. The first challenge is to decide where to start given the amount of information. There are some easy fixes (I already created two PRs, namely MAINT: improve ppf of pearson3 in scipy.stats by chrisb83 · Pull Request #26589 · scipy/scipy · GitHub and ENH: add isf to powerlaw, recipinvgauss, rdist, semicircular, moyal by chrisb83 · Pull Request #26617 · scipy/scipy · GitHub , reviewing such PRs is not a lot of work). I also tried fixing all 10 issues identified for one distribution ( ENH: Improve numerical accuracy of crystallball in scipy.stats by chrisb83 · Pull Request #26672 · scipy/scipy · GitHub ) which is then quite a bit of effort to review in detail (I spent a few hours this weekend going through it). The human review step remains key in my view before submitting a PR to make sure it makes sense, in line with our AI policy. However, it is likely that the review process will become a bottleneck and a strong model is very good at reviewing changes (more likely to detect problems in a change than I am 
Any thoughts on how we want to use language models in our review process?
In the context of creating/reviewing PRs, we could also improve our guidance. For example, for numerical accuracy of statistical distributions, I could try to summarize patterns to look out for like cancellation, log(1+x) etc (essentially such patterns were specified in one of the skills I used, see below). This could be part of the development guide or it could be used to help with development/code reviews (e.g., SKILL.md files).
Sorry about the long message, I thought it is a good time to discuss such questions.
Best
Christoph
Approach to review the distribution methods:
It relies on skills and agents defined in markdown files, essentially a skill described what kind of patterns to look for and an agent then uses it to analyze a given distribution. another agent verifies the results. An orchestrator coordinates the work. (I am sure this can be improved, I designed it with the help of Claude Code)
More details: see dist-accuracy/README.md at main · chrisb83/dist-accuracy · GitHub
Full details: see agents/skills in dist-accuracy/.claude at main · chrisb83/dist-accuracy · GitHub
Findings:
See reports per distribution in dist-accuracy/reports at main · chrisb83/dist-accuracy · GitHub + central INDEX.md
The third PR above is for dist-accuracy/reports/crystalball.md at main · chrisb83/dist-accuracy · GitHub