It has been a few months since we first implemented an AI policy. I think this has been somewhat helpful in providing guidance for the new world we find ourselves in.
That being said, I still see a quite a few PRs from new contributors which consists of simply plugging the issue into their favourite coding agent and immediately opening a PR. Except for the most trivial of issues I think these type of contributions have close to no value. Engaging with these types of PRs is pretty pointless as you are just indirectly prompting the llm yourself. I feel these PRs also have a negative impact on maintainer moral.
My proposal is that for first time contributors they must write the code themselves. If someone is going to become a valuable contributor to SciPy then spending time and effort to understand how to go about solving an open issue is important. Using an llm simply short circuits this process. This would be enforced through the declaration process which I appreciate is imperfect.
What do others think? Is this a step to far? I do worry about finding the right balance between not putting valuable new contributors off Vs protecting the project against a deluge of low quality code.
These types of PRs is what I call “low effort PRs” and I agree with you there is no value in them. Two distinction that I think are important though: (1) it doesn’t matter if you are a first timer or a 100 timer, don’t make low effort contributions. (2) you don’t need AI to make “low effort” contributions ( code spell and friends).
I have seen a similar policy in other projects. For example mypy has a rather liberal AI policy, but explicitly discourages using an LLM if you’re a new contributor:
In general, mypy takes a neutral stance on using various LLM code assistants to make contributions. LLMs are just another tool, and contributors bear full responsibility for the code they are submitting. Disclosing the use of LLMs in pull request description is recommended, but not required.
However, we discourage use of LLMs by new contributors. We are interested in growing long-term contributors who have good understanding of mypy code. Pull requests from new contributors that are mostly generated by LLMs with little human input will be closed.
I tend to agree with this, because in my experience, most new contributors that use AI to write their contributions you’ll never see again. I’m guessing that’s because they just do it so they can claim on their CV that they contributed to . In practice, those sort of “contributions” do more net harm than they do good, as those are usually low-quality fixes for trivial matters that take a lot of time to review, which is often in vain because those review comments often won’t even be addressed.
So I’m +1 on discouraging new users to use AI tools to write PR’s for them. But I wonder if fully disallowing AI tool use for new users is a bit too strict, because AI can also help learn, e.g. how to navigate the codebase, pre-reviewing changes, etc.
I fully agree with Tim! In this context, I like the term contributor poker, which, in my opinion, captures the issue very well: Every contribution requires a certain amount of maintainers’ resources. Since these resources are scarce, the necessary prioritization calls for “playing poker” on which contributions will provide the best long-term value for the project.
Perhaps it would be beneficial to rename the “AI Policy” to something like “Low-Effort Contributions and AI Policy” and explain how contributions are ranked and why some are rejected.
Anecdotally I am not sure I agree fully that the issue is low effort contributions. For example, my first contribution to SciPy would probably fall under this category but it was enough to expose me to things like structure of SciPy and it’s developer tools to work towards contributions that weren’t low effort. So I am actually sympathetic to low effort contributions if they are on the road to something more substantial. The issue is using AI breaks this pathway as you don’t actually learn anything in the process.
Of course, the amount of effort is not an accurate indicator of a contribution’s value. Another example is fixing typos, which is typically low effort and offers reasonable value, IMO. “low-effort” may not be the most suitable term. Something like “low-effort and low-value” or “low-quality” might be more appropriate.
In summary, I think it is worthwhile to communicate concisely to (first-time) contributors that
we prioritize high-quality contributions due to resource constraints, and
heavy use of AI and/or low effort (as in time invested) are typical albeit imperfect indicators of a low-value contribution.
However, I am not sure what the best way to achieve this is.
I don’t know what your first contribution was. Mine was fixing two typos in a the documentation. It took me a whole afternoon, and I was in the same room as the maintainers.
For me this is not a low effort contribution, it took real effort to figure out how to use GitHub, build the docs, etc. Even if the output was “just” two trivial typos fixed.
How do you tell the difference between a low effort typo fix contribution and one where someone put in effort? I’d argue you can get a good feeling by counting how many typos they fixed. If they fix two or three in one document, or the same typo in two or three documents it was probably a human trying to figure out how to contribute. 57 typos across 32 documents? They probably used some kind of automatic tool.
You might be an experienced contributor, I still don’t really want a PR from you that is the result of unsupervised AI.
For me it isn’t about “did you use AI or not” nor “are you a newcomer or not”. It is just about “did you put some effort into this?”
And yes, there could be experienced contributors who spend 4minutes fixing two typos. For them that is a low effort PR, but I’d argue that we can spot that kind of contribution and merge it with 1min of review. So while technically it might be “low effort” it is likely to be packaged up in a way that also makes it “low effort” to review. Click, click, type, type, click - we all move on in life with two typos fixed
Ah ok I think I misunderstood what you meant by low effort, thanks for clarifying! Mine was a two word doc change that took far too long to figure out …
This is alarming. What would be most helpful to alleviate that?
add a bot to autoclose PRs which lack the AI disclosure [1]
have some informal rough rotation schedule for who does low-quality PR triage (without any hard evidence, it looks like Jake’s doing the lion’s share of it, and it certainly takes a toll)
something else?
FWIW, I am rather sceptical about low-effort/high-effort distinction. Anything that’s high-effort to me is low-effort to somebody else (as in, there exists at least one other person to whom it’s low effort etc).
What we make the decision on is more low-quality vs high-quality, but it takes an experienced maintainer to assess.
[1] the irony is not lost on me, of me suggesting to add a bot
What would be most helpful to me here is to update the issue template (and maybe the PR template as well) to add a mandatory section about how the issue was encountered and why it is important to solve. What I’m seeing in terms of PRs that are more noise than helpful seems to have a high likelihood to be connected to issues that were found by lazily pointing an LLM or other code auditing/linting/whatever tool at the code base. And I’m not interested in that, I want to spend time on things with real-world relevance to actual users.
I think we have to do more, but I’m somewhat concerned about this kind of restriction. If I’m thinking about how I would want to be treated and what rules I want to encounter when contribution to another OSS project I’m new to, this rule would be counterproductive. It would feel a bit arbitrary[1], and might even mean I have to redo work I already did in the context of debugging the problem or working on the missing feature in that other library - because I’m not reading the PR template before I’ve understood or fixed the problem.
The thought that comes up here for me is that some of our most valuable contributors aren’t only the ones that keep coming back and grow into a structural contributor or maintainer role. They are also the one-off contributors like distro packagers and maintainers of our upstream and downstream dependencies. They have a real problem, solve it, and move on.
The contributors that cause the PR/issue volume problem we’re dealing with are the ones that show up to get their name on the commit log, rather than to solve a real-world problem.
Something like the following points might be suitable for our PR template. It is an attempt to formulate what we expect from contributors. “Contributor Duties” may not be the best name for this (I guess “Code of Conduct” is already taken ).
Contributor Duties (Draft)
Thank you for your interest in contributing to SciPy. We welcome contributions of all sizes and appreciate the time and effort it takes to improve the project.
Communicate as a Human
Issues and pull requests are intended for communication between contributors and maintainers. They are ill-suited for communicating with AI agents.
Contributors are expected to engage directly in discussions and to explain, justify, and iterate on their contributions themselves.
Take Ownership of Your Contributions
Contributors are responsible for the changes they propose, including those generated by AI agents or other third parties.
Any AI-generated content must be disclosed.
Before submitting a contribution, ensure that you understand not only the code changes, but also the broader context, motivation, and potential impact on users and the project.
Focus on Impact
We are grateful for every contribution. However, maintainer time and review capacity are limited. As a result, we must prioritize contributions and may reject those that do not provide significant improvement.
Consider the impact, usefulness, and long-term maintenance cost of your contributions.
Be Responsive During Review
Code review is a collaborative process. Contributors are expected to respond to feedback, address requested changes, and keep discussions moving in a timely manner.
Unresponsive or abandoned contributions may be closed to keep the review backlog manageable.
Be Respectful and Kind
Thank you for helping make SciPy better. We appreciate your contributions and collaboration.
Disclaimer: AI had a significant part in writing this draft.
Many of the measures discussed above assume cooperative contributers that read the guidelines and then act on them. However, those kind of contributers were never a problem. It’s the “point an LLM at the code base and open PRs” contributions.
Other code bases add SKILL.md files or AGENT.md files that tell the LLM to not write PRs in this repository, confront the user with our contributer policies, or to watermark the PR. Maybe SciPy could do the same to enable the automatic filtering mentioned above.
This is one example where agents are told that PRs with non-human owner result in a ban
I am pretty sure I have seen more examples, but don’t remember which ones that were. Another example of prompt injection to catch LLMs are e.g. the NeurIPS reviews this year:
Re AGENTS.md , it’s not so simple - see this thread from half a year ago that focused on that: AGENTS.md and CLAUDE.md addition to SciPy repository . I’d suggest looking at the UX and “does it even work” aspects in that thread and continue there if there’s new insights.
code spell and friends).
).