ANN: MLReproMutate - mutation testing for reproducibility safeguards in ML research repositories

Announcements
1

Hi everyone,

I hope this is an appropriate place to share this. I’ve been working on a small Python tool called MLReproMutate, and I thought it might be relevant to people here who work with ML research repositories.

The question behind it is fairly simple: if you change something like a random seed or a cross-validation setting in an ML research repository, do the repository’s tests, CI, or validation scripts actually notice?

The tool makes one controlled change at a time and runs a validation command against it. It checks the unmodified project first, so a failing baseline is kept separate from the mutation result.

At the moment there are four operators:

  • random-seed changes a supported literal seed from N to N + 1
  • dependency-pin changes package==version to package>=version
  • data-split removes explicit stratification from supported train_test_split calls
  • cv-fold-count changes an explicit n_splits value from N to N + 1 for supported scikit-learn splitters

For example, given:

random.seed(42)

the random-seed operator changes it to 43 and then runs whatever validation command you specify.

Install from PyPI:

pip install mlrepromutate

A basic run looks like this:

mlrepromutate run PROJECT \
    --operator random-seed \
    --command "pytest -q"

If you only want to inspect what the tool would mutate, without running the project:

mlrepromutate detect PROJECT \
    --operator random-seed

One important point: if a mutation survives, that does not mean the repository or its results are irreproducible. It only means that the validation workflow you selected did not detect that particular change.

I developed the tool alongside an empirical study, so most of my testing so far has been on repositories and patterns I encountered there. What I’d like to test now is how it behaves on ML research repositories I haven’t seen before.

If you maintain or work with one, I’d be interested in things like:

  • code patterns the current operators fail to detect;
  • candidates that look wrong or ambiguous;
  • validation setups that are awkward to run through the tool;
  • cases where a mutation behaves differently from what you expected;
  • reproducibility-related choices that might be worth adding as operators.

Runs that don’t work as expected are useful too. If you come across an unsupported pattern or an unexpected result, I’d be glad to hear about it, especially if you can share a small example or open an issue.

GitHub: GitHub - ilyuka/MLReproMutate: Mutation testing for reproducibility safeguards in machine-learning research software · GitHub

PyPI: Client Challenge