Scipy-doctest: select only doctests or both doctests and unit tests

Contributor & Development Discussion scipy
1

scipy_doctest pytest plugin, which is used by numpy, scipy and scikit-learn for doctesting, currently has an API quirck: it changes the meaning of pytest --doctest-modules option:

  • by default, pytest --doctest-modules collects and runs both doctests and regular unit tests
  • if scipy_doctest is installed, it changes pytest --doctest-modules to skip unit tests and only run doctests (this is behind numpy’s spin check-docs and similar stanzas in other packages).

While I still believe in the value of separating doctests from unit tests, the behavior proves suprising to users, in part because of the pre-established pytest behavior.
The last instance is Stop skipping tests · Issue #198 · scipy/scipy_doctest · GitHub

Therefore, let’s make the behavior configurable, and let’s make it less surprising. Here’s a plan:

  • In the next minor version, 1.8, add a CLI option, --doctest-only-doctests, so that

    • $ pytest --doctest-modules is backwards compatible (no change w.r.t. status quo)
    • $ pytest --doctest-modules --doctest-only-doctests=true is exactly equivalent
    • $ pytest --doctest-modules --doctest-only-doctests=false runs both doctests and unit tests

  • In the next major version, 2.0, flip the default to False, so that

    • $ pytest --doctest-modules runs both doctests and unit tests; this is a breaking change which brings consistency with the default pytest behavior (which will no longer depend on whether scipy_doctest is installed or not)
    • $ pytest --doctest-modules --doctest-only-doctests=false is exactly equivalent
    • $ pytest --doctest-modules --doctest-only-doctests=true will be required to keep the current behavior

User impact:

After scipy_doctest==1.8 is out, NumPy, SciPy and other spin users will need to add a lower limit scipy_doctest>=1.8 to their pyproject.toml and add --doctest-only-doctests=true to their spin check-docs command. From that point on, they are no longer affected by any further changes.

After scipy_doctest==2.0 is out, all other existing users, which manually invoke pytest --doctest-modules, will need to either use --doctest-only-doctests=true in their pytest invocations, or embrace the pytest default.

New users, who only start using version 2.0, will silently get consistent behavior.

Thoughts?

For completeness, here is the issue and the PR to add the new option.

1 Like
2

+1 to both the general aims and proposed implementation/rollout!