RFC: refguide-check doctester refactored into package

Evgeni refactored the doctesting from scipy’s refguide-check into a standalone package. He’s looking for feedback.

Is there an advantage for us to switch from our current system (standard doctest) to this? It is pretty much a drop-in replacement for the standard doctester, but has the following features (from the README):

  • Doctesting is floating-point aware. In a nutshell, the core check is np.allclose(want, got, atol=..., rtol=...) , with user-controllable abs and relative tolerances. In the example above ( sans # doctest: +SKIP ), want is the desired output, array([0.333, 0.669, 1]) and got is the actual output from numpy: array([0.33333333, 0.66666667, 1. ]) .

  • Human-readable skip markers. Consider

    >>> np.random.randint(100)   # may vary
    42
    
  • A user-configurable list of stopwords. If an example contains a stopword, it is checked to be valid python, but the output is not checked. This can be useful e.g. for not littering the documentation with the output of import matplotlib.pyplot as plt; plt.xlim([2.3, 4.5]) .

  • Doctest discovery is somewhat more flexible then the standard library doctest module. Specifically, one can use testmod(module, strategy='api') to only examine public objects of a module. This is helpful for complex packages, with non-trivial internal file structure.

  • User configuration . Essentially all aspects of the behavior are user configurable via a DTConfig instance attributes. See the DTConfig docstring for details.

Sounds good on paper. I’ll try the “drop-in” part locally and see how easy and useful it might be to skimage.

Okay, I took a look. The biggest blockers seem to be test discovery and our reliance on implicit imports in our doctests and maybe our use of lazy imports.

In a lot of places our docstrings rely on the function of the docstring to be known / imported implicitly in the test context. However, this implicit import seems to fail on a lot of places, e.g. running

from scpdt import testmod
from skimage import morphology
testmod(morphology, strategy="api")

prints NameError: name 'erosion' is not defined in "/home/lg/Projects/skimage/skimage/morphology/gray.py", line 109, in erosion among many others because erosion() is not imported in its own docstring.

The closest I could get around this was with creating the test context myself:

# test_scpdt_skimage.py

import numpy as np
import skimage
import scpdt


def create_test_context(module):
    # Construct our own execution context for the tests. It seems, that scpdt
    # doesn't find / import the function of tested docstring all the time
    globs = dict(np=np)
    globs.update(vars(module))
    try:
        globs.update({name: getattr(module, name) for name in module.__all__})
    except AttributeError:
        pass
    return globs


report = {}
for name in skimage.submodules:
    module = getattr(skimage, name)
    globs = create_test_context(module)
    res, hist = scpdt.testmod(module, globs=globs, strategy="api")
    report[name] = (res, hist)

total_failed = sum(res.failed for res, _ in report.values())
total_attempted = sum(res.attempted for res, _ in report.values())

print(f"{total_failed=}, {total_attempted=}")
print(f"{skimage.__version__=}")
print(f"{scpdt.__version__=}")

This script fails only for 13 of 1447 found doctests (still some NameErrors but also problems with # doctest: +ELLIPSIS) (see attached log below). I also adapted the proof-of-concept for scipy to our library but this approach has the same problems with required implicit imports. I’m not sure if SciPy is just more explicit about their imports or if they do something else different. Our lazy module importing may also be a reason why this works for SciPy but not for skimage? Also I’m not sure if this script is discovering all of our doctests.

Currently, it seems to me that integration would be quite hacky still but that most of our tests would run just fine if we used explicit imports of the function in it’s own docstring. I think a PR for this would be a benefit for skimage on its own, independent on our decision to use scpdt!

Ideally we’d have a plugin for pytest to for a more reliably discovery of doctests.

test_scpdt_skimage.py log

perimeter
---------

File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 329, in perimeter
Failed example:
perimeter(img_coins, neighborhood=4)  # doctest: +ELLIPSIS
Expected:
7796.867...
Got:
0.0

perimeter
---------

File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 331, in perimeter
Failed example:
perimeter(img_coins, neighborhood=8)  # doctest: +ELLIPSIS
Expected:
8806.268...
Got:
0.0

perimeter_crofton
-----------------

File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 301, in perimeter_crofton
Failed example:
perimeter_crofton(img_coins, directions=2)  # doctest: +ELLIPSIS
Expected:
8144.578...
Got:
0.0

perimeter_crofton
-----------------

File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 303, in perimeter_crofton
Failed example:
perimeter_crofton(img_coins, directions=4)  # doctest: +ELLIPSIS
Expected:
7837.077...
Got:
0.0
/home/lg/Projects/skimage/skimage/measure/fit.py:622: RuntimeWarning: divide by zero encountered in double_scalars
return np.ceil(np.log(nom) / np.log(denom))


2 items had failures:
2 of   5 in perimeter
2 of   5 in perimeter_crofton
***Test Failed*** 4 failures.

area_closing
------------

File "/home/lg/Projects/skimage/skimage/morphology/max_tree.py", line 448, in area_closing
Failed example:
P, S = max_tree(invert(f))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest area_closing[7]>", line 1, in <module>
P, S = max_tree(invert(f))
NameError: name 'invert' is not defined

diameter_closing
----------------

File "/home/lg/Projects/skimage/skimage/morphology/max_tree.py", line 555, in diameter_closing
Failed example:
P, S = max_tree(invert(f))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest diameter_closing[7]>", line 1, in <module>
P, S = max_tree(invert(f))
NameError: name 'invert' is not defined


2 items had failures:
2 of   9 in area_closing
2 of   9 in diameter_closing
***Test Failed*** 4 failures.

denoise_wavelet
---------------

File "/home/lg/Projects/skimage/skimage/restoration/_denoise.py", line 518, in denoise_wavelet
Failed example:
img = img_as_float(data.astronaut())
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest denoise_wavelet[1]>", line 1, in <module>
img = img_as_float(data.astronaut())
NameError: name 'img_as_float' is not defined


1 items had failures:
5 of   7 in denoise_wavelet
***Test Failed*** 5 failures.
total_failed=13, total_attempted=1447
skimage.__version__='0.20.0.dev0+git20220616.0e28f6397'
scpdt.__version__='0.1'