Okay, I took a look. The biggest blockers seem to be test discovery and our reliance on implicit imports in our doctests and maybe our use of lazy imports.
In a lot of places our docstrings rely on the function of the docstring to be known / imported implicitly in the test context. However, this implicit import seems to fail on a lot of places, e.g. running
from scpdt import testmod
from skimage import morphology
testmod(morphology, strategy="api")
prints NameError: name 'erosion' is not defined in "/home/lg/Projects/skimage/skimage/morphology/gray.py", line 109, in erosion among many others because erosion() is not imported in its own docstring.
The closest I could get around this was with creating the test context myself:
# test_scpdt_skimage.py
import numpy as np
import skimage
import scpdt
def create_test_context(module):
# Construct our own execution context for the tests. It seems, that scpdt
# doesn't find / import the function of tested docstring all the time
globs = dict(np=np)
globs.update(vars(module))
try:
globs.update({name: getattr(module, name) for name in module.__all__})
except AttributeError:
pass
return globs
report = {}
for name in skimage.submodules:
module = getattr(skimage, name)
globs = create_test_context(module)
res, hist = scpdt.testmod(module, globs=globs, strategy="api")
report[name] = (res, hist)
total_failed = sum(res.failed for res, _ in report.values())
total_attempted = sum(res.attempted for res, _ in report.values())
print(f"{total_failed=}, {total_attempted=}")
print(f"{skimage.__version__=}")
print(f"{scpdt.__version__=}")
This script fails only for 13 of 1447 found doctests (still some NameErrors but also problems with # doctest: +ELLIPSIS) (see attached log below). I also adapted the proof-of-concept for scipy to our library but this approach has the same problems with required implicit imports. I’m not sure if SciPy is just more explicit about their imports or if they do something else different. Our lazy module importing may also be a reason why this works for SciPy but not for skimage? Also I’m not sure if this script is discovering all of our doctests.
Currently, it seems to me that integration would be quite hacky still but that most of our tests would run just fine if we used explicit imports of the function in it’s own docstring. I think a PR for this would be a benefit for skimage on its own, independent on our decision to use scpdt!
Ideally we’d have a plugin for pytest to for a more reliably discovery of doctests.
test_scpdt_skimage.py log
perimeter
---------
File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 329, in perimeter
Failed example:
perimeter(img_coins, neighborhood=4) # doctest: +ELLIPSIS
Expected:
7796.867...
Got:
0.0
perimeter
---------
File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 331, in perimeter
Failed example:
perimeter(img_coins, neighborhood=8) # doctest: +ELLIPSIS
Expected:
8806.268...
Got:
0.0
perimeter_crofton
-----------------
File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 301, in perimeter_crofton
Failed example:
perimeter_crofton(img_coins, directions=2) # doctest: +ELLIPSIS
Expected:
8144.578...
Got:
0.0
perimeter_crofton
-----------------
File "/home/lg/Projects/skimage/skimage/measure/_regionprops_utils.py", line 303, in perimeter_crofton
Failed example:
perimeter_crofton(img_coins, directions=4) # doctest: +ELLIPSIS
Expected:
7837.077...
Got:
0.0
/home/lg/Projects/skimage/skimage/measure/fit.py:622: RuntimeWarning: divide by zero encountered in double_scalars
return np.ceil(np.log(nom) / np.log(denom))
2 items had failures:
2 of 5 in perimeter
2 of 5 in perimeter_crofton
***Test Failed*** 4 failures.
area_closing
------------
File "/home/lg/Projects/skimage/skimage/morphology/max_tree.py", line 448, in area_closing
Failed example:
P, S = max_tree(invert(f))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest area_closing[7]>", line 1, in <module>
P, S = max_tree(invert(f))
NameError: name 'invert' is not defined
diameter_closing
----------------
File "/home/lg/Projects/skimage/skimage/morphology/max_tree.py", line 555, in diameter_closing
Failed example:
P, S = max_tree(invert(f))
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest diameter_closing[7]>", line 1, in <module>
P, S = max_tree(invert(f))
NameError: name 'invert' is not defined
2 items had failures:
2 of 9 in area_closing
2 of 9 in diameter_closing
***Test Failed*** 4 failures.
denoise_wavelet
---------------
File "/home/lg/Projects/skimage/skimage/restoration/_denoise.py", line 518, in denoise_wavelet
Failed example:
img = img_as_float(data.astronaut())
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest denoise_wavelet[1]>", line 1, in <module>
img = img_as_float(data.astronaut())
NameError: name 'img_as_float' is not defined
1 items had failures:
5 of 7 in denoise_wavelet
***Test Failed*** 5 failures.
total_failed=13, total_attempted=1447
skimage.__version__='0.20.0.dev0+git20220616.0e28f6397'
scpdt.__version__='0.1'
