Hi @mwb, yes we should do this, but it is not clear exactly how to proceed. SciPy will probably follow after NumPy, and NumPy’s discussion is unresolved: Making the numpy test suite separately installable · Issue #26289 · numpy/numpy · GitHub.
Both SciPy and NumPy have the underlying infrastructure in the build system to separate out installation of tests already, hence distributors and anyone building from source are able to achieve this. E.g. on conda-forge there is scipy-tests - conda-forge (6.9MB) split out from https://prefix.dev/channels/conda-forge/packages/scipy (16.3MB).