I recently observed that tests for each module is part of Scipy build. This increases the size of Scipy library which isn’t useful specially when you are trying to install these libraries within a Lambda layer. I was wondering if we can skip the tests for each module during Scipy build. This should make the library lightweight.
scipy from conda-forge is now shipped without the test suite. The tests are optionally installable with the package scipy-tests.
Are you installing scipy from PyPI?
Yes I have been using Pypi to install Scipy. Any reason it’s not available on Pypi?
All the large data files used during the test suite are located separately. The python test files themselves are not large, removing won’t actually save that much space. It’s better to distribute them so people can test on their own computer if they have a problem.
Others may know more, but I think we are unlikely to ship SciPy without the tests on PyPI, at least unless/until PEP 771 – Default Extras for Python Software Packages | peps.python.org gets accepted and implemented, for the reasons @andyfaff mentioned. For users for which this makes a difference, I think our recommendation would be to install SciPy from conda-forge.
Do you have a particular preference for PyPI @nitinrohan19, or could conda-forge work in your use-case? I don’t know how you are using Lambda, but if you are using Docker containers, Shipping conda environments to production using pixi | QuantCo Engineering Blog contains some useful advice on slimming down images when using packages from conda-forge via Pixi.
Thanks @andyfaff and @lucascolley for your replies. I understand that the files might be required for users to test. But from design perspective shouldn’t test be an optional dependency. So if someone doesn’t want it they can avoid it similar to the implementation for Conda forge. @lucascolley I see the pending Pip work but my concern is that this can still be done my taking tests as optional. If someone wants it then can get them. It just make things more flexible.
1 Like
I think there may be backwards-compatibility concerns related to dropping tests from default pip installations. PEP 771 – Default Extras for Python Software Packages | peps.python.org would allow us to keep shipping tests by default while giving users the option of installing without them.
Maybe we could just switch the default over to what we have in conda-forge and create a tests extra for PyPI, but I imagine there is a reason we haven’t done so.
We had some similar discussions in the past around shipping tests and docs. I am also with the opinion that we should not do that. Right now for all my docker builds I have some find/delete to remove all tests and docs folders from installed packages. Then in a multistage builds you copy the Python folders. It’s mostly scientific Python libraries that do things differently compared to the rest of the community.
Yeah I think that would be best and there should be no harm in doing that.
Even I have to do the same. I think it’s a simple fix and we should do it.