SciPy has had a tool, called dev.py, for a while now. It is essentially a smarter, more Pythonic, easier to use Makefile that captures the typical developer commands. I’ve written a simplified but configurable version of that tool that can be customized for ecosystem projects (although, this was designed specifically for those who have to build binaries with Meson, since the Meson workflow is not that well known yet—I’m thinking scikit-image, pandas, astropy, etc.)
This is an experiment: we’re not sure how useful it will be vs simply documenting commands that need to be run. But, I’d like you to please try it out and see how it can be improved.
Here’s a typical skimage workflow using dev.py:
./spin build
./spin test -- -m 'not slow'
You can customize the commands available—they’re just Python functions that can receive parsed arguments, and that can also access the pyproject.toml configuration.
What do you think about things like Hatchling? It seems like in the end it would provide a similar feature set no? You could add a build plugin that would call Meson.
Hatchling looks like it does a whole bunch of things other than just provide a cli. I’d prefer to stick with something really simple.
It also looks like this project is just out of its shell. Here is their docs on plugins. I could figure out how to write a custom plugin that can handle its own argument parsing, but perhaps it’s possible?
Hi @stefanv and @jarrodmillman looks like you all have already put a lot of work into this.
Is the goal to have something like a pure-Python build system for projects using Meson/ninja?
Or is it a more general pure-Python makefile-like thing
Just a note that it might be worth looking at nox
Not sure if it would give you what you want for Meson/ninja builds but looking at your _build functions etc I could guess those could easily be nox “sessions”
This is what scikit-hep uses, specifically recommending it as an alternative to Makefile because it is platform independent and friendlier to newbie developers:
as well as many of the core Python projects like pip
and cibuildwheel
Would be great if the whole Python community could work together instead of half of scientific Python re-inventing the wheel
Thanks @nicholdav. That looks similar to what we’re building here. I think even in this early phase, though, the dev.py interface is quite a bit easier to use. --help does what you expect it to, also for subcommands. With tox, you have sessions, but I cannot figure out how to ask for help on session arguments. E.g., tox -s test --help just prints out the default help. The goal here is to have something that is easy to use and easy to extend with arbitrary commands that each can have their own flags, and can also access project metadata.
For additional context: NumPy and SciPy (and probably several other projects) have used a runtests.py for many years. The dev.py interface in SciPy is basically a much-enhanced version of that. It doesn’t have to be restricted to Meson indeed.
I’ll also point out that there’s two separate things to consider here, conceptually:
Is it useful to have an all-in-one developer CLI that is self-documenting and shows a contributor everything that they need to contribute to a project (where that can include more niche things like benchmarking a PR vs. main, or release tasks),
What tools to use to implement such a CLI (the suggestion about nox further up is more about that).
One comment I’ll make here is that whatever the implementation is, for complex projects with compiled code, it’s pretty essential to separate setting up a development environment from doing all of the potential things one needs to do inside a complete development environment. Most Python tooling, including nox, that come from folks who never have to deal with building compiled code are choosing the wrong default here (and for nox, it only worked in a venv for a long time).
