Numpydantic - array typing and validation for pydantic and beyond

[sneakers-the-rat]

Hello everyone!

@stefanv recommended i share this here, hopefully y’all find it interesting!

I just released a package numpydantic that provides generic typing for array shape and dtype validation using pydantic (and also generally), post here: Dr. jonny phd: "Here's an ~ official ~ release announcement for #…" - Neuromatch Social

The idea is that we not only want to be able to specify arrays in pydantic, specify shape and dtype constraints for those arrays, but also be able to treat that specification as being generic across array implementations. I see this topic has come up a few times before in different ways:

Support for array types other than NumPy

I created a repo that has a demo of dispatching to different backends with a uarray-based approach Benchmark results and some discussion is given the README there. I mostly autogenerated the uarray multimethods on the scikit-image side using a script provided in the repository above, but it does create an additional layer of indirection that can make things a little harder for new contributors to understand. The Dask backend there is a more user-friendly way of using apply_parallel where a use…

A proposed design for supporting multiple array types across SciPy, scikit-learn, scikit-image and beyond

Hi all, Ivan Yashchuk and I - with input from a number of maintainers of all libraries involved - wrote a blog post about a topic that is hopefully exciting to many of you: making support for multiple (non-NumPy) array types in SciPy, scikit-learn, scikit-image and other such libraries a reality. Here is the post: A vision for extensibility to GPU & distributed support for SciPy, scikit-learn, scikit-image and beyond | Quansight Labs Because this topic spans across multiple libraries at the co…

This package isn’t intending to be a unified interface to all arrays, but a passthrough to various array backends after validation - it does provide some proxy classes for eg. treating videos and hdf5 arrays as being very similar to lazy-loaded numpy-like arrays. This is part of a broader data formats effort to be able to specify array data in formats like neurodata without borders and etc. as abstract schema and then generate many flexible representations of that schema, decoupling schema from implementation.

would love to hear thoughts - it’s a v1 release (well v1.1) so there are of course tons of improvements left to do, and i’d love to hear what could be made better or how this might be useful!!!

[rgommers]

Thanks for sharing @sneakers-the-rat.

@sneakers-the-rat

The idea is that we not only want to be able to specify arrays in pydantic, specify shape and dtype constraints for those arrays

You may be interested in TYP: Make array _ShapeType covariant by Jacob-Stevens-Haas · Pull Request #26081 · numpy/numpy · GitHub, which is very close to the finish line and adds a new np.typing.Array object which is shape and dtype-aware.

@sneakers-the-rat

but also be able to treat that specification as being generic across array implementations

You’re linking to discussions about runtime support that is generic across array libraries. That is fairly far along, see for example:

• Python array API standard — Python array API standard 2023.12 documentation
• NumPy 2.0.0 Release Notes — NumPy v2.1.dev0 Manual (" upport for the array API standard in the main numpy namespace.")
• Support for the array API standard — SciPy v1.14.0.dev Manual
• 11.1. Array API support (experimental) — scikit-learn 1.5.0 documentation

Generic static tying on top of that has a longer way to go - in particular if it also has to be shape-aware. I think we’ll only see that once all of the above has stabilized.