Type hinting / annotating SPEC

Many ecosystem projects are beginning to discuss how and when to implement type hinting / annotating. For example, see

Type hinting / annotation seems like a generally desirable feature and one that most projects in the ecosystem will need to sort out sooner or later. However, this functionality is new and seems to be evolving and improving. For developers who are not closely following the situation, it is unclear how mature this functionality is, what the current costs and benefits of implementing it now are, and what the best way to implement this functionality is. For example, does it make it easier or harder for first-time contributors? Is there anything special needed to handle shared data structures like NumPy arrays? How important is it for the ecosystem projects to implement this functionality in a similar way? Does (and, if so, how does) this impact shared tooling (e.g., numpydoc)?

Rather than having each project spend time researching this topic and independently figuring out how and when to implement it, this SPEC could provide much needed guidance about why, when, and how ecosystem projects should consider implementing this functionality. It could also pool the limited expertise across projects. This would help those experts form a community to discuss this issue and would provide maintainers who have not had the time to learn about this functionality yet a group of colleagues whose collective opinion can be relied upon.

I don’t know enough about this issue to suggest exactly what this SPEC should look like and I am not the right person to draft such a SPEC. But speaking as a NetworkX developer, I believe something like the following would be very helpful for my project and I am certain many other projects would similarly benefit.

Certain projects are investigating and starting to implement type hints / annotations. Perhaps the contributors working on those features would coauthor a new SPEC. In order for the new SPEC to be accepted, the authors would need to provide some basic details about the type hinting and how it would benefit the ecosystem (particularly focusing on how coordinating this would benefit the ecosystem). It should also identify a few projects and people who will be prototyping this functionality. After it is accepted, the coauthors from the various Core and other ecosystem projects would coordinate with one another, review one another work in the various project repositories, and distill from this experience guidance for why, when, and how other ecosystem projects should follow. Much of the actual work would take place in the individual projects prototyping this feature. But the coauthors who in the past would have been siloed in their individual projects would form a mini-community that can review one another work and leverage each other experience.

During this pre-Endorsement period, projects would decide for themselves whether they want to also join the effort to prototypes type hinting in the ecosystem. If so, they should consider asking one of their team to join the SPEC as a coauthor to help in the collaborative process. The SPEC would list the projects prototyping this feature as well as the individual contributors working on it. This will help individual projects interested in prototyping type hinting / annotating by increasing the pool of expertise available to discuss and review new PRs.

For projects not ready to prototype type hinting / annotating, it would provide a better way to respond to new PRs adding type hinting / annotating support to their project. Currently, for example, NetworkX core developers aren’t prepared to decide how we should move forward with type hinting / annotating. So when contributors submit PRs with type hinting / annotating we have to ask them to remove it. When those contributors ask what our plan is, we currently don’t have a good answer. This risks discouraging this contributors from continuing to contribute to the project. However, if a SPEC like this is accepted and is being actively worked on by others in the ecosystem, we would be able to explain to those contributors that we are waiting for the SPEC to be endorsed by some of the other Core Projects. We could also ask those contributors to review the SPEC and, if interested, suggest changes and improvements. We could even ask them to consider coauthoring the SPEC to ensure that it will take our projects needs into consideration. Then once the prototyping effort is finished and the SPEC has been endorsed by the interested Core Projects, NetworkX core development team can review the SPEC and start accepting PRs to add support for type hinting / annotating based on the plan and guidance provided by the SPEC. This would make it easier for our core developers to have confidence that they are moving in the correct direction without requiring us to do the research and prototyping ourselves. Hopefully, this would also speed up the process by which NetworkX implements type hinting / annotating. Moreover, it would ensure that type hinting / annotating is implemented in a consistent way across projects.

While in draft form, the SPEC should provide information for both projects interested in joining the prototyping effort as well as information for projects that are waiting for the prototyping effort to finalize a endorsed SPEC.

There are a lot of great points above. I think type annotations are an excellent topic for a SPEC. It would be particularly valuable to have a conversation from the perspective of the scientific Python ecosystem. From the Rationale and Goals of PEP 484:

Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring.

A bit later in the document…

… third party packages would have to be developed to implement specific runtime type checking functionality, for example using decorators or metaclasses. Using type hints for performance optimizations is left as an exercise for the reader.

Though not the emphasis in the PEP, I suspect that the later would be a stronger driver for adoption amongst scientific Python packages.

Looking at the benefits and challenges from the perspective of scientific Python projects would be very valuable and (eventually) developing some sort of best-practices or procedure for consistent adoption across projects would definitely help push the pace for feature adoption.

FYI, we had a lengthy discussion over at SciPy about static typing.

While it’s still an open debate, the current situation is that we are allowing developers to use type hints when they want to.

1 Like

Could you summarize the expectation around typing arrays? I think that was the biggest hold-up so far, but I know Bas is making progress.