A pragmatic pathway towards skimage2

Hi everyone,

As you may be aware, the team has been working on a way towards addressing some long-standing issues in the existing skimage API. The proposal, written up by @jni, is known as SKIP4.

Today, Matthew Brett and I had a long discussion on how to enact SKIP4 without losing a major part of our user community—something we cannot afford. The Python 2 to 3 transition is still fresh in our minds, along with the pain it caused (and some codes have still not been ported to Python 3).

So, it may be helpful to write down some principles of a transition. I think these should include:

  1. Do not want to estrange a large part of our existing user base. This implies making any porting of code straightforward (a challenge with the Py 2 to 3 port was that you had to read and think about the code you were porting; it was not just a matter of applying certain prescribed recipes).

  2. Ensure that most users eventually land on skimage2. If, e.g., we provided a supported skimage1 and skimage2, there would be no incentive to transition.

  3. Do not place an undue burden on maintainers. This would, e.g., preclude long term support of both skimage1 and skimage2.

  4. Initially, allow users to use the old and the new APIs in conjunction (this should help greatly in porting code).

It is not trivial to satisfy these three principles simultaneously. After considering many options, the following potential route crystallized:

  1. We release skimage2 (distributed as pip install scikit-image). This becomes the only supported release, but see (2).
  2. In skimage2, maintain a version of skimage that utilizes skimage2 as its engine, but provides users with the old API experience.
  3. Over time, deprecate the functions in the skimage1 interface: first with warnings, later with hard deprecations. Importantly, we provide clear instructions with examples on how to move from the old to the new code (we know what that looks like already, since we maintain the skimage1 “back-port”). Porting instructions also go into the migration guide.
  4. Eventually (and there’s no rush on this), remove the skimage namespace entirely, raising an error explaining to the developer what to use instead.

This repackages some of the ideas in SKIP3—so that’s worthwhile background reading.

Let me know your thoughts!


Thanks for the effort of making it a smooth transition.

I have a few questions.

What version do I get if I do pip install scikit-image and then do import skimage.

Can I do pip install scikit-image and then import skimage2?

Can I do pip install skimage2?

Sorry if this has been said before, I’m a bit confused by these skimage1 and skimage2.

:wave: @martinberoiz!

Very good questions. In the current SKIP4, mentioned above, I believe we intended to make two packages: scikit-image and skimage2. However, in the proposal above, I suggest we keep the package name scikit-image, but that it will contain both skimage and skimage2 namespaces.

This means that everyone using the scikit-image package right now will be upgraded to the new package including the skimage2 namespace, whether they use it or not.

Docs will be updated to use skimage2, and skimage namespace functions will gradually become harder to use as they complain more loudly about being deprecated.

1 Like

I like this approach! For the record, the motivation for the two separate packages was:

  • eventually, depending on scikit-image without a <=1.0 will cause a library to break. With the separate packages for scikit-image and skimage2, you can be assured that scikit-image will continue to work.
  • I was really looking forward to having our package name and import name match! :joy:

Having said this, the maintenance burden is much lower if we have the two namespaces in the one package. So I think it’s a good plan, thanks @stefanv for sharing! (I am filled with regret about not having attended the NumFOCUS summit. :joy:)

Thanks for the feedback @jni! Of course, nothing precludes us from registering skimage2, like we have skimage :slight_smile:

Or, perhaps skimage should become the de-facto package, which depends on scikit-image. I haven’t thought this aspect through.

Great suggestion @stefanv! So comparing this suggestion to the points raised in SKIP4:

Although semantic versioning [6] technically allows API changes with major version bumps, we must acknowledge that (1) an enormous number of projects depend on scikit-image and would thus be affected by backwards incompatible changes, and (2) it is not yet common practice in the scientific Python community to put upper version bounds on dependencies, so it is very unlikely that anyone used scikit-image<1.* or scikit-image<2.* in their dependency list. This implies that releasing a version 2.0 of scikit-image with breaking API changes would disrupt a large number of users.

This disruption would happen in all proposed cases were we introduce a new API. However, if we can stretch out this disruption gradually I’m all for it.

Additionally, such wide-sweeping changes would invalidate a large number of StackOverflow and other user guides.

Again, this will happen in all proposed cases were we introduce a new API.

Finally, releasing a new version with a large number of changes prevents users from gradually migrating to the new API: an old code base must be migrated wholesale because it is impossible to depend on both versions of the API. This would represent an enormous barrier of entry for many users.

This would be addressed by the new proposal.

I seem to remember a 5th principle “somebody’s law” which is to prevent subtle changes that go unnoticed but invalidate scientific results. I can’t find a reference to it anymore in SKIP4 but I remembr it as a big concern? In any case, this is also prevented by never introducing these kind of changes to the original skimage namespace but keeping them in skimage2.

All in all I’m cautiously in favor of this “package with two modules” approach. :+1:

I forgot about this concern, and it does give me pause…

@lagru That was the Hinsen principle: the same code should not yield different results after a library upgrade.

Re: broken S/O examples, we can choose for how long we maintain the old version of skimage inside of skimage2. Also remember that breakage will always describe an upgrade path, which means S/O readers will be guided to port solutions to skimage2 (and perhaps even update S/O in response!).

I think the point here is that we’ll have to keep the fully deprecated skimage1 around for quite a while, perhaps forever.

1 Like

My view is that this will happen anyway over time. We are constantly changing our API and @stefanv has some good points that this proposal deals with this in a better way than our normal deprecation cycles which are invisible after a few releases.

I hope that “forever” won’t be the case. I don’t think the maintenance cost (build chain, keeping up with dependencies, …) will be worth it long-term. Especially because earlier versions will still be available to some degree.

One more thing. Once we’ve moved code to skimage2 and updated it, it might be impossible to emulate the old behavior or results. This something we are willing to accept, right? We would still remove the old code eventually? On my part, I’d say yes to those questions.

After a while, the skimage namespace will be nothing more than a shell. There will be no image processing code inside, only the porting instructions remain. So, carrying that around won’t be much of a maintenance burden—it’s essentially just documentation.

It is true, this may happen in some cases, in which case we will need to duplicate some code back into the old namespace. I hope that those cases are few, but there is no technical barrier to making this work.

Thank you @stefanv, I like this proposition.

Concerning this famous Hinsen principle, as it was already mentioned, we already broke it at many occasions when fixing bugs (see `convex_hull_image` returns different results between 0.18.0 and 0.19.3 · Issue #6510 · scikit-image/scikit-image · GitHub for example or phase_cross_correlation is no more correct in 0.19.3 (instead of 0.18.1) · Issue #6456 · scikit-image/scikit-image · GitHub).

A partial solution to this problem is to support pip installing any version of skimage using git branches like proposed here.

We may also publish the timeline with the different scikit-image release dates and use it as reference to specify which version of the package is concerned by the S/O question according to the asking date… This is also applicable to youtube tutorials…

Unfortunately, most users won’t be able to install without wheels, and we only build those for releases. But, you can certainly specify versions as in pip install scikit-image==0.14.

This would be good to have as part of the migration docs.