Revise API in `skimage.data` to expose dataset file paths?

[lagru]

With @stefanv’s permission I am moving a discussion we had here so that everyone can participate.

@stefanv:

With our latest changes to skimage.data_dir , I think we’ve rendered that variable fairly useless. E.g., if you have a new installation of skimage, that path does not even exist, but it is returned as the value of skimage.data_dir . I’m not quite sure what to do about it, because I know some people prefer for us not to copy files unnecessary. But on the other hand, it’s a very common pattern to do:

    >>> filename = os.path.join(skimage.data_dir, 'camera.png')

See, e.g.: Revisit scikit-image io section · Issue #568 · scipy-lectures/scientific-python-lectures · GitHub Any ideas on how we can improve this situation?

One option is to update our ski.data.x interface to something like:

ski.data.cat(filename=True)

So, either you get the array or the filename back

That said, we always have a data_dir with some images in it, and that’s mainly what people look for with skimage.data_dir, so perhaps we should go back to providing the “in package” directory for ski.data_dir.

@lagru:

I actually played around with ways to make the paths of our data functions accessible. Something I came up with that allowed us to not modify our functions in anyway was something like:

import skimage as ski
with ski.data.record_paths as paths:
    ski.data.eagle()
paths  # contains all internally fetched paths

Though, that was more playing around than actually thinking this is a good API. A better API I can think of is to provide a _path -function for each data retrieving function. E.g. eagle_path … Which we could even achieve by just combining the context manager approach with some __getattr__ overloading. Concerning data_dir : first I don’t think we should encourage skimage.data_dir going forward and should move away from the data_dir + "undocumented_name_i_have_to_know.png . How about the following API?

• We introduce skimage.data.cache_dir . We can do so as a function but with __getattr__ we could probably make this an attribute (I think the first option is simpler). This function / attribute will always return a valid path to an existing cache directory. That directory may be empty and might have just been created, which we properly document. The cache in the name should hint to users that files being present is not something they should rely on.
• We provide a way to retrieve paths from data functions, e.g. the _path solution or the context manager.
• We can keep data_dir around but I would probably discourage it’s usage in favor of the above API. It makes our API border fuzzy. E.g. our doctests have used data files that aren’t documented anywhere. And it’s unclear how this attribute interacts with a library that fetches data to a cache. Ideally the doesn’t have to care about the location.

If users want a directory with all of scikit-image’s data files, they can just use download_all(directory="/my/own/data_dir"). We should make download_all return the target download directory.

@stefanv:

That is heading in the right direction, I think.

What about:

• skimage.data.cache_dir ← cache location
• skimage.data.cat() → file is always copied to cache locationWill this make those Docker people upset again?

Because this is a very straightforward API, in that you can do:

ski.data.cat()
ski.data.eagle()

filenames = glob.glob(os.path.join(ski.data.cache_dir, '*.png'))

Another option, quite clean:

fn = fetch(ski.data.cat)

But, I’m just not sure… all of the options are somewhat confusing.

Perhaps we should not allow file-level handling at all. If users want to play around with a specific file, they can do:

io.imsave(skimage.data.cat(), 'cat.png')

That is very explicit, at least.