Follow symbolic links in DirectoryStore

[alimanfoo]

Not sure if this is possible or how easy to implement, but it might be convenient if DirectoryStore followed symbolic links, allowing symlinks to be used to create cross-links within a hierarchy, or to link between different stores (e.g., to create different views of data within a store).

xref #297

[jakirkham]

How would we handle this in other stores?

[alimanfoo]

I think it's only relevant to DirectoryStore and NestedDirectoryStore as these expose the hierarchy as a directory structure, so symlinks to directories provides a means to do some useful things. Not sure symlinks would be useful or make any sense for other stores.

[jakirkham]

Guess I was imagining them being some text with a path. Though this really depends on whether links are first class objects (like how they are in HDF5).

This might also come up if a user converts between two different types of stores.

[jakirkham]

FWIW another use case for links (as well as object and region references) is NWB where they want to reference data in multiple places. They actually started implementing something on top of Zarr, but the lack of these features was a blocker for them.

xref: NeurodataWithoutBorders/pynwb#230 (comment)

[alimanfoo]

Thanks for following this up. FWIW to keep things organised I'd suggest raising a separate issue to cover discussion of implementing links and region references a la HDF5, and keeping this issue scoped to the lower-level issue of whether DirectoryStore should follow symbolic file system links.

…

On Fri, 4 Jan 2019 at 07:02, jakirkham ***@***.***> wrote: FWIW another use case for links (as well as object and region references) is NWB where they want to reference data in multiple places. They actually started implementing something on top of Zarr, but the lack of these features was a blocker for them. xref: NeurodataWithoutBorders/pynwb#230 (comment) <NeurodataWithoutBorders/pynwb#230 (comment)> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#298 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8QowXmQuYBtVX3n3a8nPrSiX9mGvkks5u_vybgaJpZM4WZVyj> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health Big Data Institute Li Ka Shing Centre for Health Information and Discovery Old Road Campus Headington Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 or +44 (0)7866 541624 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: @alimanfoo <https://twitter.com/alimanfoo> Please feel free to resend your email and/or contact me by other means if you need an urgent reply.

[jakirkham]

Sure. Have raised issue ( #389 ).

[d-v-b]

today the file system store makes no promises about how symlinks are handled. if we did naively follow symlinks, we could create topological abominations, e.g. a group that contains its ancestors. handling this would add considerable complexity to the zarr data model, or the local file system store, all for something that has no analogue in cloud object storage. so I will close this as not planned.

that being said, there are problems with how we handle symlinks today in the file system store that we should resolve. I will open issues for these.