Unified PyFilesystem across RSpace (Gallery, ELN/Workspace, Inventory)

[tilorspace]

Tracking issue for implementing a unified PyFilesystem across RSpace (Gallery, ELN/Workspace, Inventory).

Jira: RSDEV-564 - PyFilesystem for RSpace ELN
Design & review: docs/unified-pyfilesystem-investigation.md (branch investigate/unified-pyfilesystem); review items R1 to R15 are referenced below by number.
Plan source: docs/unified-pyfilesystem-implementation-plan.md

---

Implementation plan: unified PyFilesystem across RSpace

Decisions this plan is built on

• Architecture: three focused FS classes (Gallery, Workspace, Inventory) composed with MountFS behind one RSpaceFilesystem(...) factory at /gallery, /workspace, /inventory, sharing one client set.
• Inventory depth: full hierarchy - containers list child containers/samples/subsamples and attachments, drillable to the leaf (R3).
• Write posture: read-only by default. writable=True enables upload/makedir; destructive ops gated separately behind allow_delete=True (R14).
• Workspace document fields: modelled as subfolders, typed (R1). remove of a file in a field unlinks the reference only, never deletes the underlying Gallery file (R5).
• makedir in Workspace: folders only.
• Names: global-ID paths (default path_style="id") plus a labelled-tree helper; opt-in path_style="labelled" for Name [GID] segments (R9).
• Opener: host-only URL, key from RSPACE_API_KEY/keyring, never in the URL (R15).
• Back-compat: old import paths kept as deprecating shims; the missing inv/fs.py is added (fixes docs: broken import in PyFilesystem usage guide (rspace_client.inv.fs does not exist) #57).

Target module layout

rspace_client/fs/
    __init__.py       # public exports: RSpaceFilesystem, the three FS classes, print_tree
    base.py           # RSpaceFSBase(FS): shared openbin shim, Mode validation, write-guard, Info helper
    paths.py          # per-branch path grammar / parser (replaces bare path_to_id) (R2)
    gallery.py        # GalleryFilesystem (moved from eln/fs.py, lazy root)
    inventory.py      # InventoryFilesystem (full hierarchy + browsable root + pagination)
    workspace.py      # WorkspaceFilesystem (new, ELN)
    rspace.py         # RSpaceFilesystem(...) factory -> MountFS, shared clients, posture flags
    tree.py           # print_tree(...) labelled view (R9 option 1)
    opener.py         # PyFilesystem opener + fs.opener entry point (R15 host-only)

rspace_client/eln/fs.py            # back-compat shim -> fs.gallery (DeprecationWarning)
rspace_client/inv/attachment_fs.py # back-compat shim -> fs.inventory (DeprecationWarning)
rspace_client/inv/fs.py            # NEW shim so the documented import works (#57)

Path grammar (R2)

Each segment below a mount point is classified by position, not by blindly stripping a prefix.

• Gallery /gallery/<GF|GL>... - every segment is a global ID; root resolves to the Gallery top folder.
• Inventory /inventory/<section-or-bench>[/page-N]/<record-gid>[/<record-gid>...]/<IF> - level 1 is a fixed section (Containers, Samples, Templates) or a bench (segment is the bench global ID, labelled with its name); optional page-N pseudo-dir under a large section; then record global IDs nesting per IC -> IC|SA|SS + IF, SA -> SS + IF, SS -> IF; leaf attachments IF.
• Workspace /workspace[/page-N]/<FL|NB|SD>.../<SD>/<fieldGID>/<GL> - root resolves to the Home tree (system folders filtered, R11); FL/NB are directories; SD is a directory of fields; a field global ID is a directory of its linked files GL.

paths.py exposes a resolver returning a typed target (kind + identifiers) that each FS method switches on. Section/bench/page tokens are recognised explicitly; anything else must be a valid global ID or the resolver raises fs.errors.ResourceNotFound.

Write-guard (base.py)

RSpaceFSBase.__init__(..., writable=False, allow_delete=False). Helpers: _require_writable() raises fs.errors.ResourceReadOnly(path) unless writable; _require_delete() raises unless allow_delete. upload/openbin(w)/makedir call _require_writable; remove/removedir call _require_delete. The factory passes both flags to every sub-FS so the posture is uniform.

Phases

Each phase is a self-contained, shippable PR that keeps existing behaviour working.

Phase 1 - Foundation and refactor (no user-visible behaviour change)

• Create rspace_client/fs/ with base.py and paths.py.
• Move GalleryFilesystem to fs/gallery.py as a thin RSpaceFSBase subclass; make its Gallery-root lookup lazy (R4).
• Move InventoryAttachmentFilesystem to fs/inventory.py as InventoryFilesystem, behaviour unchanged (hierarchy added in Phase 3).
• Add deprecating shims at the old paths plus new inv/fs.py (docs: broken import in PyFilesystem usage guide (rspace_client.inv.fs does not exist) #57).
• Port eln_fs_test.py/inv_attachment_fs_test.py; add shim-import tests asserting the DeprecationWarning.
• Acceptance: existing tests pass unchanged; old imports still work and warn; inv/fs.py import from the docs now succeeds.

Phase 2 - Unified mount, posture, names, opener

• rspace.py: RSpaceFilesystem(server, api_key, *, writable=False, allow_delete=False, path_style="id", page_size=10, flatten_pages=False) builds a MountFS, constructs one ELNClient and one InventoryClient, mounts /gallery and /inventory (workspace in Phase 4).
• Enforce read-only-by-default via the write-guard.
• tree.py: print_tree(fs, path="/", style="labelled") reading rspace.name, appending the short global ID only on sibling-name collisions (R9).
• opener.py: opener parsing rspace://<host>, key from RSPACE_API_KEY/keyring, rejecting a key@host userinfo component (R15); register the fs.opener entry point.
• Tests: listdir('/') == ['gallery','inventory'] (then + workspace after Phase 4); a read-only upload raises ResourceReadOnly; a writable one succeeds; opener with a key-in-URL is rejected; labelled tree disambiguates a duplicate name.
• Acceptance: a user can construct one FS and browse Gallery and Inventory; open_fs("rspace://host") works with the env var set.

Phase 3 - Inventory full hierarchy and pagination

• Browsable root: listdir('/inventory') returns the fixed sections plus the user's bench(es) via get_workbenches().
• Section listings use list_top_level_containers/list_samples/list_sample_templates with Pagination; expose page-N pseudo-dirs from totalHits/page_size, with flatten_pages=True following next links.
• Record directories list child records and attachments, using get_container_by_id(..., include_content=True) and the sample/subsample getters (R3).
• upload to a record attaches (writable-gated); remove of an IF deletes the attachment (delete-gated).
• Tests: root sections + bench; a container listing mixing sub-records and attachments; page maths; flatten_pages; drilling IC -> SS -> IF.
• Acceptance: Inventory navigable from the mount root down to any attachment without knowing a global ID in advance.

Phase 4 - Workspace (ELN) filesystem

• listdir('/workspace') = Home tree via list_folder_tree(typesToInclude=[...]), system folders filtered (R11). FL/NB are directories; SD is a directory of fields.
• Fields from get_document(...). Requirement (R1): only file-bearing field types are exposed as browsable directories - TEXT (embeds <fileId> links) and ATTACHMENT (holds a file). The other ten types (STRING, NUMBER, DATE, TIME, CHOICE, RADIO, REFERENCE, URI, IDENTIFIER, LINK) are omitted from the listing, so a user can only browse into fields that can actually contain files. Each exposed field is a directory keyed by its globalId, labelled with name, writable only when the FS is writable. getinfo on a document reports how many fields were hidden so the omission is not silent.
• listdir(field) = that field's files (no HTML parsing).
• upload(field, file) (writable-gated): upload_file to the Gallery, then append a <fileId=..> link to that specific field via a read-modify-write; treat a locked/signed-document API error as fs.errors.ResourceReadOnly (R6).
• download(field/<GL>, out): download that file.
• remove(field/<GL>) (delete-gated): unlink only - strip the <fileId=..> from the field content; do not delete the Gallery file (R5).
• makedir under /workspace: folders only via create_folder(notebook=False); a notebook/document path raises the usual FS error.
• Large folders/notebooks paginate with the same page-N scheme.
• Tests: Home listing filters system folders; document lists only TEXT/ATTACHMENT fields (others omitted, hidden count reported); upload appends a link to the right field; unlink-remove leaves the Gallery file; folder-only makedir; read-only default blocks writes.
• Acceptance: a user can browse to a document, pick a field, and upload/download files; destructive and write ops obey the posture flags.

Phase 5 - Docs, examples, finalisation

• Rewrite the PyFilesystem section of usage-guide.md; fix the broken inv import example.
• Update the README feature table.
• Add an examples/ script showing mount, labelled tree, and an ELN document upload.
• Record the deprecation-window decision in CHANGELOG.md.

Testing strategy

Follow the existing pattern: unittest + requests mocked via unittest.mock.patch, reusing tests/data/*.json fixtures and adding ELN document and container-with-content fixtures. Add a MountFS routing test and a fs.walk-from-root smoke test (R8). No live-server calls in CI.

Review-item coverage map

Item	Where handled
R1 typed fields	Phase 4
R2 path grammar	paths.py, Phase 1
R3 inventory hierarchy	Phase 3
R4 lazy roots	Phase 1 (Gallery), Phase 3/4 (others)
R5 field remove = unlink	Phase 4
R6 read-modify-write / locked docs	Phase 4
R7 makedir at root	Phase 1 base + per-branch tests
R8 walk from /	Phase 2 tests
R9 names / labelled tree	Phase 2 (tree.py, path_style)
R10 same file, many paths	documented, Phase 5
R11 Home overlaps Gallery / system folders	Phase 4
R12 page-N is a window	Phase 3, documented
R13 only own Home/benches	documented Phase 5; "Shared" is future work
R14 write posture	base write-guard, Phase 2
R15 opener host-only	Phase 2 (opener.py)

Risks and rollback

• Each phase is additive and back-compatible; the shims mean nothing breaks if later phases slip. Phases 3 and 4 are the largest and can ship independently.
• The Inventory include_content=True walk can be chatty; page_size and lazy per-directory fetching bound it, and flatten_pages stays opt-in.
• If MountFS routing or walk misbehaves at the root (R8), Phase 2 is the gate that catches it before Workspace lands.

Still open (not blocking Phase 1)

• Deprecation window for the old eln.fs/inv.attachment_fs imports. Suggest keep through 2.x, remove in 3.0.
• "Shared"/group content in Workspace and Inventory is out of scope here (R13).
• utils.py command-line key exposure noted in the review is not part of this work.