IO optimization for loading dataset

[dentiny]

This PR optimizes lance dataset loading, mainly a few changes

• In the current implementation, there're 4 sequential IO requests, including dataset existence, load schema, load columns, load dataset data; in this PR I merged them into two: metadata loading and actual data loading
• Update backend API from async to sync, because the lance dataset API we use is sync already
• Parallelize the schema loading and actual data loading via FastAPI thread-pool execution

Followup items

• Inline blob rendering could lead to latency increase (related PR Support blob type visualization #72), so I will followup with a blob lazy loading PR

AI usage disclaimer:
GPT-5.5 helped me make the code change, I deployed my own fork to k8s and verified it worked

[gordonmurray]

Thanks for the performance work. The synchronous FastAPI handlers and combined metadata request are a good fit for this application, and the existing matrix checks are useful.

I’m requesting a few changes before merge:

1. Add coverage for the new GET /datasets/{dataset_name}/metadata response when the Arrow schema has metadata, including non-ASCII and non-UTF-8 byte values. The endpoint currently returns schema.metadata directly, while the existing API contract is JSON; this should be deliberately serialized and tested rather than relying on FastAPI’s encoder.

2. Add/adjust the behavioral specification in docs/spec.md for the new endpoint and the metadata-loading behavior. The frontend now depends on /metadata instead of the separate schema/columns requests.

3. Add the required CHANGELOG.md entry under [Unreleased], referencing the relevant issue (or open/link an issue for this optimization if one does not exist).

4. Please run the full workspace matrix after the final changes, since this alters all dataset handlers and introduces a new endpoint used by the frontend.

Once those are addressed, I expect this to be a good candidate to merge.

[dentiny]

Once those are addressed, I expect this to be a good candidate to merge.

Thanks for the detailed feedback! I should address them all :-)

[gordonmurray]

The spec request in my review was my mistake. docs/spec.md is not in this repo. It is a local file that I keep outside the checkout. The only mention of it upstream is a docstring in backend/tests/test_api.py. You had no way to see the file. I will remove that stale reference.

Please remove docs/spec.md from this PR. Your tests already cover the new endpoint. That is what I wanted. I will update my own copy after the merge.

Please ignore the same request in #71 and #72.

The rest of the PR is good. The metadata serialization is correct, and the test for non-ASCII keys and non-UTF-8 bytes covers the case I asked about. It also corrects /schema, which returned the raw bytes dictionary before. Thanks for opening #75 and for the changelog entry that cites it.

I ran the full version matrix against your branch. All 33 tests pass on the seven lancedb versions, 0.3.1 to 0.33.0. When you remove the spec file, I will merge this.

[gordonmurray]

I pushed two commits to your branch, to save you a round trip.

The first commit removes docs/spec.md. The second merges main. That merge resolves a conflict in CHANGELOG.md. My own changelog entry from #78 went to the same place as yours, so the two lines collided. Both lines are in the file now.

Nothing else changed. I ran the full version matrix on the merged result. All 33 tests pass on the seven lancedb versions, 0.3.1 to 0.33.0.

I did not force push, so your local branch will fast-forward. You do not need to do anything.

[gordonmurray]

All four points from my earlier review are addressed.

The metadata serialization is correct, and the tests cover non-ASCII and non-UTF-8 schema metadata. The changelog entry cites #75.

I ran the full version matrix on the merged branch. All 33 tests pass on the seven lancedb versions, 0.3.1 to 0.33.0. CI is green.

Thanks for the work on this.