Add basic POSIX mirror tree support by AlCutter · Pull Request #1017 · transparency-dev/tessera

AlCutter · 2026-06-23T11:20:26Z

This PR adds a first cut at fleshing out the POSIX storage support for MTC mirrors.

Towards #945

roger2hk · 2026-06-24T21:51:04Z

+		if err := unlock(); err != nil {
+			panic(err)
+		}
+		m.s.mu.Unlock()


Same comment as the previous pull request. Immediately defer the mutex unlock after it is locked.

d'oh, this was grandfathered in. Fixed.

roger2hk · 2026-06-24T21:53:40Z

+
+	lh := make([][]byte, 0, maxBundles)
+	n := 0
+	for ri := range layout.Range(from, to, sourceSize) {


Suggested change

for ri := range layout.Range(from, to, sourceSize) {

for ri := range layout.Range(from, to - from, sourceSize) {

tessera/api/layout/paths.go

Lines 45 to 49 in 20980b0

// Range returns an iterator over a list of RangeInfo structs which describe the bundles/tiles

// necessary to cover the specified range of individual entries/hashes `[from, min(from+N, treeSize) )`.

//

// If from >= treeSize or N == 0, the returned iterator will yield no elements.

func Range(from, N, treeSize uint64) iter.Seq[RangeInfo] {

Good spot, fixed, and factored this func out to be shared with MigrationTarget too (which was also affected).

roger2hk · 2026-06-24T21:59:15Z

-	return 0, nil, errors.New("unimplemented")
+	targetSize := bundleIdx * layout.EntryBundleWidth
+	for b := range bundles {
+		p := len(b.Entries)


If the number of entries is 0, it will be treated as a full tile writing empty bundle data to the full tile file.

Not sure how this could happen in a valid request, but added a check to be defensive.
Also added a check that we can't get partial sized bundles mid-stream.

This function is public, so just in case.

roger2hk · 2026-06-24T22:03:54Z

+// marshalTlogEntryBundle returns a tlog-tiles compatible serialization of the provided entrybundle.
+func marshalTlogEntryBundle(b api.EntryBundle) ([]byte, error) {
+	// Max size we could possibly write out.
+	data := make([]byte, 0, len(b.Entries)*(2+1<<16))


Note that this preallocation would consume around 16 MiB for a full bundle.

That's correct - I went back and forth on this for a bit, in the end I went with this since it's "only" 16MB, and there should only really ever ~one "live" alloc of this per mirror.

Left a comment about it in case we want to go back in the future and change to a more conservative default and let Go's slice reallocator sort it out.

roger2hk · 2026-06-24T22:05:48Z

+func (m *MirrorWriter) fetchLeafHashes(ctx context.Context, from, to, sourceSize uint64) ([][]byte, error) {
+	const maxBundles = 300
+
+	lh := make([][]byte, 0, maxBundles)


There are 256 hashes in one bundle. Should this be maxBundles * layout.EntryBundleWidth instead of maxBundles?

Good catch, updated.

roger2hk · 2026-06-25T17:09:22Z

+	defer m.s.mu.Unlock()
+	unlock, err := m.s.lockFile(ctx, treeStateLock)
+	if err != nil {
+		panic(err)


If the ctx is cancelled or timed out, this will trigger the panic.

As in the comment on the other PR, I'll leave these consistent as panics for now, and we can decide what to do about them in a separate PR.

roger2hk · 2026-06-25T17:09:53Z

-	return 0, nil, errors.New("unimplemented")
+	targetSize := bundleIdx * layout.EntryBundleWidth
+	for b := range bundles {
+		p := len(b.Entries)


This function is public, so just in case.

AlCutter requested a review from roger2hk June 23, 2026 11:20

AlCutter assigned roger2hk Jun 23, 2026

AlCutter force-pushed the posix_mirror_tree_support branch 4 times, most recently from 692034b to c21f4be Compare June 24, 2026 11:56

AlCutter changed the title ~~Add POSIX mirror tree support~~ Add basic POSIX mirror tree support Jun 24, 2026

AlCutter marked this pull request as ready for review June 24, 2026 12:16

AlCutter requested a review from a team as a code owner June 24, 2026 12:16

roger2hk reviewed Jun 24, 2026

View reviewed changes

AlCutter force-pushed the posix_mirror_tree_support branch from 7944162 to bc70fd3 Compare June 25, 2026 11:25

AlCutter added 2 commits June 25, 2026 15:11

Add POSIX mirror tree support

01bf057

Comments

f13ce01

AlCutter force-pushed the posix_mirror_tree_support branch from bc70fd3 to f13ce01 Compare June 25, 2026 15:12

roger2hk approved these changes Jun 25, 2026

View reviewed changes

AlCutter merged commit 577ff18 into transparency-dev:main Jun 25, 2026
20 checks passed

AlCutter deleted the posix_mirror_tree_support branch June 25, 2026 18:40

	for ri := range layout.Range(from, to, sourceSize) {
	for ri := range layout.Range(from, to - from, sourceSize) {

	// Range returns an iterator over a list of RangeInfo structs which describe the bundles/tiles
	// necessary to cover the specified range of individual entries/hashes `[from, min(from+N, treeSize) )`.
	//
	// If from >= treeSize or N == 0, the returned iterator will yield no elements.
	func Range(from, N, treeSize uint64) iter.Seq[RangeInfo] {

Uh oh!

Conversation

AlCutter commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AlCutter commented Jun 23, 2026 •

edited

Loading