Suggestions on the DJ engine (MAD-759)

docs/adr/0010-suggestions.md

@@ -20,6 +20,8 @@ The parts of autopilot that find songs like a seed move into `internal/suggest`:
 - **Songs** come from the asker's usable links only (their own and shared ones), so every suggestion can be queued by them. Services that recommend give similar songs and the artist's top songs. Services that only search (Spotify, nugs.net) give more by the seed's artist, found by searching their name. Each seed asks at most two services, recommenders first.
 - **Not suggested:** anything waiting, playing, or in the room's last 200 plays — nor songs by an artist the room turned away. That's the DJ's rule (ADR 0012): a skip turns that vibe away while it outweighs the room's liking (full listens, queued songs, hearts), and it fades with a 2-hour half-life, so a skip long ago no longer counts (MAD-762). Songs are compared the way autopilot compares them.
 - **The list** mixes seeds, taking turns. Picks within a seed are random among the top of its candidates, so a reshuffle gives a different list. Each suggestion carries its seed (`because`), for a byline like "Like Heroes, Bob's pick".
+- **The DJ picks them (MAD-759).** When the server has a music graph (ADR 0012), suggestions come from the same DJ as autopilot: the room's taste profile, the walk through similar artists, scoring, and cross-service matching, at explore 25 (near the vibe). `mine` narrows the taste to the asker's own (`dj.Request.Member`). `group` uses the room's, members' tastes mixed as for autopilot. `source=queue` reads the taste from the members' songs playing and waiting (`dj.Request.Queued`), the asker's for `mine`; what played through stands in when nothing's queued. What the room heard and turned away stays the room's either way. The DJ finds each pick only on the asker's links that search, so every suggestion can still be queued by them. A list of n draws 2n candidates from a shortlist at least that long, at most 2 songs per artist, and looks for up to 6 at once. Each suggestion's `because` is the member's song by the artist it leads from; a pick only autopilot's songs led to has none, and is left out.
+- **Without the DJ.** With no graph, or when it knows nothing near the vibe, the services' own recommendations stand in, as described above (seeds, `Finder.Like`).
 - **Cost:** a list is cached in memory for two minutes per room, person, scope, source and queue version; `refresh=true` builds a new one. Building one is bounded at 15 seconds; whatever was found by then is returned.
 - Guests can ask in their own room; they get songs from the server's shared services.
 

docs/adr/0012-smart-dj.md

@@ -81,7 +81,7 @@ MusicBrainz joins when `SYNCPHONY_MUSICBRAINZ_URL` isn't off. `TestLive` in `int
 
 ## The DJ (stage 2)
 
-`internal/dj` picks songs, and `autopilot` asks it first. It knows nothing about queues or turns: it's given what the room did, and returns songs to try, best first. It doesn't import `suggest`, so suggestions can move onto it later (MAD-759).
+`internal/dj` picks songs, and `autopilot` asks it first. It knows nothing about queues or turns: it's given what the room did, and returns songs to try, best first. It doesn't import `suggest`. Suggestions (ADR 0010) ask it too (MAD-759).
 
 ### The room's taste (MAD-753)
 
@@ -253,6 +253,6 @@ If the graph has no sources, knows nothing near the room's taste, or none of its
 - The first fill after a cold start knows only the artists it fetched in time. The warmer catches up from the songs being queued.
 - `AutopilotInfo.reason` is stored with each song; it isn't in the API until MAD-760.
 - A room's long-term taste is the room's: it's deleted with the room. It names members by ID, and keeps an artist only while its likes or skips are above 0.005, about 15 nights of silence for one play.
-- Suggestions (ADR 0010) turn artists away by the same rule as the DJ (`suggest.TurnedAway` reads `dj.Profile.Avoid`), so a skip fades there too (MAD-762). MAD-759 moves the rest of suggestions onto the DJ.
+- Suggestions (ADR 0010) come from the DJ too (MAD-759): the asker's taste for "your vibe", the room's for "group vibe", found only on the asker's links. Without a graph they fall back on the services' recommendations, turning artists away by the DJ's rule (`suggest.TurnedAway` reads `dj.Profile.Avoid`), so a skip fades there too (MAD-762).
 - Tempo is often unknown: Deezer has a BPM for some songs and 0 for many, and few libraries tag it. Set flow treats 0 as unknown, so a room with little known about its songs plays as it did before; it flows better as the cache warms.
 - Energy is a guess from tags and tempo. A band tagged "rock" plays its ballads as rock.

server/cmd/syncphony/main.go

@@ -226,9 +226,12 @@ func run() error {
 	pilot := autopilot.New(db, roomSvc, queueSvc, a.links, presence)
 	pilot.Player = player
 	pilot.Memory = &dj.Memory{DB: db}
+	// Suggestions (ADR 0010) ask the same DJ.
+	suggestions := suggest.New(db, roomSvc, a.links)
 	if len(graph.Sources()) > 0 {
 		pilot.Graph = graph
 		pilot.Memory.Graph = graph
+		suggestions.Graph = graph
 	}
 	defer pilot.Close()
 	onChange, onUpdate := queueSvc.OnChange, roomSvc.OnUpdate
@@ -243,7 +246,7 @@ func run() error {
 	nightSvc := nights.New(db, a.bus)
 	api := &httpapi.Server{
 		Version: version, StartedAt: time.Now().UTC(), Admin: admin.New(db, filepath.Join(cfg.DataDir, "backups")), Auth: accounts, Links: a.links, Lyrics: lyricsSvc, LinerNotes: notes, Artwork: art, Palettes: palettes,
-		Rooms: roomSvc, Queue: queueSvc, Playback: player, Nights: nightSvc, Suggest: suggest.New(db, roomSvc, a.links), Autopilot: pilot,
+		Rooms: roomSvc, Queue: queueSvc, Playback: player, Nights: nightSvc, Suggest: suggestions, Autopilot: pilot,
 		Bus: a.bus, Presence: presence,
 		BaseURL: cfg.BaseURL, TrustedProxies: cfg.TrustedProxies,
 	}

server/internal/autopilot/autopilot.go

@@ -357,7 +357,7 @@ func (f *fill) remember(upcoming []store.QueueItem) {
 	for i, h := range f.history {
 		f.seen.Item(h.QueueItem)
 		if i < discoveryAvoid {
-			if a := suggest.ArtistKey(suggest.TrackOf(h.QueueItem)); a != "" {
+			if a := suggest.ArtistKey(dj.TrackOf(h.QueueItem)); a != "" {
 				f.recentArtists[a] = true
 			}
 		}
@@ -556,7 +556,7 @@ func (f *fill) fromGraph(ctx context.Context, users []string) []pick {
 			}}
 		}
 		if p.Seed.ID != "" {
-			seed := suggest.TrackOf(p.Seed)
+			seed := dj.TrackOf(p.Seed)
 			info.SeedItemID, info.SeedTitle = p.Seed.ID, seed.Title
 			if len(seed.Artists) > 0 {
 				info.SeedArtist = seed.Artists[0].Name

server/internal/dj/dj.go

@@ -15,6 +15,7 @@ import (
 	"cmp"
 	"context"
 	"log/slog"
+	"sync"
 	"time"
 
 	"github.com/madeofpendletonwool/syncphony/server/internal/match"
@@ -48,6 +49,8 @@ const (
 	linksPerSong = 2
 	// defaultWant is how many picks Choose returns, unless asked.
 	defaultWant = 2
+	// findAtOnce is how many picks are looked for on services at once.
+	findAtOnce = 6
 )
 
 // Engine picks songs.
@@ -78,6 +81,13 @@ type Request struct {
 	// EnergyCurve follows the room's energy curve (rooms.Autopilot): a
 	// gentle rise over a session, then settling, moved by the time of day.
 	EnergyCurve bool
+	// Member narrows the taste to one member's, for their own suggestions
+	// ("your vibe", ADR 0010). The room's skips still turn artists away.
+	Member string
+	// Queued reads the taste from the members' songs playing and waiting
+	// (Member's, if set), when there are any, so a queued change of vibe
+	// is the vibe.
+	Queued bool
 }
 
 // Pick is a song the DJ chose.
@@ -131,51 +141,82 @@ func (e *Engine) Choose(ctx context.Context, r Request) []Pick {
 	if in.Related == nil {
 		in.Related = Related(ctx, e.Graph)
 	}
-	p := NewProfile(in)
+	p := NewProfile(in).narrow(in, r.Member, r.Queued)
 	if len(p.Artists) == 0 || len(r.Links) == 0 {
 		return nil
 	}
 	order := e.shortlist(ctx, p, r, in.Tags)
 	want := cmp.Or(r.Want, defaultWant)
+	sessions := &lockedSessions{s: r.Sessions}
 	var out []Pick
-	for _, c := range order {
+	taken := map[provider.TrackRef]bool{}
+	for len(order) > 0 && len(out) < want {
 		if ctx.Err() != nil {
 			break // out of time: play what was found
 		}
-		t, ok := e.find(ctx, r, p, c)
-		if !ok {
-			continue
-		}
-		taste := p.Artists[c.via]
-		pick := Pick{Track: t, Why: Why{
-			Kind: c.kind, Similarity: c.similarity, Popularity: c.popularity, Novelty: c.novelty,
-			Prior: c.prior, Score: c.score, DeepCut: c.deepCut, Sources: c.sources, LovedAt: c.lovedAt, Flow: c.flow,
-			Bridge: c.bridge, Turn: c.turn,
-		}}
-		switch {
-		case c.kind == KindThrowback:
-			pick.Why.Via = r.LongTerm.Artists[c.via].Name
-		case taste != nil:
-			pick.ForUser, pick.Seed, pick.Why.Via = taste.Fan(), taste.Seed, taste.Artist.Name
+		// Look for as many as are still wanted at once, in the draw's
+		// order, so a fill tries the same candidates as one by one.
+		batch := order[:min(len(order), want-len(out), findAtOnce)]
+		order = order[len(batch):]
+		found := make([]provider.Track, len(batch))
+		ok := make([]bool, len(batch))
+		var wg sync.WaitGroup
+		for i, c := range batch {
+			wg.Go(func() { found[i], ok[i] = e.find(ctx, r, sessions, p, c) })
 		}
-		// A bridge is for the one of its members served longer ago; a turn,
-		// for its member, if the song is near their taste.
-		switch {
-		case c.bridge != nil:
-			pick.ForUser = p.servedLongestAgo(c.bridge.Users[:])
-			pick.Seed = p.memberSeed(pick.ForUser, c.reachVia[pick.ForUser])
-		case c.turn != "" && c.reach[c.turn] >= bridgeReach:
-			pick.ForUser = c.turn
-			pick.Seed = p.memberSeed(c.turn, c.reachVia[c.turn])
-		}
-		out = append(out, pick)
-		if len(out) >= want {
-			break
+		wg.Wait()
+		for i, c := range batch {
+			if !ok[i] || taken[found[i].Ref] {
+				continue
+			}
+			taken[found[i].Ref] = true
+			out = append(out, p.pick(r, c, found[i]))
 		}
 	}
 	return out
 }
 
+// pick is candidate c, found as t, with why it was chosen and whom for.
+func (p Profile) pick(r Request, c candidate, t provider.Track) Pick {
+	taste := p.Artists[c.via]
+	pick := Pick{Track: t, Why: Why{
+		Kind: c.kind, Similarity: c.similarity, Popularity: c.popularity, Novelty: c.novelty,
+		Prior: c.prior, Score: c.score, DeepCut: c.deepCut, Sources: c.sources, LovedAt: c.lovedAt, Flow: c.flow,
+		Bridge: c.bridge, Turn: c.turn,
+	}}
+	switch {
+	case c.kind == KindThrowback:
+		pick.Why.Via = r.LongTerm.Artists[c.via].Name
+	case taste != nil:
+		pick.ForUser, pick.Seed, pick.Why.Via = taste.Fan(), taste.Seed, taste.Artist.Name
+	}
+	// A bridge is for the one of its members served longer ago; a turn,
+	// for its member, if the song is near their taste.
+	switch {
+	case c.bridge != nil:
+		pick.ForUser = p.servedLongestAgo(c.bridge.Users[:])
+		pick.Seed = p.memberSeed(pick.ForUser, c.reachVia[pick.ForUser])
+	case c.turn != "" && c.reach[c.turn] >= bridgeReach:
+		pick.ForUser = c.turn
+		pick.Seed = p.memberSeed(c.turn, c.reachVia[c.turn])
+	}
+	return pick
+}
+
+// lockedSessions opens links one at a time, for finding picks at once:
+// a Sessions like suggest.Finder isn't safe for concurrent use. The
+// sessions it opens are.
+type lockedSessions struct {
+	mu sync.Mutex
+	s  Sessions
+}
+
+func (l *lockedSessions) Open(ctx context.Context, linkID string) (provider.Session, bool) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	return l.s.Open(ctx, linkID)
+}
+
 // shortlist returns the candidates to try, in order: drawn from the best
 // the walk found, scored on how they flow from what the room just heard.
 func (e *Engine) shortlist(ctx context.Context, p Profile, r Request, tags map[string][]musicgraph.Tag) []candidate {
@@ -189,11 +230,13 @@ func (e *Engine) shortlist(ctx context.Context, p Profile, r Request, tags map[s
 	cancel()
 	deepCut := e.Rand(100) < deepCutPercent
 	mx := mixFor(p)
-	pool := e.flowing(ctx, p, tags, diverse(mixed(score(cands, p, r.LongTerm, x, deepCut), p, mx)), r.EnergyCurve, r.Now)
+	// A long list (suggestions) draws from a longer shortlist.
+	n := max(drawn, 2*cmp.Or(r.Want, defaultWant))
+	pool := e.flowing(ctx, p, tags, diverse(mixed(score(cands, p, r.LongTerm, x, deepCut), p, mx), max(shortlist, n)), r.EnergyCurve, r.Now)
 	for i := range pool {
 		pool[i].turn = mx.turn
 	}
-	return draw(pool, drawn, lerp(tempFamiliar, tempExplore, x), e.Rand)
+	return draw(pool, n, lerp(tempFamiliar, tempExplore, x), e.Rand)
 }
 
 // CachedTags returns the tags of the artists the room played lately, as
@@ -222,14 +265,14 @@ func CachedTags(ctx context.Context, g Graph, in Input) map[string][]musicgraph.
 
 // find looks for a candidate on the room's services: first the one the
 // song it leads from played from, then the others in order.
-func (e *Engine) find(ctx context.Context, r Request, p Profile, c candidate) (provider.Track, bool) {
+func (e *Engine) find(ctx context.Context, r Request, sessions Sessions, p Profile, c candidate) (provider.Track, bool) {
 	want := provider.Track{Title: c.song.Title, ISRC: c.song.ISRC, MBID: c.song.MBID, Artists: []provider.ArtistCredit{{Name: c.song.Artist.Name}}}
 	links := r.Links
 	if taste := p.Artists[c.via]; taste != nil && taste.Seed.ID != "" {
 		links = first(links, source(taste.Seed))
 	}
 	for _, l := range links[:min(len(links), linksPerSong)] {
-		sess, ok := r.Sessions.Open(ctx, l.ID)
+		sess, ok := sessions.Open(ctx, l.ID)
 		if !ok {
 			continue
 		}

server/internal/dj/dj_test.go

@@ -7,6 +7,7 @@ import (
 	"database/sql"
 	"encoding/json"
 	"fmt"
+	"maps"
 	"math"
 	"slices"
 	"sync"
@@ -123,6 +124,41 @@ func TestProfileSkips(t *testing.T) {
 	}
 }
 
+// Suggestions narrow the taste to a member's, or to what's queued (ADR
+// 0010); the room's skips still turn artists away.
+func TestProfileNarrow(t *testing.T) {
+	alices := item("alice", "Hers", "One")
+	h := []store.ListHistoryRow{
+		played(alices, now.Add(-10*time.Minute), store.EndFinished),
+		played(item("bob", "His", "Two"), now.Add(-20*time.Minute), store.EndFinished),
+		skipped(item("bob", "Skipped", "Three"), now.Add(-30*time.Minute), 5*time.Second),
+	}
+	waiting := item("bob", "Waiting", "Four")
+	waiting.State = store.ItemQueued
+	in := Input{History: h, Upcoming: []store.QueueItem{waiting}, Now: now}
+	room := NewProfile(in)
+
+	mine := room.narrow(in, "alice", false)
+	if len(mine.Artists) != 1 || weight(mine, "Hers") != 1 || mine.Artists["hers"].Seed.ID != alices.ID {
+		t.Errorf("alice's taste = %v: want only Hers, seeded by her song", slices.Collect(maps.Keys(mine.Artists)))
+	}
+	if !mine.Avoid["skipped"] || !mine.Heard[SongKey("His", "Two")] {
+		t.Error("alice's taste forgot what the room skipped or heard")
+	}
+	if len(mine.memberVibes) != 1 {
+		t.Errorf("alice's taste mixes %d members", len(mine.memberVibes))
+	}
+
+	queued := room.narrow(in, "", true)
+	if len(queued.Artists) != 1 || weight(queued, "Waiting") != 1 {
+		t.Errorf("the queued taste = %v: want only Waiting", slices.Collect(maps.Keys(queued.Artists)))
+	}
+	// Alice queued nothing: what she played stands in.
+	if both := room.narrow(in, "alice", true); weight(both, "Hers") != 1 || len(both.Artists) != 1 {
+		t.Errorf("alice's queued taste = %v: want Hers", slices.Collect(maps.Keys(both.Artists)))
+	}
+}
+
 // A skip long ago, decayed to almost nothing, doesn't turn an artist away
 // (MAD-762); a recent one does.
 func TestProfileSkipsFade(t *testing.T) {

server/internal/dj/live_test.go

@@ -90,8 +90,8 @@ func TestLive(t *testing.T) {
 	}
 	for _, explore := range []float64{0, 0.5, 1} {
 		cands := e.walk(t.Context(), p, lt, explore, nil)
-		show(fmt.Sprintf("explore %.1f, %d candidates", explore, len(cands)), diverse(score(cands, p, lt, explore, false)))
+		show(fmt.Sprintf("explore %.1f, %d candidates", explore, len(cands)), diverse(score(cands, p, lt, explore, false), shortlist))
 	}
 	back := Throwbacks(p, lt)
-	show(fmt.Sprintf("a throwback fill, bringing back %v", back), diverse(score(e.walk(t.Context(), p, lt, 0.25, back), p, lt, 0.25, false)))
+	show(fmt.Sprintf("a throwback fill, bringing back %v", back), diverse(score(e.walk(t.Context(), p, lt, 0.25, back), p, lt, 0.25, false), shortlist))
 }

server/internal/dj/profile.go

@@ -205,6 +205,50 @@ func NewProfile(in Input) Profile {
 	return p
 }
 
+// narrow reads a narrower taste than the room's, for suggestions (ADR
+// 0010): the members' songs playing and waiting, if queued and there are
+// any; and of those, one member's, if member isn't "". What the room
+// heard, turned away and doubts stays the room's.
+func (p Profile) narrow(in Input, member string, queued bool) Profile {
+	if member == "" && !queued {
+		return p
+	}
+	if queued {
+		var up []store.QueueItem
+		for _, it := range in.Upcoming {
+			if !it.IsAutopilot() && (member == "" || it.AddedBy == member) {
+				up = append(up, it)
+			}
+		}
+		if len(up) > 0 {
+			q := Input{Upcoming: up, Now: in.Now}
+			p.Artists, p.RecentSongs = tastes(q, reactions(q, profileReach), time.Time{}).Artists, up
+		}
+	}
+	if member != "" {
+		most := 0.0
+		for _, t := range p.Artists {
+			most = max(most, t.ByMember[member])
+		}
+		mine := map[string]*Taste{}
+		for a, t := range p.Artists {
+			w := t.ByMember[member]
+			if w <= 0 {
+				continue
+			}
+			seed := t.seeds[member]
+			mine[a] = &Taste{
+				Artist: t.Artist, Weight: w / most, Plays: t.Plays, ByMember: map[string]float64{member: w / most},
+				Seed: seed, seeds: map[string]store.QueueItem{member: seed},
+			}
+		}
+		p.Artists = mine
+		p.RecentSongs = slices.DeleteFunc(slices.Clone(p.RecentSongs), func(it store.QueueItem) bool { return it.AddedBy != member })
+	}
+	p.members(in)
+	return p
+}
+
 // tastes adds up the room's reactions into the artists it likes and turns
 // away, those since boost (if not zero) counting vibeBoost times as much.
 func tastes(in Input, rs []reaction, boost time.Time) Profile {

server/internal/dj/score.go

@@ -28,7 +28,8 @@ const (
 	// shortlist is how many of the best candidates a pick is drawn from,
 	// at most perArtist of any one artist, so a pick chooses between
 	// artists rather than among one's songs; drawn is how many are drawn
-	// to try in order.
+	// to try in order. Asking for more picks draws twice as many, from a
+	// shortlist at least as long.
 	shortlist = 25
 	perArtist = 2
 	drawn     = 8
@@ -116,16 +117,16 @@ func score(cands []candidate, p Profile, lt LongTerm, explore float64, deepCut b
 }
 
 // diverse keeps, in order, the best perArtist songs of each artist, up to
-// shortlist songs.
-func diverse(scored []candidate) []candidate {
+// n songs.
+func diverse(scored []candidate, n int) []candidate {
 	var out []candidate
 	count := map[string]int{}
 	for _, c := range scored {
 		if count[c.artist] < perArtist {
 			count[c.artist]++
 			out = append(out, c)
 		}
-		if len(out) == shortlist {
+		if len(out) == n {
 			break
 		}
 	}
@@ -136,7 +137,7 @@ func diverse(scored []candidate) []candidate {
 // probability that grows with its score (softmax at temp), without
 // replacement. rand returns a number in [0, n).
 func draw(scored []candidate, n int, temp float64, rand func(int) int) []candidate {
-	pool := diverse(scored)
+	pool := diverse(scored, max(shortlist, n))
 	var out []candidate
 	for len(out) < n && len(pool) > 0 {
 		top := pool[0].score