How I taught an AI to think like a musician

Source separation uses Demucs (htdemucs_ft). All downstream analysis runs on separated stems, not the mix — this significantly improves tonal analysis quality by removing bleed from other instruments.

Feature extraction uses Essentia:

• Key/scale via three-algorithm consensus: EDMA, Krumhansl, Temperley
• Chord timeline with frame-accurate timestamps via HPCP chroma analysis
• Beat positions, meter, and per-beat loudness via beat tracking
• Per-stem note events carrying: time, duration, pitch, velocity, beat subdivision, and onset-to-beat offset in milliseconds
• Per-stem: onset rate, spectral centroid, energy contour, pitch range, voiced ratio

Vocal content uses Whisper for lyric transcription and CLAP (laion/larger_clap_music_and_speech) for semantic tagging — comparing audio against natural-language descriptors to surface stylistic context that MIR features alone can’t capture.

All of this runs asynchronously at upload time. Window-level analysis only runs at query time, on the selected region.

The trace above reveals the core architectural issue: the LLM was receiving track-level feature summaries and responding as if they were window-level observations. The analysis layer had correctly flagged the window:

{
  "heuristic_id": "H004_cross_stem_lock_score",
  "status": "empty_slice",
  "score": null,
  "notes": "No audio/note events in selected slice"
}

I catalogued four failure modes:

1. Out-of-window drift — track-level summaries passed as if window-specific, producing responses about the song at large when a specific 8–32 bar window was selected.
2. Chord-motion overfocus — harmony-first responses regardless of whether the interesting feature was rhythmic, timbral, or structural.
3. Single-stem rejection — the discuss gate was calibrated too conservatively; a single piano stem with rich harmonic and timing evidence over a 30-second window would be declined as “too sparse.”
4. Broad-question abstention — “what’s happening here?” collapsed into generic disclaimers rather than a focused narrative overview.

The root in all four cases: the model received feature data but no structured guidance on what kind of musical moment this was and which features mattered for it.

Primary hypothesis: a deterministic narrative substrate — salience weighting, tension arc, interaction labels, strict window scope enforcement — will improve musician-like explanation quality compared to routing-only context.

Each run used three fixed prompts across all five tracks and all annotated windows: (1) “What is happening in this section?” (2) “Describe the musical narrative in this selected window.” (3) “How do the selected stems interact here?”

Objective metrics per run:

• Scope violation rate — events mentioned outside the selected window
• Chord-dominance ratio — harmony-only claims ÷ total claims
• Cross-stem interaction density — explicit relational claims per response
• Uncertainty discipline — uncertain cases acknowledged vs. guessed
• Narrative axis coverage — presence of tension-release, phrasing, and interaction angles

Human rubric (1–5 each): in-window fidelity, musician readability, tension/release coherence, melody/rhythm integration, cross-stem narrative coherence, foreground anchoring clarity.

Per-window annotation fields: foreground stem, tension arc type (grounded / searching / intensified / release / unresolved), arc peak timestamp, interaction type (support / contrast / counterline / call-response / doubling), groove label, tension level, and free-text notes. Each annotation carried a confidence score. The annotation schema served as the ground truth for all calibration evaluation.

The router uses a focus taxonomy of six intent categories and 15 deterministic heuristics computed from the audio analysis at query time. Several heuristics surfaced significant data integrity problems during development.

H001 — Waltz/Compound Pulse

Detects compound-meter feel by checking bass emphasis on the downbeat and harmonic events on the remaining beats, weighted against a syncopation penalty. Gated to only activate when the detected meter is triple or compound triple — correctly skipped entirely for 4/4 tracks.

H003 — Offbeat Density (initially dead)

Returned missing_input: Too few onset events for every single window — 11/11 failures in the first pass. Root cause: the underlying rhythm analysis tool only dispatched results for isolated (non-overlapping) single-stem brackets. Any multi-stem selection never produced a valid quantized onset pattern. Fixed by computing offbeat proportion directly from beat-subdivision data already present on each per-window note event record, bypassing the failing dispatch path.

H004 — Cross-Stem Lock Score (corrupted data)

Combines onset coincidence rate, mean timing offset between stems, and offset jitter into a single lock score. The raw data feeding the offset calculation was wrong during calibration: Track E first verse showed a mean inter-stem offset of +4033ms; Track D piano intro showed −11502ms. A four-second offset is physically impossible in music. Root cause: when a stem had no pitched notes (drums always hit this path), the code fell back to full-track onset times from the upload-time analysis summary — a list covering the entire song, not the selected window. Injecting timestamps from outside the window made coincidence rates meaningless. Fixed by windowing onset times to the selected region before computing offsets.

H010 — Tonal Stability (redesigned twice)

The original version combined track-level key strength and track-level dissonance into a single score. Result: a constant near-identical value for every window on the same track regardless of what was selected. This created a tonal floor that won the frame competition by default whenever other signals were weak.

The redesign replaced both inputs with two window-level components: (1) pitch-class concentration — how focused the note mass is on a small set of pitch classes within the window, and (2) phrase-repetition similarity — whether the pitch-class distribution is consistent across musical phrases in the window. Multiplying the two together made the score high only when a window is both tonally focused and internally consistent — not just because professional recordings always have some pitch focus. Score range expanded from a flat 0.49–0.52 to genuine variance across windows.

A competing-signal dampening step was also added: when strong rhythmic or vocal signals are present, the tonal score is proportionally reduced. This was necessary because any focused musical phrase — even a rhythmic one — has concentrated pitch content in the technical sense.

H014 — Call/Response Clarity (structural limitation)

This heuristic measured note-level alternation between stems — event-by-event stem switches in a merged onset timeline. The problem: genuine call-and-response is phrase-level alternation. Two stems trading four-bar phrases produce a low alternation score because within each phrase all events are from the same stem. Two stems with interleaved note onsets (drums + bass playing simultaneously) produce a high score despite having no call/response relationship. Track B verse-chorus (genuine call/response, vocal over keyboard) and Track C verse (not call/response, but vocal phrasing happens to interleave with bass) scored nearly identically. The heuristic was measuring note-level interleaving, not phrase-level handoff. The interaction frame was removed from the selectable set pending a phrase-level redesign.

The five frames and their contributing signals:

Specific window outcomes from calibration:

Track E — piano intro (13s)  PASS

Human: tonal · Selector: tonal. Solo piano with no competing rhythmic or vocal signals. Tonal stability score very high (highly concentrated, consistent pitch content across phrases). Clean selection.

Track C — verse-to-chorus boundary (20s)  PASS

Human: form · Selector: form. Section boundary falls roughly mid-window. Structural signal fired at maximum; section contrast score low but present after the energy data fix. Without the energy fix, contrast scored near zero because the verse→chorus energy jump — primarily a loudness/instrument-density shift — was not captured by density and rhythm metrics alone.

Track E — late chorus (30s)  FAIL (accepted ambiguity)

Human: pulse · Selector: foreground. The 6/8 waltz feel was detectable from the compound-pulse heuristic, but the vocal alignment score pushed foreground above pulse. Root cause: the heuristics cannot perceive “rhythm asserting against vocal” — the listener’s sense that the rhythmic feel is the dominant character despite an active vocal. That perceptual relationship requires phrase-level analysis the current system doesn’t have.

Track D — piano intro (25s, live recording)  FAIL (edge case)

Human: pulse · Selector: tonal. Tonal and pulse frame scores separated by a narrow margin — effectively a coin flip. Root cause: the live recording’s room ambience caused tonal stability to read very high (diffuse room sound creates consistent, concentrated pitch-class content that the pitch concentration measure can’t distinguish from intentional harmony). Documented as a known live-recording edge case.

The frame selector’s output includes: dominant frame, optional secondary frame, an interpretive lens directive (a short plain-English instruction for the LLM), a cautions list (e.g. “do not interpret harmonic motion as instability unless timing evidence also weakens”), and a confidence score. Windows exceeding 30 seconds return frame: unknown with a window_too_long caution rather than a low-confidence guess.

The boundary object between agents is a typed findings structure. The translator receives this and nothing else from the analysis layer. A critical constraint on the analyst’s output: the plain-English summary field is limited to 2–3 sentences with no numbers, no internal identifiers, and no tool output names. Everything measurable stays in structured sub-objects; the summary is purely interpretive.

An example summary field:

“The bass and kick are loosely locked — bass is consistently rushing by ~15ms, which sits at the perceptible threshold. The chord progression is stable and tonal-center-focused with low tension. Melodic register is mid-range and moderately busy.”

The structured findings cover five musical dimensions: groove (lock quality, timing offset, feel label), harmony (key, chord motion, tension level and points, modal function), melody (contour, phrase structure, register, pitch range), texture (register relationship between stems, lead stem, masking risk), and dynamics (shape, energy level, transient character). The translator can access any of these by field but only the summary is required reading — the translator adapts its level of detail based on what the user asked.

The generation directive is assembled deterministically — no model call. Heuristic scores combine with intent keywords from the user’s request to produce a set of generation hints. The mapping is rule-based: a low offbeat density score combined with a “funky” or “groove” keyword produces a hint like “add notes at offbeat positions; 16th-note subdivision preferred.” A high register-overlap score combined with “stand out” produces a hint to push the target stem into a higher register range. The generator LLM receives these hints rather than raw scores.

The directive also carries: the generation route (surgical fix / melodic rewrite / free composition / density change / register shift / harmonic rewrite / borrow from source / compose over source stems / rhythm transfer), the current notes in the window in the same schema the generator outputs — giving the generator a concrete reference state regardless of how much is being changed — and hard constraints including the key, allowed pitch classes, and a beat-count ceiling the generator cannot exceed.

The original bug driving this redesign: the intent capture layer used a single field that was doing two jobs simultaneously — acting as a mode gate (does this trigger note generation?) and describing the conversation direction (what is the user trying to achieve?). The LLM would sometimes classify a discussion question as a generation request if the phrasing sounded like “what if.” This produced MIDI output when the user wanted analysis. The fix: mode comes from the UI only. Intent capture now produces typed structured output, not a free-text classification.

The recurring architectural failure had a single root cause: track-level data presented as if it were window-level observations. It appeared independently in three heuristics:

• H002 (backbeat) — all three Track C windows returned an identical value, including a quiet intro with no drums in the selection. The score was the drums stem’s overall track regularity, not the window-level beat.
• H009 (microtiming) — 9 out of 11 positive calibration windows returned values within a 0.004-point range — effectively identical. The fallback was using the stem’s pre-computed beat regularity from upload-time analysis. Professional recordings are consistently tight — but identically tight across all windows, which provides no discriminating signal between a rhythmically anchored section and any other.
• Original H010 (tonal stability) — constant near-identical values across every window of the same track. Key strength and dissonance were properties of the track overall, not of the window.

The diagnosis in one sentence from the design notes: “The two heuristics with the most form/tonal dominance are the two that measure track-level properties, not window-level properties.”

From a research framing in the literature review: “The likely novel pieces in your system are: applying evidence-fusion and frame-selection specifically to stem-aware analysis, where roles and cross-stem interactions are central; using the meta-layer to drive a conversational LLM that produces human-readable musical narrative, rather than just scalar predictions or tags; treating ‘interpretive frames’ as first-class objects in routing and prompt construction, effectively turning MIR insights into a controllable narrative UX.”

The current static assignments: drums is always rhythm, bass is always bass, vocals are always melody (unless semantic tags indicate backing choir). The “other” stem uses track-level onset rate, spectral centroid, and note count to pick between melody, harmony, and rhythm. Confidence values are hardcoded, not data-derived.

Three gaps the next iteration addresses:

• Phrase contiguity — distinguishing a melodic lead (sustained phrase-grouped notes) from a rhythmic fill (isolated events) within the window. Requires segmenting notes by pause duration and testing whether the resulting groups have melodic contour.
• Relative register — each stem’s pitch position relative to the other selected stems in the window. A piano playing in a high register is in a lead role; the same piano in a low register is in an accompaniment role.
• Cross-stem beat correlation — computing the actual onset offset between two stems (“bass enters consistently 12ms behind the kick”) rather than just whether they coincide. This requires beat-tracking timestamps as a phase anchor — which also unblocks the microtiming heuristic that currently falls back to track-level data.

The tools to compute all three already exist in the codebase. The work is wiring them into a window-level classification pass and integrating the result into the frame selector output.