TL;DR

We kept running into the same paradox in production: models that score higher on offline single-turn quality don't reliably win on long-horizon A/B metrics—retention, re-engagement, conversation depth. Some models look worse in head-to-head reply comparisons yet keep users talking longer, coming back more often, and opening the app the next day.

This is not evaluation noise. It is objective misalignment: current alignment methods optimize for the reply. Consumer AI needs to optimize for the trajectory.

In assistant use cases, the loop is prompt → answer → done, and treating preference as a per-reply attribute works fine. But in AI companions, interactive fiction, roleplay, and creative collaboration—the consumer AI products where users spend hours, not minutes—what users actually evaluate is a trajectory: Is the persona consistent? Is the narrative advancing? Are promises being honored? Is the emotional arc sustainable? The ultimate vote is not a thumbs-up. It is whether they come back tomorrow.

This paper proposes a shift in the training objective: if we want to train better models for long-horizon consumer AI, we need to move the optimization target from winning the next turn to earning the next session.

We call this Temporal Preference Learning.

1 The Structural Mismatch Between Single-Turn RLHF and Consumer AI

RLHF has become the dominant alignment paradigm for large language models. But it rests on a strong and rarely stated assumption: preference is a property of a single reply. In standard RLHF, this assumption is baked into the entire training objective. Given the same prompt, annotators compare two replies and pick the preferred one; a reward model learns to predict that judgment; the policy is then optimized to generate higher-scoring replies. This works beautifully in assistant settings because the task is nearly single-turn by nature: the user asks, the model answers, and most of the value can be judged within that exchange. Consumer AI products don't work this way. In AI companions, interactive fiction, roleplay, and creative co-authoring, the fundamental unit of value is not a single reply—it is an extended interaction trajectory. A conversation might span 20, 50, or hundreds of turns across multiple sessions. Users gradually form expectations around persona stability, memory continuity, narrative progression, emotional pacing, and promise fulfillment. What they evaluate is no longer "Is this reply good?" but "Is this relationship worth continuing?" This creates a structural optimization mismatch. Single-turn RLHF optimizes for: does this reply win in an isolated comparison? What actually determines long-term value in consumer products are trajectory-level properties: cross-turn consistency, sustainable narrative tension, the ability to deliver on delayed promises, and whether the user keeps investing attention. When the proxy objective diverges from the true objective, the model systematically learns to "win this turn" rather than "sustain the experience." It becomes locally more pleasing but globally more hollow—single-turn scores go up while users lose reasons to keep going.

2 Preference Myopia: Four Failure Modes

Preference is not a static label. It is a temporally structured signal. The same reply can shift from "good" to "bad" depending on when it occurs in the trajectory, what preceded it, and what it forecloses.

Why does single-turn alignment systematically fail in long-horizon settings? We call this phenomenon Preference Myopia: the optimization process sees only the current-turn cross-section of preference and is blind to its full temporal structure. In long-horizon interactions, a user's true preference has at least four dimensions:

• Instant gratification — does this turn look good in isolation? (turn-level quality)
• Trajectory momentum — does it make the conversation worth continuing? (narrative momentum)
• Delayed payoff — does it plant hooks that get honored later? (promise fulfillment)
• Temporal decay — is the same stimulus pattern wearing out? (novelty decay)

Of these four, only the first can be directly observed in a single-turn comparison. The other three are inherently temporal. Single-turn preference learning is structurally biased toward what can be redeemed in the current turn: more eloquent phrasing, stronger emotional payoff, more "impressive"-looking text. These win in offline comparisons but overdraw the future—narrative stalls, promises go unfulfilled, personas drift, stimulation overload accelerates fatigue. This is also why we found that the structure of reading speed and dwell time is a better proxy for true preference than thumbs-up/thumbs-down: it carries the dynamics of user attention and naturally encodes both temporal decay and trajectory position.

3 What We See in Production at Scale

We observe two stable, reproducible phenomena in a live system with tens of millions of daily active users.

3.1 Offline Ranking Has Unstable Predictive Power for Online Metrics

We rank models by offline single-turn preference (ELO / RM score) and validate against long-horizon online metrics: D1/D7 retention, re-engagement rate, conversation depth, and effective reading time. Intuitively, higher offline rank should predict stronger online performance. It doesn't. The rank correlation between offline ordering and online uplift is unstable, with frequent reversals where the offline winner lands at or below the median online.

3.2 Introducing Temporal Signals Systematically Improves Long-Horizon Metrics

The mismatch is not intractable. When we incorporate behaviorally grounded temporal signals into the reward—denoised effective reading time, whether the user continues chatting, whether they return—long-horizon online metrics improve systematically. We apply length normalization and anti-gaming constraints so the model cannot score points simply by padding replies or manufacturing shallow engagement. Results from a representative experiment:

When reward learns to see time, the model starts optimizing for long-term value.

4 Temporal Preference Learning

We call this training paradigm Temporal Preference Learning. The core change fits in one sentence:

Upgrade preference supervision from turn-level to trajectory-level, and explicitly model temporal structure.

4.1 Objective: From Per-Turn Reward to Trajectory Return

Let a conversation trajectory be

τ = (s₁, a₁, s₂, a₂, …, s_T, a_T)

where is the dialogue state at turn (including history) and is the model's reply. The standard single-turn RLHF objective optimizes a simple sum of per-step rewards:

This implicitly assumes that each turn's value can be assessed independently, that the quality of a reply does not depend on its position in the trajectory, and that all turns carry equal weight. In long-horizon consumer conversations, all three assumptions break down. In Temporal Preference Learning, we rewrite the objective as a temporally structured trajectory return:

Two essential differences from single-turn RLHF: First, reward is no longer equally weighted. Different turns contribute differently to the overall experience. Later payoffs, cross-session continuity, and genuine attention all mean that "one point of reward" at different trajectory positions represents different real value. γ(Δt) captures this temporal structure—not as a mechanical step-level discount, but as a function of real elapsed time that treats intra-session and inter-session influence differently. Second, reward is no longer turn-local. r_temporal(sₜ, aₜ, τ_{<t}) does not just evaluate "Is this reply good on its own?" but "Does this reply, placed in this trajectory, make the experience more worth continuing?" This lets the reward model capture properties that single-turn RMs structurally cannot see:

• Promise fulfillment: The hook planted at turn 5—was it paid off by turn 15?
• Persona consistency: Is the character's voice and stance drifting?
• Narrative progress: Is this turn advancing the story, or spinning in place?
• Emotional sustainability: Is the model burning through the user's emotional threshold?

In implementation, this decomposes as: where is baseline reply quality (similar to a standard RM), measures alignment with historical trajectory (persona, promises, facts), measures state advancement, and are tunable weights. Intuition: the unit of reward shifts from "Is this reply good?" to "Does this reply make the trajectory more worth continuing?"

4.2 Data: Trajectory-Segment Preference with Horizon Tags

Standard single-turn supervision looks like { prompt, answer_a, answer_b, preference }. Temporal Preference Learning writes temporal structure into the sample itself. The minimal data unit is a horizon-tagged trajectory-segment preference.

The key is not the number of fields. It is two structural requirements: preference labels must carry a horizon tag that distinguishes "immediately pleasing" from "delayed payoff," and samples must be bound to trajectory position and real time—otherwise the model cannot learn preference decay or long-range credit assignment.

4.3 Supervision Signals: Two Scalable Sources of Implicit Preference

With the objective and data structure defined, the key question becomes an engineering one: where do these temporally structured preference labels come from? In production, we rely on two classes of scalable implicit preference, both sharing the same property: they generate comparable differences under maximally similar context, reducing contamination from user engagement level or topic quality.

(A) Same-Session Tail Pairing

The first signal comes from within a single session. We take two segments from the same conversation trajectory that share a common prefix but diverge in their tail behavior, and construct a preference pair. Because both samples share the same user, character setup, and most of the preceding context, the difference is primarily attributable to tail-segment quality—allowing us to answer a cleaner question: given the same starting point, which continuation is more worth pursuing? This construction naturally controls for the hardest confounders. Cross-user comparisons are easily contaminated by engagement level, reading habits, and topic preference. Cross-session comparisons get biased by opening quality and character appeal. Same-session tail pairing holds these variables as constant as possible, focusing supervision on whether tail quality is improving, holding, or degrading. In implementation, we combine denoised behavioral time signals (effective reading speed, continued turns, local drop-off, abrupt session termination) to assess tail trajectory quality. To reduce natural variance in reading speed and dwell patterns across users, filtering rules should preferably rely on within-user or even within-session relative percentile comparisons, layered with necessary global absolute thresholds, rather than a single global standard.

(B) Regeneration as a Natural A/B Test

The second signal comes from user-initiated regeneration. When a user asks the model to "try again" in nearly identical context, the system obtains a remarkably clean natural experiment: same user, same context, same intent, explicit re-sampling of model output. The rejected first reply already carries a weak negative signal; the version that subsequently receives continued reading, continued conversation, or better downstream trajectory constitutes a stronger positive candidate. Compared to tail pairing, regeneration offers a more explicit same-condition contrast and is especially useful for learning the link between local reply quality and short-to-medium-term trajectory impact. Its limitation is selection bias—regeneration only occurs when the user cares enough or is dissatisfied enough to trigger it. It is best used as a high-precision supervision source, not the sole one. In practice, a more robust approach is to first score and filter regeneration pairs through a reward model (or temporal RM) before feeding them into DPO, GRPO, or other preference optimization pipelines, rather than using the raw pairs end-to-end. This absorbs noise from logging gaps, instrumentation artifacts, and idiosyncratic user behavior before converting signals into training gradients.

5 Engineering: From Signal to Trainable Gradient

The first risk of incorporating temporal signals is obvious: time can be gamed. A model can pad replies with low-information-density filler to stretch dwell time without creating real value. The engineering goal is not "make sessions longer" but: earn sustainable attention under a quality constraint—reward “worth continuing,” not “continuation itself.” In other words, the core engineering question is: how do we convert “the user stayed longer” into “what the model did right.”

5.1 Denoising: Exit ≠ Failure

Not every session end means the model failed. Users leave for phone calls, completed goals, or natural narrative closure. Treating exit/stay as raw reward contaminates gradients with false negatives. Our first step is separating "natural endings" from the penalty signal:

• Narrative closure detection — identify resolution sequences, farewell patterns, and goal-completion markers.
• Temporal cross-validation — combine click/scroll/dwell distributions with interaction density to filter background dwell, anomalous durations, and low-intent traffic.
• Labeling — only abnormal exits (abrupt departure without narrative closure) are labeled as negative signal; natural endings are excluded from the penalty gradient.

5.2 Credit Assignment: Long-Range Attribution Must Be Controlled

The biggest enemy of long-horizon training is not signal sparsity—it is misattribution. When a trajectory collapses at turn 20, which earlier turn is responsible? A long interaction horizon lets negative feedback propagate freely along the chain, making the model overly conservative and afraid to advance. Our approach: assign failure signal primarily to actions that actually changed the trajectory's direction. We don't ask "which turns appeared in this conversation?" but "which turns introduced significant state transitions, caused consistency breaks, or pushed the trajectory toward a collapse-prone region?" In practice, we use a decay-weighted attribution mechanism with a bounded lookback window that prioritizes high-risk turning points: w(t,t′)=exp(−η⋅d(st′,s+1′))⋅𝟙[break(t′)>δ] where d(sₜ', sₜ'₊₁) measures state drift magnitude (embedding distance) and break(t') detects consistency ruptures (persona shift, promise violation, narrative discontinuity). Only turns with significant drift and a consistency break receive substantial failure weight.

We allow the model to fail. We do not allow misplaced blame to drag the entire trajectory into the penalty.

5.3 Anti-Gaming: Reward Sustainable Attention, Not Padding

Any time-based reward will be exploited unless anti-gaming is designed into the objective itself. Rewarding raw duration drives the model toward verbosity, redundant confirmation, excessive empathy—superficially "sticky" but substantively hollow. Temporal reward must be gated: time enters the gradient only when a quality threshold is met. where gate is a quality gating function: gate(sₜ, aₜ) = σ(q(sₜ, aₜ) − q₀) · penalty(aₜ). Here q(sₜ, aₜ) is an independent quality score (information density, non-repetition, narrative progress), q₀ is a baseline quality threshold, σ is a sigmoid implementing a soft gate, and penalty(aₜ) discounts low-information-gain, repetitive, or template-soothing responses. The intuition is simple: time is not the reward itself, but an amplifier that activates only after quality clears the bar. What we truly reward is not “making the conversation longer,” but earning longer, more stable, and more sustainable attention without sacrificing quality.

Time is not the reward. It is an amplifier that activates only after quality clears the bar.

On the engineering side, we defend on both the training and serving fronts: length normalization and repetition penalties discount filler; progress weighting rewards event advancement, promise fulfillment, and state maintenance over raw conversation length; online guardrails pair primary metrics (retention, return rate) with red-flag monitors (padding rate, complaint rate, regen spikes)—any pattern of "duration up but return rate down" triggers an immediate rollback.

5.4 The Training Loop

Temporal Preference Learning is not a one-shot pipeline. It is a continuously calibrated feedback loop. Once the model starts optimizing temporally structured reward, user behavior, model strategy, and data distribution all shift. Implicit signals that worked under the previous policy may be reinterpreted under the new one. Without a closed loop, reward quickly falls behind policy. We design the system as a continuous iteration cycle:

Logs → Denoise → Horizon Labels → Reward Model → Policy Iteration → Online A/B → Backfill

Every step answers the same question: does the signal currently being optimized still represent real long-term user value? The moment "duration up, retention down" appears, we roll back. The loop exists to keep temporal signals honest.

6 Worked Example: Turning Reading Speed into Temporal Reward

Principles are easy to state. The hard part is grounding them in a concrete signal that is scalable, noise-resistant, and hard to game. Here we walk through reading speed as a complete worked example.

6.1 Starting Point: The Ceiling of Single-Reply Classification

We first used reading speed as a feature in a reward model trained to predict user satisfaction with individual replies. The model successfully learned the relationship between reading speed and preference—confirming sufficient signal-to-noise ratio. But this version had a structural flaw: no pairwise data, so it could only do single-reply classification, not preference ranking.

6.2 Bottleneck: Context Contamination

A more serious problem emerged in deployment. The classifier sees the entire session, and its judgments are heavily influenced by context—especially user query quality. If the first half of the conversation has high user engagement, the model tends to score later replies highly even when reply quality is degrading. Context quality and reply quality become entangled; the model cannot separate them.

6.3 Solution: Same-Session Pairing

The solution comes from TPL's core design. We construct preference pairs within the same session: given a 20-turn conversation where turn 10 meets negative-sample criteria (fast reading speed, declining downstream quality) and turn 20 meets positive-sample criteria (slow reading speed, sustained engagement), the two form a training pair. This simultaneously solves both problems:

• Context contamination eliminated — both samples share the same user, character, and preceding context. The difference comes only from tail-segment quality.
• Natural trajectory scorer — the model is effectively performing quality ranking across positions within the same session, evaluating trajectory quality trends.

6.4 Result

Same-session pairing improved the reward model's offline win rate and delivered a corresponding +2.3% D7 retention uplift online. This validates the core hypothesis of Temporal Preference Learning: temporal signals are not noise. They can be structurally incorporated into preference learning. When the optimization unit upgrades from a single reply to a trajectory, reward model prediction quality improves materially.

7 Conclusion: Optimize Trajectories, Not Turns

Single-turn RLHF taught models to win the current turn. But consumer AI does not win in turns—it wins in trajectories. Users vote with attention, with continuation, with return visits. Failure is rarely one bad reply. It is a slow degradation that single-turn labels never see. Temporal Preference Learning makes a simple claim: optimize trajectories, not turns. Write the time dimension explicitly into the objective function. Let temporal signals enter the gradient—and use denoising, guardrails, and closed-loop calibration to keep them honest. As consumer AI systems grow longer-horizon and more interactive, optimizing single replies will increasingly diverge from the true objective.

When interaction becomes long-horizon, preference inevitably becomes temporally structured. Whoever learns to optimize trajectories first gets closest to the real goal of consumer AI.