Time to First Token: What Claude's Message Threads Change

If your AI agent feels sluggish, the problem is often time to first token: the pause between sending a request and seeing the first word of the reply. On November 2, Anthropic put a feature called message threads into public beta for the Claude Messages API, and it targets exactly that pause in long-running agents (Anthropic).

The idea is simple. Instead of resending the entire conversation on every turn, your app sends only what's new plus the ID of the last reply. Anthropic says that with threads on by default for most Claude Code users, requests carrying 500k to 1M tokens saw median time to first token drop by about 35%. It saves time, not money, and it comes with strings attached.

TL;DR#

• Message threads let a Claude agent send only new messages each turn, and Anthropic reports about 35% lower median time to first token on 500k to 1M token requests.
• Token pricing and rate limits don't change, threads expire after 5 minutes of inactivity, and they aren't eligible for zero data retention or HIPAA.
• It's worth testing for agents that carry huge context. Short chats, batch jobs and regulated data should stay on the normal path.

What Anthropic shipped with message threads#

Until now, the Messages API has been stateless. Every agent turn uploads the full history: system prompt, tools, messages and tool results. For an agent that has read half a codebase, that's hundreds of thousands of tokens on every step.

Threads change the request shape. Per the official docs, you send the full conversation once with a create flag, then continue with only the new messages:

// first turn: full history, system prompt and tools
"thread": { "type": "create" }

// later turns: only new messages, no system or tools
"thread": { "type": "continue", "previous_message_id": "<last response id>" }

Every request in a thread needs the beta header message-threads-2026-08-12, and each response ID can be continued once, so the thread is append-only. Anthropic holds the thread in memory, encrypted, without writing it to disk. It expires 5 minutes after the last request and is deleted no more than an hour after last use. The feature is available on the Claude API, Claude Platform on AWS, Claude models on Google Cloud and Microsoft Foundry (Anthropic).

Why time to first token matters more than you think#

Time to first token (TTFT) covers everything before the model starts answering: uploading the request, queueing and processing the prompt. It's the part users feel as "is this thing broken?"

In an agent, it compounds. A task might run 15 or 20 loop steps, and every step pays the TTFT tax again. A few seconds of dead air per step turns a two-minute job into a five-minute one, and people start doing it by hand again.

Where the latency actually comes from#

LLM latency in agent loops stacks up in three places: payload size, the model reading the prompt, and your own code between turns (slow tools, retries, database lookups). Threads only attack the first. Claude still reads the full conversation on every turn (Anthropic), and in our experience a lot of real-world slowness hides in the third.

The trade-offs: what threads fix and what they don't#

Here's how threaded requests compare with the classic stateless approach:

It saves time, not money#

Token usage stays the same because the model still processes the whole conversation (The Next Web). If cost is your problem, prompt caching is still the lever, and the docs are explicit that a thread doesn't cache your prompt for you. You set cache breakpoints yourself, with a limit of four per threaded request.

Threads are best-effort#

There's no API to list, read or delete threads. If one expires, a continue request returns a 404 with thread_not_found, and you have to resend the full conversation. Fixed fields such as the model, tool choice and thinking settings must match across the thread or you get a 400. A turn answered by a fallback model isn't stored at all (Unite.AI). So your app must keep its own copy of every message regardless.

What this means for businesses running AI workflows#

Most business automations don't carry a million tokens of context. A support triage agent or invoice matcher works with a ticket, a few records and some instructions. Anthropic only published figures for the 500k to 1M token range, so don't expect much on small requests.

It gets interesting for heavier work: agents that review long contract sets or work through a large codebase. Those loops carry big context for many steps, so a third off each wait adds up.

There are three cases where we'd keep threads off for now:

• Regulated data. Threads aren't ZDR or HIPAA eligible, and organizations with HIPAA readiness enabled can't use them. If your agent touches patient data, that settles it.
• Batch jobs. Overnight reconciliation runs through Message Batches don't support threads. Batch work isn't latency-sensitive anyway.
• Multi-provider setups. If you route between vendors, server-side state is one more provider-specific behavior. Keep it behind your own interface.

How to reduce LLM latency in your agent: a checklist#

Measure first, then work through this list:

1. Log TTFT separately from total time for every model call, tagged by agent step.
2. Check context size per turn. If most calls are under a few thousand tokens, threads won't buy you much. Trim context first.
3. Check your compliance flags. If ZDR or HIPAA applies, stop here and stay stateless.
4. Store the full transcript in your own database. Threads are a speed-up, not a source of truth.
5. Build the replay path first. On thread_not_found, rebuild the request from your transcript and create a new thread.
6. Pin the fixed fields (model, tools, thinking settings) per thread so you never hit a fingerprint mismatch mid-task.
7. Set cache breakpoints deliberately, since threads won't do it for you.
8. Stream responses to users so perceived latency drops.
9. Roll out behind a flag and compare TTFT and error rates against the stateless path for a week.

The fallback logic looks like this:

agent step
   |
   v
continue thread  (ok) -> next step
   |
 (404 thread_not_found)
   v
rebuild full request from your transcript
   |
   v
create new thread -> next step

How MagicMakers Lab approaches this#

When we build agents through our AI Integration & Automation work, we treat model-side features like threads as optimizations layered on top of a system that already works without them. The transcript lives in your database, every model call is logged with its latency, and provider-specific features sit behind one interface. It's the approach that let Framico ship 200+ orders a day with zero manual steps: the automation doesn't depend on any single moving part behaving perfectly.

Key takeaways#

• Message threads cut median time to first token by about 35% for 500k to 1M token Claude requests, according to Anthropic.
• They don't lower token costs or rate limits; prompt caching is still the cost lever.
• Threads expire after 5 minutes idle, so a replay path and your own transcript are mandatory.
• They aren't ZDR or HIPAA eligible and don't work with Message Batches.
• Measure TTFT per agent step before deciding whether threads are worth it for your workload.

FAQ#

What is time to first token in an LLM?#

Time to first token is the delay between sending a request to a language model and receiving the first piece of its reply. It includes network transfer, queueing and the model reading your prompt. For chat interfaces and agents it shapes how responsive the product feels, often more than total generation time does.

Do Claude message threads reduce API costs?#

No. Anthropic says standard token rates and existing rate limits still apply, because Claude reads the full conversation on every turn even when you only send new messages. Threads reduce request size and waiting time. To cut cost, set up prompt caching breakpoints, which threads don't add for you.

Are Claude message threads HIPAA compliant?#

No. Anthropic's documentation lists message threads as not eligible for zero data retention or HIPAA, and organizations with HIPAA readiness enabled can't use the feature. Threads are held in memory, encrypted and never written to disk, but if your agent handles regulated health data, stay with standard stateless requests.

What happens when a Claude message thread expires?#

A thread expires 5 minutes after its last request. A continue request then returns a 404 with the error code thread_not_found, and your app must resend the full conversation in a new create request. That's why you need to keep your own copy of every message and build that replay path in from day one.

If your agents feel slow or your AI workflows keep needing a human to nudge them along, we can look at where the time actually goes and what's worth fixing first. Book a free audit.

Sources#

• Make long-running agents faster with message threads (Anthropic)
• Message threads documentation (Claude Platform Docs)
• Anthropic Launches Message Threads Beta for Long-Running Agents (Unite.AI)
• Anthropic's message threads let Claude agents skip resending chats (The Next Web)