Back to Blogs
#ai#typescript#testing#debugging

Retrieval Ran First. Was Its Result Actually Ready?

The trace starts with retrieval. Generation appears underneath it. The ordering test passes. But...

4 min read
Cover for Retrieval Ran First. Was Its Result Actually Ready?

The trace starts with retrieval. Generation appears underneath it. The ordering test passes.

But generation began while retrieval was still running.

“A appears before B” can describe the order in which events were encountered. “A's result was available before B began” describes an execution dependency. “B actually consumed A's result” is a stronger data-flow claim. They are not three ways of saying the same thing.

I maintain AgentInspect. This article tests those distinctions with controlled promises and agent-inspect@6.31.35. The fixture does not invoke a model, search engine, or vector database.

Make the race deterministic

Avoid a demonstration that depends on a slow network. Hold retrieval open until the test explicitly releases it:

import { inspectRun, step } from "agent-inspect";

await inspectRun("overlapping-retrieval", async () => {
  let release;
  const ready = new Promise(resolve => { release = resolve; });

  const lookup = step.tool("retrieve_policy", () => ready);

  await step("generate_answer", async () => "synthetic answer", {
    type: "llm",
  });

  release({ policyId: "fixture-policy" });
  await lookup;
}, { traceDir: ".agent-inspect", silent: true });
Enter fullscreen mode Exit fullscreen mode

Here type: "llm" labels an instrumented boundary. It does not turn the callback into a model call or capture hidden reasoning. The callback returns a fixed string.

Retrieval is invoked before generation, but the result cannot become available until after generation completes. The fixture encodes that relationship without guessing a millisecond delay.

Two timelines compare retrieval starting first with retrieval finishing before generation starts.

Start order is insufficient when the second operation depends on the first result.

Use a typed relation across tool and model steps

After opening the generated file with openTraceFile from agent-inspect/readers, evaluate the relation:

import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";

const contract = defineTraceContract({
  run: { requireCompleted: true },
  steps: {
    orderRelations: [{
      before: { kind: "TOOL", name: "retrieve_policy" },
      after: { kind: "LLM", name: "generate_answer" },
      mode: "happens-before",
      requireEndpoints: true,
    }],
  },
});

const result = evaluateTraceContractRead(trace, contract);
Enter fullscreen mode Exit fullscreen mode

Use the typed relation because this crosses a TOOL-to-LLM boundary. tools.requiredOrder is a tool-only surface, not a generic list of arbitrary workflow nodes.

TraceContract is beta, and steps.orderRelations is explicitly experimental in this release. Keep the version pin and test the exact semantics you intend to rely on.

The measured results for the overlap fixture are:

Mode Result Question being tested
first-occurrence Pass Was the first matching retrieval encountered first?
happens-before Fail Did the first matching retrieval finish before generation started?
all-occurrences Fail Did all matching retrieval work finish before any matching generation started?

Overlap can produce diagnostic information even when first-occurrence ordering passes. Read findings and warnings, not only the top-level label.

Stronger is not always more correct

all-occurrences is a global boundary, not an automatic pairing algorithm.

Consider a legitimate two-turn interaction: retrieve, generate, retrieve again, generate again. A rule requiring all retrieval to precede all generation can reject that workflow even if each individual answer waited for its own retrieval.

That is not a reason to weaken the rule until it passes. It is a reason to model the intended scope. Use separate runs or explicit application-level links that associate each answer with its retrieval result. Don’t assume that matching names imply pairwise dependency.

Similarly, requireEndpoints: true matters. An ordering statement should not quietly satisfy your product requirement when one of the necessary endpoints is missing.

Fix the application dependency first

The corresponding positive control awaits retrieval before invoking generation:

const policy = await step.tool("retrieve_policy", async () => ({
  policyId: "fixture-policy",
}));

await step("generate_answer", async () => policy.policyId, {
  type: "llm",
});
Enter fullscreen mode Exit fullscreen mode

Inside a captured run, this fixture passes the tested all-occurrences relation. The change is the dependency in the application, not a delay inserted into the test.

For a real model call, completion still does not establish consumption. Record a bounded result reference at retrieval and the reference actually supplied to generation. An application assertion can compare those references without storing the retrieved document or prompt.

Even matching references prove wiring, not factual grounding. Whether the answer uses the evidence correctly requires a separate domain or semantic check.

Keep incomplete execution visible

The current implementation has a narrower boundary worth documenting. In the companion fixture, one generate_answer starts early and never finishes; another matching generation completes later. The isolated all-occurrences ordering rule passes, while the full contract with completion checking fails through run.status.

This was reproduced against 6.31.35. It is not a claim that the ordering rule completely diagnoses every unfinished occurrence. Do not ship an ordering-only gate or describe that diagnostic gap as fixed.

The test deliberately creates an unawaited pending operation to expose the boundary. That is a negative control, not an application pattern to copy. Keep a valid fixture, an overlap fixture, a repeated-turn fixture, and an incomplete fixture in your contract tests.

Ask for the smallest defensible conclusion

A trace can establish that a recorded step preceded another. With adequate timestamps and boundaries, it can establish that one finished before the other began. With application-owned references, it can help establish which result was supplied downstream.

None of those facts alone proves that an answer is correct.

The practical improvement is to name the obligation precisely. If generation needs retrieval output, test result availability. If it must use a particular retrieval result, test the data-flow link. If the answer must be grounded, test the answer against that evidence.

One green ordering check should not have to impersonate all three.

Implementation references