Part 7 - The Pipeline That Feeds Your Agent Is Broken

RAG assumes you have clean, current, well-structured text to retrieve against. Most teams do not. The documents are stale, the formats are inconsistent, the same fact lives in three systems with three answers, and the index is built off whichever extract ran last Tuesday. The pipeline that feeds the retriever is where most RAG systems actually fail, and it has almost nothing to do with AI.

Data pipelines for AI sit at an awkward intersection. They are not glamorous enough to get engineering headcount, not AI enough to get ML headcount, and not "infra" enough for the platform team to own. So nobody owns them, and they grow the way unsupervised things grow: sprawling, fragile, and confidently wrong.

The model does not know your data is bad. It will produce a fluent, confident answer based on the document that was last updated fourteen months ago. The answer will look right. It will occasionally be wrong in ways that take days to trace back to the source.

TL;DR

A production AI data pipeline does four distinct jobs: extraction, normalization, enrichment, and indexing. Most teams build only indexing, so failures in the first three propagate silently into the index and surface later as "the model got it wrong."

• Separate the four stages. One script that reads files and pushes embeddings is stage four of four. The other three are where the data actually breaks.
• A dead-letter queue is not optional. Documents that fail extraction or normalization vanish without one, and nobody finds out for months.
• Freshness is tiered, not scheduled. Operational records need near-real-time updates, reference material can be weekly. One nightly batch guarantees silent staleness for the fast tier. Deletions need their own handling; append-only pipelines never remove anything.
• Lineage belongs at chunk level. Source document, version, pipeline run, and transformation steps. "The model said it" is not an audit trail.
• Monitor the pipeline separately from the agent. Semantic drift passes every stage successfully and only shows up as degraded answers.

The pipeline has four stages: extraction, normalization, enrichment, and indexing. Most teams build only the last one.

What are the four stages of an AI data pipeline?

Production data pipelines for AI systems do four distinct things, and the most common failure is collapsing them into one.

Extraction is getting data out of its source format into something the rest of the pipeline can work with. Source formats include PDF with embedded tables, scanned documents run through OCR that quietly drops columns, HTML pages where the content is 30% nav chrome, JSON APIs that return nulls where you expect strings, and legacy databases that encode status codes as integers only two people understand. Extraction looks simple because most demos use plain text files. Production data is rarely plain text files.

Normalization is getting that extracted content into a consistent shape. Different systems use different date formats, different field names for the same concept, different encodings, different standards for what "null" means. The email-processing agent you are building needs to match incoming contract references against your CRM. If the CRM stores account IDs as ACC-00123 and the contracts use 00123, that is a normalization failure, and it looks like retrieval failure when you debug it later.

Enrichment is adding context the source documents do not contain but the retriever needs. A document titled "Product Update Q3" has no implicit metadata; a retriever with no filter surface has to scan everything. Enrichment means tagging it: product line, document type, date range, owning team, access tier. Most teams skip this entirely and then wonder why their retrieval is imprecise.

Indexing is the part most teams actually build. They write a script that reads files, chunks them, embeds them, and pushes them into a vector database. This is step four of four, and doing only step four means the first three problems are silently propagated into the index.

# Shape of a four-stage pipeline
for doc in extract_from_sources(sources):        # stage 1: format, connectivity, nulls
    normalized = normalize(doc)                   # stage 2: schemas, encoding, IDs
    if normalized is None:
        dead_letter_queue.write(doc)              # extraction or norm failures land here
        continue
    enriched = enrich(normalized)                 # stage 3: metadata, tags, access tier
    chunks = chunk(enriched)                      # structural chunking from part 6
    index.upsert(chunks)                          # stage 4: embed, store
    lineage.record(doc.source, chunks)            # track what came from where

The dead-letter queue is not optional. Every pipeline has documents that fail extraction or normalization. Without a dead-letter queue, they disappear silently. Three months later you discover that the compliance documents in your CRM have never been indexed because the export script times out on documents over 5 MB, and nobody noticed because the other documents kept returning answers.

The chunking step above is a modeling decision, not preprocessing. That argument, along with hybrid retrieval and re-ranking, is the subject of why most production RAG is quietly wrong. This article is about what happens before any of it.

Why does one batch cadence produce stale answers?

Most data pipelines run once. They were built as a proof of concept, they successfully loaded the corpus, and then they were "done." The corpus is now stale. The index reflects the world as it was when the script last ran, which might be the initial load, might be a manual re-run six weeks ago, might be a nightly job that silently failed eleven days ago and sent nobody an alert.

Freshness matters differently by document type. The email-processing agent you are building needs to match against your current SLA terms, your active product SKUs, and your live CRM records. SLA terms update quarterly, product SKUs update weekly, CRM records update continuously. A single batch pipeline with a single cadence cannot serve all three correctly.

Different document types need different update strategies. Collapsing them into one batch run is the most common freshness error.

The correct design uses a tiered approach: near-real-time for operational records, daily for frequently-updated content, weekly for stable reference material. This is not new thinking; operational databases have had hot/warm/cold tier strategies for decades. AI pipelines re-discover this pattern the hard way after the first production incident where an agent quotes a price that changed yesterday.

Freshness is also a deletion problem. When a document is deleted from the source, does it get removed from the index? Most pipelines do append-only writes. Deleted documents stay in the index indefinitely. The agent answers questions using content that no longer exists in your authoritative system.

How do you trace a wrong answer back to its source?

When an agent gives a wrong answer, you need to know where the wrong information came from. Without lineage, that question has no clean answer. You know the context window, but you do not know which chunk came from which document, which version of that document, which pipeline run, which source.

Lineage tracking is not complicated but it is effortful, and it gets skipped because it has no impact on demo quality. Every chunk in the index should carry: the source document identifier, the pipeline run timestamp, the source system version or last-modified time, and the transformation steps applied. When something goes wrong, you trace the chunk backward to its origin.

Every chunk carries enough metadata to walk backward to the document, version, and pipeline run that produced it.

This connects directly to compliance requirements later in the series. Regulated industries are increasingly asking "show your work" questions about AI outputs. "The model said it" is not a defensible audit trail. "The model said it because it read chunk X from document Y, version Z, ingested on date D" is.

// Lineage record per chunk
{
  "chunk_id":        "c_abc123",
  "source_id":       "doc_8921",
  "source_system":   "confluence",
  "source_version":  "2025-11-02T14:33:00Z",
  "pipeline_run_id": "run_20251103_0200",
  "transforms":      ["html_extract", "normalize_unicode", "structural_chunk"],
  "access_tier":     "internal",
  "tags":            ["product", "pricing", "q4-2025"]
}

What is semantic drift and why does nobody catch it?

There is a class of pipeline failure that is almost impossible to detect from the outside: semantic drift. The source system changes its schema, or starts using a term in a new way, or begins appending boilerplate to every document. The pipeline keeps running. The extraction succeeds. The normalization succeeds. The enrichment succeeds. The index fills up. The answers shift. Nobody notices until a user escalates a bad answer, and by then hundreds of similar answers have gone out.

Semantic drift requires monitoring at the pipeline level, not just the model level. This means:

• Distribution checks on chunk length (sudden change means chunking behavior changed)
• Embedding drift detection (new clusters appearing in the index)
• Metadata coverage checks (percentage of chunks tagged vs. untagged)
• Freshness gauges per document type and source

Most teams monitor the agent. Few monitor the pipeline. The pipeline is quieter but at least as likely to be the source of degradation.

Most teams instrument the agent (right side). The pipeline (left side) degrades silently without its own instrumentation.

This is not a new observation in data engineering. Anyone who has run a data warehouse knows that pipelines need their own observability layer. AI teams re-learn this because the failure mode presents at the model layer, which is where AI teams know how to look.

What does a good AI data pipeline look like?

A production data pipeline for an AI system has these five properties. Most teams have at most two.

First: idempotent runs. Running the pipeline twice produces the same index as running it once. This is a basic operational property but surprisingly rare; most quick-build pipelines accumulate duplicates on re-run.

Second: isolated failure handling. One bad document does not fail the batch. Extraction failures, normalization failures, and enrichment failures are caught per document, logged with enough detail to debug, and routed to a dead-letter queue.

Third: lineage at every stage. Every chunk in the index can be traced to its source, its pipeline run, and its transformation history.

Fourth: freshness guarantees per tier. The pipeline runs on a schedule appropriate to the update cadence of each document type, not one schedule for everything.

Fifth: pipeline-level observability. Separate from agent observability. Metrics on extraction yield, normalization success rate, chunk count by source, index coverage, and freshness by tier. The pipeline operator can answer "is the pipeline healthy?" without querying the agent.

Most AI pipelines are built in an afternoon as a prerequisite to building the interesting part. They are fragile, append-only, monitored imperfectly if at all, and treated as solved once the index is populated. The index is never solved. It is a continuous system, and a continuous system needs continuous operations.

The model is not the bottleneck. The pipeline that feeds it might be. And unlike the model, the pipeline is entirely your responsibility.

Most teams building RAG will spend weeks tuning their retrieval architecture and never audit what is actually in the index. The documents may be stale, the normalization may be silently dropping fields, the freshness tier for live operational records may be a weekly batch. The agent will keep answering. The answers will keep drifting.

A data pipeline is not a one-time build. It is infrastructure. Treat it that way before you spend more time on embedding model comparisons. If you are earlier in the series, the same logic applies one level up: most agentic projects cannot defend their own ROI because the operational cost of infrastructure like this never makes it into the business case.

Coming Up in This Series

Next up: Tool design and MCP: the real skill behind "tool calling." Most teams treat tool design as an afterthought: give the agent a function, describe it loosely, let the model sort it out. That approach works in demos and fails in production. Tool contracts, error surfaces, MCP as a standard, and the judgment calls that separate tools that agents can actually use from tools that look right in the schema but break at runtime.

---

If this resonated and you're building production AI systems, follow along. The series covers the 21 things I think senior AI engineers and architects need to reason about: RAG pipelines, tool design, security, evaluation, cost, and the operational patterns that separate demos from systems you can actually run.