Your dbt descriptions are your prompt

One sentence in a dbt YAML file said that USD normalisation was a documented follow-up. It was wrong: the mart it described already had USD columns on every monetary measure. The sentence sat in the model’s description: field, nobody read it as anything but documentation, and for as long as that mart had been in its dictionary it was the first thing our analyst agent was told about that table.

The agent is a text-to-SQL analyst we built for a travel and fintech group. This post is about the pipe that carried that sentence from a YAML file to the agent’s prompt, why we think of the YAML as configuration rather than documentation, and what we changed after the sentence turned out to be wrong in three separate ways. The warehouse is Postgres, the transforms are dbt, and everything here runs on a staging environment.

The constraints came first

The agent answers finance and operations questions against a warehouse built from two source systems, and one person operates the whole stack. That ruled out several designs before we wrote any dbt.

Constraint	Consequence
One operator, no data-catalog team	Documentation cannot live in a second system that needs its own upkeep
An LLM writes the SQL	It needs a dictionary of tables and columns at question time, in plain text
Descriptions must ship with the code	Reviewable in git, deployed by the same pipeline as the models
Plain Postgres, no extra metadata service	The dictionary has to be readable from the catalog the agent already connects to

The last row decided it. Postgres already has a place for text attached to objects: the COMMENT on a table or column. dbt can write to it.

The pipe

dbt has a config called persist_docs. We set it once on the marts directory in dbt_project.yml:

    marts:
      +materialized: table
      +schema: analytics
      # Persist model/column descriptions into Postgres COMMENTs so the
      # AI analyst agent can read the data dictionary straight from the catalog.
      +persist_docs:
        relation: true
        columns: true

With that in place, every description: in the marts YAML becomes a COMMENT ON TABLE or COMMENT ON COLUMN statement when the mart is built. The text no longer lives only in a docs site nobody opens. It lives in the Postgres catalog, where obj_description() and col_description() return it to any client.

The agent’s schema loader is one such client. At the first question after a process starts, it runs one catalog query across the allowed schemas and joins each table and column to its comment:

@functools.lru_cache(maxsize=1)
def load_schema_context() -> str:
    """Build the data dictionary the model reasons over, from live Postgres COMMENTs
    on the marts (populated by dbt persist_docs), across ALL allowed schemas."""
    ...
        select n.nspname as schema_name,
               c.relname as table_name,
               obj_description(c.oid) as table_desc,
               a.attname as column_name,
               col_description(c.oid, a.attnum) as column_desc,
               format_type(a.atttypid, a.atttypmod) as data_type
        from pg_class c
        join pg_namespace n on n.oid = c.relnamespace
        join pg_attribute a on a.attrelid = c.oid and a.attnum > 0 and not a.attisdropped
        where n.nspname = ANY(%s) and c.relkind in ('r','v','m')
        order by n.nspname, c.relname, a.attnum

The result is a flat block of text, one TABLE line per relation and one line per column with its type and comment. It goes into the system prompt next to a few-shot library of verified example queries; together they are about 20,000 tokens (one trace measured 20,910 cached tokens). The agent can also call a describe_relation tool mid-loop, the same catalog read scoped to one table.

The whole path, including the two steps that are easy to forget, is here:

A line like this one is not a note about the system:

- name: amount_total_usd_signed
  description: >
    amount_total_usd signed - positive for invoices, negative for refunds.
    Sum this for net revenue.

It is an instruction to the analyst. When the agent picks the right column for a revenue question, it is partly because that YAML told it to. The instruction is versioned in git and deployed by dbt run. The same YAML also feeds the business users’ BI tool, so one description ships to two products.

The symptom

The mart in question is fct_product_orders: one row per order for the group’s main product line, the sell side of that business. Its table description read, in the part that matters:

Monetary measures without a suffix are in the company base currency;
`*_lcy` measures are in the transaction's local currency. USD normalization
is a documented follow-up (needs the company base currency confirmed).

An agent reading that is told three things, and none of them was true.

What the description said	What the table was
USD normalisation is a follow-up	Every `*_usd` measure existed. The USD columns had been in the mart since its first version
Unsuffixed measures are in the company base currency	They are in the order’s own currency, and in this tenant that was a single local currency on every row
`*_lcy` measures exist	This mart has none. They live on the travel marts

We do not have a transcript of a question that exposed it, so we will not invent one. What we know is what the agent’s context contained. The column-level text for gross_amount_usd was correct (“prefer this for revenue”), while the table-level text announced that such columns were still a future task. The dictionary contradicted itself, and the model had to pick which half to believe on every question that touched that line’s revenue.

The mart’s first version already had the *_usd columns and the sentence. A currency refactor later rewrote the FX logic and edited the same YAML file, and the sentence survived it. We corrected it six days after that refactor.

The wrong first guesses

Two reflexes cost us time.

First guess: the data is wrong, not the text. When a description and a table disagree, the instinct is to suspect the table. We had already done this once on the same mart. An audit flagged a suspected FX error: staging labelled the amount column as company-base currency, while the mart converted it using the order’s own currency id. If those two differed, revenue would be off by a factor of roughly 360. The check took minutes. Every order was in one currency, and the amount equalled its local-currency twin in 99.1% of rows. The conversion was right. Only the label was wrong. A mislabelled comment had looked like a financial bug.

The lesson was that text in this warehouse can be wrong independently of the data. We treated it as a one-off.

Second guess: it is one wrong sentence. We expected to fix the “documented follow-up” line. The commit that corrected the mart is titled “it was wrong in three ways, not one”, because every claim we checked against the table failed. The same trap also sat at column level. cost_usd was described as “0 where cost is not tracked in this module”. In fact it is 0 on every row, because that module does not record supplier cost at all. Read as an instruction, “0 where cost is not tracked” means sometimes missing, so a careful analyst, human or model, would reasonably report margin_usd as a margin for the rows where cost is present. There are no such rows. margin_usd equals revenue on this table, and margin_pct inherits the same defect.

That is a worse kind of stale. The first claim was false. This one was almost true, which invites a confident wrong answer.

Root cause

Three things combined. None is a bug in dbt or Postgres.

A description is prose with no tests. The same YAML carries unique, not_null and relationships tests that fail loudly. Nothing asserts that a sentence about currency is still true, so a change to the model and a change to its sentence are two independent edits, and only one is enforced.
The text recorded project state. “Follow-up” is a statement about a plan, and plans move.
We wrote for the wrong reader. We wrote it for a person who might open a docs page. Its real reader was a model that takes every sentence as true on every question.

The loader also degrades quietly: on a catalog error it returns a placeholder string, which the cache then keeps until the next restart, and the agent carries on. A stale description does not even trigger that. It looks like a healthy dictionary.

The fix

The text fix was the easy part. The table description now states what is true and warns about what is not. (Table, product, currency and supplier names are generalised in this excerpt.)

- name: fct_product_orders
  description: >
    Product orders - the sell side of the product business.
    Grain: one row per order. Unsuffixed monetary measures are in the ORDER's own
    currency (currency_id); the `*_usd` measures are normalized to USD in the SELLING
    COMPANY's own dated rate space, so prefer those for any reporting.
    CAUTION - `cost_usd` is 0 on every row, because this module does not carry supplier
    cost. `margin_usd` therefore equals revenue here and is NOT a margin. For true
    margin use `fct_product_order_economics`, which joins the supplier cost.

The column descriptions changed with it. cost_usd now says it is always 0, “not sometimes missing”, and names the table with real cost. margin_usd says it equals revenue and is not a margin. margin_pct says it is unreliable here and points to the economics mart.

Two choices in the rewrite were deliberate. It states the wrong inference outright (ALWAYS 0, NOT a margin) instead of hinting at it. And it names the table that does hold true margin, so a warning gives the agent a route as well as a stop.

The deploy chain mattered more than the text, and it is where a fix can quietly do nothing.

Editing YAML does not edit the agent

Correcting the description changed a file in git. It reached the agent only after a chain of steps, and two are not obvious.

Step	What happens	The trap
Push the YAML change	Git has the new text	Nothing downstream has changed yet
CD builds and deploys a new image	The orchestrator image carries the dbt project	The orchestrator runs the project baked into its image. Triggering the transform job before this step rewrites the old comments
Trigger the marts transform	`persist_docs` rewrites the COMMENTs	Only now does the catalog hold the new text
Restart the agent backend	The cached dictionary is rebuilt	The loader is cached for process life. Without a restart the agent keeps the old dictionary
Refresh the BI tool	It pulls the YAML straight from the repo	The exception: no image and no restart

We learned the cache half earlier. A new weekly mart shipped, and the agent confidently wrote SQL against a table name it had invented, because its dictionary predated the real one. The table, the grants and the docs existed. The prompt was stale. The cache is the right trade at a change rate of weeks, and it makes every re-documented mart a deploy step. The runbook now ends any dbt deployment that adds or re-documents marts with a restart of the agent backend.

A second trap sits in the same chain. The infrastructure config pins a stale orchestrator image tag, so a plain apply would roll the orchestrator back to an image carrying the old descriptions. The deploy goes through the pipeline repo’s own target, with the live tag read first. A description is configuration, and it inherits the deploy hazards of whatever carries it.

What we took from it

Four habits, three from this incident and one we already had. They are our practice, not an enforced standard.

Write facts about the table, not project status. “Follow-up” and “for now” go stale. If a limitation matters to a query, say what is true today and what to use instead.
When a measure is degenerate, say so in absolute terms. “Zero on every row” leaves no room for “sometimes”.
Name the table that answers the question the agent is likely to ask.
Put any approximation in both the SQL comment and the model description, so the agent is told as well as the next reader of the SQL. The FX rate space is stated in the mart descriptions.

The same refactor corrected the SQL header comment and left the YAML sentence beside it, so the review rule is that a description changes in the same pull request as the model it describes.

We queued an audit of the other mart descriptions, and it was still open when we last checked. The marts carry several hundred column descriptions, and fct_product_orders is the one we know about because it was found, not because we looked.

The check that exists, and the one that does not

The repository has a CI job that fails the pull request on the BI metric layer. It exists because two silent bugs there passed every other check: a metric named like a column quietly vanished, and a ratio computed with integer division rendered a confident 0.0%. The static checks need no network. They verify that metric YAML is consistent with itself. They do not verify that a sentence agrees with a table.

A related catch: a metric referencing an undocumented column failed to compile, because the BI tool only exposes columns listed in the YAML. A missing description once broke metric compilation. A wrong one never has.

What a description check could look like, none of it built:

Idea	What it would catch
Flag words like “follow-up” and “TODO”	Plan state leaking into prose
Assert every column a description names exists in the model	The `*_lcy` measures this mart never had
A data test per claim, for example `cost_usd` is always 0	The “almost true” description, and it fails the day the claim stops being true
Re-run a fixed set of verified questions after any re-documentation	The end-to-end effect on the agent (needs a scored evaluation run; the harness exists, no real run has been made)

The third row is the useful one. “cost_usd is zero on every row” is a testable assertion. If the supplier-cost module ever records cost, the test fails and forces a rewrite. A description that a test can check can expire loudly.

Where it fails

Nothing verifies prose. Every control above is a convention or a deploy step. We found fct_product_orders by reading it. The next stale description will be found the same way, or by a wrong answer.
We cannot show a wrong answer caused by this text. We know what the prompt contained and that it contradicted itself. We have no measurement of how often that changed an answer, because we have not run a scored evaluation against it. The claim is about what the agent was told, not an error rate.
The cache makes the fix lag the repository. Between a merged correction and an agent restart the two disagree. A forgotten restart is the failure that results.
The loader fails quietly. On a catalog error it substitutes a placeholder and continues, and nothing alerts on it.
The dictionary is long. Roughly 20,000 tokens of cached prefix is cheap, but every description competes with the rest. We have not measured whether emphatic wording helps.
Two products read one text. The same sentence is shown to business users in a BI tool, and text written to steer a model can read oddly to them.
Descriptions only steer. The agent’s real guarantees are the read-only database role and the SQL guard. A description can say margin_usd is not a margin. It cannot stop a query that uses it.
It is one deployment. One warehouse on a staging environment, one operator. We do not know how this holds with many authors.

What we would tell a team doing the same

Treat every description: that reaches a catalog an agent reads as a line of the prompt. Review it with the model it describes. Keep project status out of it. When a column is degenerate, say so in absolute terms and name the table with the real answer. Know your deploy chain end to end, including any image that bakes in the dbt project and any cache that holds the dictionary. And look for the test that turns a claim in a sentence into an assertion about the data, because only that version of a description can tell you when it has gone out of date.

CloudGuys builds systems like this for startups and teams. Tell us what you are building.