Why Half of Every AI-Drafted PRD Still Needs My Hands-On Work

More context, same result

Over the past several months, I’ve had LLMs draft a lot of PRDs. And not in the toss-a-prompt-in way. I built a full system for this (I wrote about the workflow in an earlier post):

• A product knowledge base maintained in Markdown, tracking the current state of the product, its module architecture, and major changes
• Our company’s PRD template defined as a skill, specifying the required sections and what belongs in each
• A defined workflow running from requirements intake and clarification to the first draft

From there, it’s rounds of manual revision until the draft is usable. After sign-off, the final version goes back into the knowledge base, and I have the model re-extract judgment criteria from it and fold them into new rules, so each finished PRD becomes the foundation for the next one.

In theory, this is a self-improving loop. Every PRD it writes should make it understand our product a little better.

But as the models got stronger, the knowledge base got thicker, and the rules and skills piled up, the result never changed:

1. The first draft looks fine. Everything seems to make sense.
2. Look closer, and there’s a long list of things you want to change.

In my experience, 40 to 60 percent of the content still needs manual rewriting before it’s ready for the requirements review meeting. That still saves real time, but it left me genuinely curious. Today’s LLMs, even the mid-tier ones, are already superhuman on paper. They know more than we do and remember better. So why, after all the context and rules and skills I keep feeding them, can’t they produce a PRD that needs only light edits before I take it into requirements review or development?

After chewing on this for a while and digging through the literature, I think the answer has two layers.

Layer 1: Tacit knowledge

No matter how diligently you feed context to an LLM, the context will never be complete.

The philosopher Michael Polanyi put it this way: “We know more than we can tell.”

When I write a PRD, beyond what’s in the documents, I’m also drawing on:

• Unspoken conventions our engineers just get, where telling them what to build is enough
• The habits we’ve formed working with other teams
• Our org structure, and how we collaborate with upstream and downstream teams
• A request someone mentioned in passing at the last meeting, which we might be able to satisfy along the way
• A recurring user issue a support teammate once brought up. It was never logged anywhere, but it came back to me while writing, and maybe it deserves a place in the doc

None of this information gets fully preserved. Some of it is shared understanding nobody thought to write down. Some of it is tacit knowledge dropped in passing, with no way of knowing at the time that it would matter later.

Maintaining that much context is close to impossible in practice, and even if you pulled it off, it would still decay as the context window grows longer.

One thing I run into constantly: no matter how many times I remind the model that “we decide what to build and what the end state looks like, and the engineering team owns the technical choices and details,” once the context fills up, it starts forgetting the instruction and writes a pile of technical architecture back into the PRD.

So “keep adding context” has a ceiling. What can’t go in is the part that matters most, and what does go in gradually leaks away the longer the task runs.

Layer 2: I know it when I see it

Even if AI someday gets an infinite context window and reaches superintelligence, there’s a more fundamental problem. Many judgment criteria simply don’t exist until I see a concrete version.

For any given requirement, at the moment of writing I only have a rough picture. I don’t actually know what the final PRD should look like. Only after someone (or an LLM) writes it out do I start making concrete calls: this section isn’t needed, that one should take a different form, the next one should be built another way.

This isn’t a personal failing. Behavioral economics has been studying it for forty years. The classic finding on preference construction (Slovic, The Construction of Preference) is that when you change how a question is asked, the same person’s preference ordering changes. Preferences often get constructed at the moment someone asks the question, rather than sitting there fully formed waiting to be read out.

More recent research watching experts evaluate LLM outputs observed the same thing and gave it a name: criteria drift. Evaluation criteria grow out of the act of grading outputs and can’t be fully written down in advance (Shankar et al. 2024, UIST; in the authors’ words, “users need criteria to grade outputs, but grading outputs helps users define criteria”). Machine learning has its own version of this: the generation-verification gap. For models and humans alike, recognizing quality is consistently easier than producing it in one shot (Cobbe et al. 2021, Lightman et al. 2023).

Much of that 40-to-60-percent revision rate has nothing to do with the LLM getting things wrong. I’m growing criteria on a concrete draft, criteria that didn’t exist before I started marking it up. No amount of upfront prompt-writing can skip that step, because the criteria only take shape in the act of revising.

So what do we do about it?

You can stop pouring effort into writing your standards into the prompt, building an ever thicker system, or feeding in ever more context. These things aren’t useless. The marginal return on piling more into them is just low, for the two reasons above:

1. Your context can never be complete.
2. Your criteria don’t fully exist until you see the output. You have criteria drift too.

Design the work around a human as the verifier instead. Get the PRD produced fast, then make cheap, frequent judgment calls on concrete output. Don’t force yourself to architect every rule upfront before a single word gets written. The ROI there just isn’t high.

The 40 to 60 percent I still rewrite isn’t the system failing. The half that still needs my hands-on work is just the job.