When a healthcare company comes to us wanting "AI in the product," the first version of the idea is almost always diagnostic. An engine that reads the scan, flags the risk, catches what the doctor missed. It's the most cinematic use case, and it's usually the wrong place to start.
Not because models can't contribute there. Because it's the hardest possible entry point: the highest regulatory bar, the longest validation cycle, the most skeptical audience, and the smallest tolerance for error. Meanwhile, the people you're building for are drowning in a completely different problem, and they will tell you about it if you ask.
The pajama-time problem
Ask a physician what they'd change about their job, and diagnosis rarely comes up. What comes up is the documentation. Time-and-motion studies have found physicians spending close to two hours on EHR and desk work for every hour of face-to-face patient time, and then finishing notes at home in the evening. Clinicians have a name for it: pajama time. It's cited in study after study as a leading driver of burnout, and burnout is cited in exit interview after exit interview.
This is the actual demand signal in healthcare AI. Not "help me be a better diagnostician," but "give me back the hours between seeing the patient and going home." The use cases that are working in production right now all attack some slice of that gap:
- Ambient documentation. The visit is recorded, transcribed, and drafted into a structured clinical note that the clinician edits and signs. This is the breakout use case of the last few years, and health systems have rolled it out to thousands of physicians because the value shows up in week one: notes closed the same day, evenings partially reclaimed.
- Inbox and message triage. Patient portal messages have exploded since the pandemic, and every one lands in a clinician's inbox. Models can classify, prioritize, and draft replies for the routine majority, medication refill logistics, normal-result follow-ups, scheduling questions, so the clinician reviews and sends instead of composing from scratch.
- Prior authorization and referral paperwork. The form asks for information that already exists in the chart. Extracting it, assembling the request, and attaching the supporting documentation is exactly the kind of structured drudgery LLMs handle well, with a human approving the final packet.
- Coding and charge capture. Suggesting billing codes from the encounter note, with the coder confirming. Missed charges are a quiet revenue leak in most systems, and a drafting model that surfaces them pays for itself quickly.
Notice the pattern: in every one of these, the model drafts and a human decides. That's not a limitation to be engineered away. It's why these use cases ship at all.
Why "the model drafts, the clinician signs" is the architecture
Healthcare punishes automation hubris more than any industry we work in. The reason the working use cases keep a clinician in the loop isn't timidity, it's three hard constraints that don't move:
- The regulatory line. Software that informs a clinical decision while leaving the clinician genuinely in charge sits in a very different regulatory category than software that makes the call. Keeping the human as the decision-maker is often the difference between shipping this year and entering a multi-year clearance process. Companies that understand this draw the product boundary deliberately, not accidentally.
- The liability line. When something goes wrong, "the AI said so" protects nobody. A workflow where the clinician reviews and signs preserves a clear accountability chain, which is what hospital legal teams look for before anything gets deployed.
- The trust line. Clinicians have watched decades of health-tech tools that added clicks and called it progress. A drafting tool earns trust incrementally: every draft is reviewable, every edit teaches you where the model falls short, and nobody is asked to bet a patient on it. A black-box recommendation asks for all the trust up front.
The practical implication for anyone building in this space: design the product so that the model's output is always inspectable and editable at the point where the human takes over, and instrument those edits. The edit rate per note section is the single most useful quality metric an ambient documentation product has. It tells you exactly where the model is weak, it builds your eval set from real clinical behavior, and it gives buyers a number they can watch improve.
What the buying organization needs, which is different
Clinicians adopt tools that save them time. Health systems buy tools that survive procurement. Both have to be true, and the second list is the one engineering teams underestimate:
- HIPAA is the floor, not the finish line. A BAA with your model provider, PHI-safe logging (traces are where PHI leaks hide, because everyone remembers to encrypt the database and forgets the observability pipeline), data residency answers, and a real deletion story.
- EHR integration or irrelevance. If the output doesn't land in Epic or Cerner as a note, an order, or an inbox draft, it doesn't exist. A separate tab is where clinical AI pilots go to die. Budget more integration time than model time; that ratio is normal here, not a sign you're doing it wrong.
- An audit trail by design. Who saw the draft, what the model produced, what the human changed, who signed. This is table stakes for the compliance review, and retrofitting it is far more expensive than building it in. It is the discipline we live with across Florence Healthcare's clinical trial platform, where every document has to stay inspection-ready.
- Evaluation evidence. Not benchmark scores, evidence on the buyer's own patient population and note types. The vendors winning health-system deals show up with a validation protocol, not a demo.
Where the ambition should actually go
None of this means healthcare AI should stay small. It means the ambitious version looks different from the diagnostic fantasy. The systems we find most interesting to build right now are agentic workflows over the administrative machinery of care: an agent that notices a referral has sat unactioned for a week and chases it, that assembles the prior-auth packet and tracks the payer's response, that reconciles the discharge summary against the medication list and flags the mismatch to a pharmacist.
That work is unglamorous and enormous. It's also where the two hours per clinician per day are buried, and every hour recovered is capacity the system desperately needs. The companies that win in this space won't be the ones with the most impressive model. They'll be the ones that understood what clinicians were actually asking for, built the boring integrations, kept the human signature on everything that matters, and measured relentlessly.
Start with the paperwork. The paperwork is the product.
