Case Study · WhyLabs

Making modelhealth visible.

WhyLabs was an AI observability platform out of Seattle. We built out the engineering team behind Observatory, the application where every statistic the platform collected became something a person could read: thousands of features across many models. Ultimately we helped WhyLabs raise $10M Series A, and supported their build up through the next phase of roadmap implementation.

Series A round raised: $10MSeries A round raised
Models monitored, two weeks from launch: 3×Models monitored, two weeks from launch
Enterprise verticals served: 3Enterprise verticals served

Engagement Overview

WhyLabs was an AI observability company in Seattle, founded by a group of Amazon machine learning alumni and spun out of the Allen Institute for AI. Their bet was that the hard part of machine learning is not building the model. It is knowing whether the one already in production is still right, six weeks after the person who trained it moved on to something else.

Two products answered that. whylogs, their open source logging library, sat inside a pipeline and wrote a statistical profile of every batch of data that passed through it, in Python, Java or Spark. Observatory was the platform those profiles were sent to: the place where a data scientist could compare them, watch a feature drift, see what it did to accuracy, and be told about it before a customer was.

We built out the engineering team behind Observatory. Four fullstack engineers worked inside WhyLabs’ own product organisation, on two-week sprints and in their review queue, and the frontend team they were part of went from two engineers to eight over the course of it. What that team owned was the application end to end: the React and TypeScript codebase, the custom visx charting that rendered whylogs profiles into something legible, the Python and FastAPI services behind it, and the component library and UI patterns that let the whole thing keep shipping as the product organisation grew around it.

$10M
Series A round raised: 3×
Models monitored, two weeks from launch: 3
Enterprise verticals served

Engagement: Embedded fullstack engineering team
Team: 4 Fullstack Engineers
Focus: Observatory web platform, data visualization, frontend architecture
Working model: 2-week sprints

Stack

Frontend

React
TypeScript
visx
D3

Backend

Python
FastAPI

Data & storage

PostgreSQL
Amazon S3

Infrastructure & DevOps

AWS
Docker
Kubernetes

AI & ML

whylogs
LangKit
OpenLLMTelemetry

Integrations

REST APIs
Webhooks
Slack
Datadog
PagerDuty

Preset monitors in WhyLabs Observatory: one-click drift and data quality monitors on a trailing seven-day baseline, next to a monitoring coverage callout and an accuracy chart with segmented performance below it

The Challenge

At kickoff the backend was the strong half. A pipeline was ingesting whylogs profiles at scale, and those profiles were never raw data but statistical sketches, which is what made the platform safe to point at a bank’s production traffic. What sat on top of it was thin. The frontend could not keep pace with the roadmap, and all of that hard-won detail reached people as static visualisations with no interactivity and no context. A drift metric with nothing to click is a number somebody has to take on faith.

The charting implementation was the specific bottleneck: slow and inflexible, which is a bad pair when the job is time-series across thousands of features and multiple models at once. The dense views mattered most and it handled them worst, so holding three profiles side by side, following a distance curve across a month, or opening the feature behind a spike all hit the same wall.

And the demand was not hypothetical. Enterprise customers in healthcare, financial services and e-commerce wanted richer dashboards, across predictive ML and generative AI both, from an audience that reads distributions for a living and spots a mis-binned histogram in about a second. With a Series A ahead of them, the frontend was also the half of the product every prospect and every investor would actually touch. It needed a complete overhaul, in technical capability and in team scale at once.

The Solution

A complete overhaul of the frontend, built on the premise that a statistical summary can be read as fluently as a record if somebody does the work of drawing it properly. Everything below is the same profile asked a different question: what is in it, what moved, what to watch for next time, what exactly changed, and by the end whether the thing being watched was a model at all or a language model. Five surfaces carry most of the weight.

The WhyLabs Observatory model dashboard: three dataset profiles compared across a feature table, each column showing its distribution and frequent items, with a distribution analysis panel open on the right

Dashboards at both altitudes, not one

A monitoring product is used by two people who want opposite things. Somebody responsible for a fleet needs to know whether anything is wrong anywhere, this morning, in one glance. Somebody debugging a specific model needs every column it saw and what each of them did. Build only for the first and the second gives up; build only for the second and the first never opens it. So we built both: the Project Dashboard and All Resources for operational health across the estate, an Overall Summary above them, and the single-resource view underneath with its profile comparison, its feature table and its per-column drilldown.

The single-resource view is where the profile machinery shows. Every column a model saw, its distribution drawn inline once per profile, its frequent items beside it, and up to three profiles held side by side so a Monday and a Thursday sat on the same row. None of that was plotted from data. Each of those distributions was a custom component drawing a whylogs sketch, which meant the binning had to be reconciled across profiles before anything could be compared, or three histograms of the same column would disagree about their own axis.

Project Dashboard
All Resources & Overall Summary
Profile Comparison
Feature-Level Debugging

The Anomaly Feed, and time-series you can actually use

Anomalies had been arriving as numbers in isolation. The Anomaly Feed made them a place: everything the analyzers found across a resource in one chronological list, filterable and drillable, so triage started from a list rather than a hunch. Behind it went the interactive time-series work, which is the part the old implementation could not carry. Drift distance per feature plotted across the window against a configurable threshold band. Frequent items stacked over the same days, which is what tells you which category grew while the distance was climbing. The current batch against a named reference profile, bar for bar.

visx is why those charts exist in that form. The requirement was responsive time-series across thousands of features and many resources, with detection, filtering and drill-down on the same surface, and no configurable chart library does that: the charts had to be components we owned. The hardest of the three is the baseline comparison, because a reference profile and a live batch rarely share a category set. Something appears, something else stops arriving, and a naive overlay silently drops both. Aligning them before rendering, and being explicit about what was missing on either side, is the difference between a chart a data scientist trusts and one they open twice and stop believing.

Anomaly Feed
Interactive Time-Series
visx Charting
Drift & Quality Drill-Down

One-click preset monitors in WhyLabs Observatory for drift and data quality, a monitoring coverage summary, an accuracy chart with prediction counts, and segmented performance by product category

The Monitor Manager: presets first, control underneath

The platform's promise was that monitoring started working the moment a profile arrived, and an interface either honours that or quietly undoes it. So the Monitor Manager opened on presets: drift on all discrete inputs, missing values, unique value outliers, each already scoped, each already pointed at a trailing seven-day baseline, each one click from live. Nobody was asked to configure a monitor before they knew what they wanted watched.

Under the presets sat the whole configuration surface, and it is the reason the presets could be that simple. Monitor type, the features and segments targeted, the analysis and its thresholds, the baseline it compared against, and the actions it fired when it tripped, all editable at whatever grain a team needed. Above both sat the number that made the rest legible: monitoring coverage, broken out across integration, drift, performance and data quality, so a team could see what was actually being watched rather than assume. And segmented performance put every category on one axis with its threshold violations marked, which is where an aggregate that looks fine turns out not to be.

Monitor Manager UI
Preset & Custom Monitors
Baselines & Thresholds
Alert Actions

Root cause without leaving the page

An anomaly is only useful if the next question can be answered where you are standing. So anomalies highlighted in real time inside the charts themselves, and hovering one gave its details and the contextual metadata around it rather than a link to somewhere else. Push further and the analysis panel opened in place: the full distribution for that column across every selected profile, a toggle between the frequent items view and the raw histogram data, and the counts underneath, so the picture and the numbers behind it were never more than a click apart. Legends were live, so switching a profile off redrew the comparison rather than dimming it.

This panel is also where the quietest result of the engagement shows. It is not a bespoke screen. It is the same chart primitives, the same table, the same empty and loading states as everywhere else, assembled differently. Establishing that component library and the patterns around it standardised the frontend architecture, and standardising it is precisely what let the team scale from two engineers to eight without consistency or velocity falling over. A panel opened from a drift chart and one opened from a segment behave identically because they cannot do otherwise, not because somebody remembered.

Real-Time Anomaly Highlighting
Hover Detail & Context
Reusable Component Library
Standardised UI Patterns

The WhyLabs platform covering large language models and generative AI alongside predictive models: monitoring, evaluation and guardrails across security and quality dimensions

Predictive models and generative AI, one ecosystem

The category moved underneath the product, and the interface had to move with it without splitting in two. A classifier and a large language model fail in almost unrelated ways: one drifts, the other is jailbroken. But the person answerable for both is the same person, and handing them two consoles would have meant handing them neither. So the same dashboard ecosystem grew to cover both, with data drift, data quality, model performance and security anomalies reading as four questions asked of one resource rather than four products.

Underneath, that meant carrying things a sketch of a feature table was never designed for: prompt and response metrics out of LangKit, traces out of OpenLLMTelemetry, and safety signals that are categorical where nearly everything else was numeric. The interface work was mostly restraint. New metric types render through the existing chart primitives, land in the existing anomaly feed, and alert through the existing integrations, so a team that already knew how to read a drift chart did not have to learn a second product to watch an LLM.

LLM Observability
Generative AI Monitoring
Security Anomalies
One Dashboard Ecosystem

And the rest of what Observatory held

The five screens above are what the product was demonstrated on. These are what made it something an organisation could actually run, and most of them exist because a monitoring tool nobody is notified by, or nobody outside one team can open, quietly stops being used.

Segments
Any categorical column could split a model into subgroups that were profiled, monitored and drilled into separately, which is how a fault affecting one region stops hiding inside a healthy aggregate.
Model-aware performance
The performance view rendered itself from the resource's own type: accuracy, ROC and a confusion matrix for a classifier, error metrics for a regression, ranking metrics for a ranker, with nothing asked of the user.
Notifications and integrations
Slack, Datadog and PagerDuty wired to the same analyzers the anomaly feed read, plus webhooks for everything else, because an observability platform that only speaks when you visit it is a dashboard.
Roles and access
Admin, member and viewer scoped per organisation, with access tokens managed alongside them, which is the difference between a tool one team runs and a platform a company adopts.
Integration quickstarts
Generated snippets for the environment a user actually had, Python, Spark, Kafka or a notebook, presented at the point where an empty project is the only thing standing between them and a first profile.
The service tier
Python and FastAPI behind the REST contract the client read through, PostgreSQL and S3 underneath, and Docker and Kubernetes on AWS holding the whole thing reproducible from a laptop to production.

Business Impact

The milestone this engagement is measured by is the round. In November 2021 WhyLabs raised a $10M Series A, co-led by Andrew Ng’s AI Fund and Defy Partners, with Madrona Venture Group and Bezos Expeditions following on, and Nortik helped get them there. A platform company at that stage is not funded on a deck. The thing being opened in those rooms was Observatory, and the team we built was the team that shipped it.

The product backed the story up. Teams in healthcare, financial services and e-commerce could monitor model health at scale and act on what they saw, because the visx work rendered complex drift distributions and time-series smoothly even on resources carrying hundreds of features. The free self-serve tier landed in the same stretch and tripled the number of models being monitored within two weeks of launch, with nobody walking those users through a demo.

What lasted was the pair of things that had to happen together: a complete dashboard ecosystem that turned complex monitoring from available into accessible, and the frontend organisation underneath it, which went from two engineers to eight without the codebase fragmenting because the component library went in before the people did. That is the part that outlasts a release, and it is why Sam Gracie describes the engagement in terms of the team rather than the tickets.