At 2 a.m., a support assistant starts quoting 3-day refunds instead of 14. Nobody has retrained the model or touched the prompt. The on-call engineer cannot say who decides whether to roll it back. That question, more than any hiring plan, is what an AI product organization has to answer in advance.
The unit of work is no longer a feature with a known output. It is a model whose behavior is probabilistic, drifts as the world changes and depends on data someone has to source and defend.
Teams built to ship features struggle with that. Data pipelines, model behavior, safety checks and releases across several environments do not fit a backlog of tickets with predictable outcomes. Get the org design wrong and each of those pressures comes back later as a delivery bottleneck, an untracked dataset or a model nobody will sign off on.
Pick the structure by queue length and duplication
Most AI organizations settle into one of three shapes, and scale decides which one fits better than fashion does.
Early in a transformation, many firms start with a center of excellence: one team that concentrates scarce ML talent, sets standards, prototypes quickly and advises product teams. It works while demand is modest.
The failure has a clear symptom, which is the queue. Say the center has 9 people and ships about 3 production models a month, or 9 a quarter, while product teams file 14 requests a quarter (illustrative numbers).
Queue growth per quarter = requests filed minus models shipped = 14 minus 9 = 5.
The backlog grows by 5 requests every quarter; by the end of the year 20 requests are waiting, close to 7 months of work at the current pace. Business lines that cannot wait that long build shadow teams outside the center's standards, and now you have the duplication problem as well.
The common next step is platform teams sitting under application teams. The platform owns the shared tools every team needs: feature stores, embedding libraries, prompt orchestration, training pipelines, a catalogue of models, test suites for model quality and safety rules. The application teams own end-to-end product problems: the features built on those components, experimentation and rollout, the UX for AI behaviors and the business outcome. Draw the line between them by change frequency. Infrastructure that changes quarterly belongs to the platform; customer logic that changes weekly belongs to the app teams. A platform team exists so that the teams serving customers have less to build and less to think about. That is why it is judged by how many of those teams actually use what it builds.
The arithmetic for carving out a platform is simple enough to run on a whiteboard. If three product teams each spend 2 engineer-months building their own evaluation harness, that is 6 engineer-months of duplicated work. A shared harness might cost 3 engineer-months to build plus half a month per team to integrate, so 4.5 in total, and every later team gets it for half a month. With two teams the saving is close to zero (4 against 4). With three or more, the platform pays.
The platform pays when 2N > 3 + 0.5N, where N is the number of teams that would otherwise build their own harness. At N = 2 both sides equal 4; at N = 3 it is 6 against 4.5, and the gap widens with every team after that.
Large firms tend to land on a federated model: central platform teams own infrastructure and governance, data scientists sit inside domain teams, and squads work to shared processes. It scales while holding a common quality bar, on one condition. A single authority must still own the definitions of shared metrics and cost, or the autonomy fractures into teams that each measure success their own way. Melvin Conway's 1968 observation applies with force here: systems end up mirroring the communication structure of the organization that builds them, so three disconnected teams tend to produce three disconnected evaluation stacks.
A decision rule that covers most cases:
- Fewer than about 10 ML people and no shared standards yet: start with a center of excellence, and leave it the quarter its backlog grows faster than it ships.
- Three or more product teams rebuilding the same pipelines, evaluations or guardrails: pull that machinery into a platform team under the application teams.
- Many domains that need autonomy but the same safety bar: go federated, with one owner for metric and cost definitions.
Every way a model fails needs a named owner
Structure only answers the 2 a.m. question if each failure mode has a name next to it. Someone has to own the model lifecycle, data quality, evaluation, and governance. The useful way to design roles is to start from what breaks and work back to who carries it.
- Problem framed badly: offline metrics look fine while users ignore the feature. Owner: the AI product manager.
- Data cannot support the task: accuracy plateaus whichever model you try. Owner: the data scientist.
- Shipped badly or drifting: latency spikes, or quality drops weeks after launch. Owner: the ML engineer with MLOps.
- Untested failure: a customer finds the hallucination or the bias before you do. Owner: the evaluation and safety specialist.
- Misplaced user trust: users accept wrong answers without checking them. Owner: the designer.
- Untraceable data: nobody can say what the model was trained on. Owner: the data governance lead.
The AI product manager frames the problem and the hypothesis, checks whether the data to support it exists, and sets evaluation criteria and guardrails before any model is built. The hard part is timing. The PM writes the definition of "good enough" (say, 92% precision on refund-policy answers with under 1% stating a wrong deadline) before anyone knows whether a model can reach it, and then has to hold that line when the first offline numbers come in at 85%.
The data scientist answers what is learnable at all, through feature work, offline evaluation and the exploratory analysis that tells the PM early whether an idea is feasible.
The ML engineer makes it run in production: integration, inference performance, drift monitoring. Behind them the MLOps engineer owns the pipeline nobody sees, from observability to automated retraining and versioning.
Evaluation and safety specialists build the golden datasets a model is measured against: fixed sets of questions with known correct answers. They probe the model for made-up facts, bias and security holes, and they decide whether a release can go out.
Designers show users how sure the model is, for example with a confidence label, and decide where a person reviews or overrides an answer. The data governance lead answers where every piece of training and inference data came from and on what basis it may be used, and keeps the audit trail that lets any decision be reconstructed later.
The list also exposes a small-company trap. With 4 ML people you cannot staff six roles, so one person carries two or three lines of it. That is fine as long as the combination is written down; the dangerous version is the one where evaluation belongs to "everyone" and the release gate belongs to nobody.
Ownership also has to survive the night shift, which means writing the rollback rule before the incident. For the refund assistant from the opening, a workable rule reads: if the share of answers stating a wrong deadline passes 1% over any 500 conversations, the on-call engineer reverts to the previous model version without waiting for anyone, and the PM reviews in the morning. The threshold is the PM's to set and the switch is MLOps' to build.
Nobody has to be woken up to make a judgment call, because the judgment was made in daylight.
Evaluation runs every week
The path from question to production loops back on itself. A weak offline result in step 4 often sends the team back to step 2:
- PM defines the problem, hypotheses, and evaluation criteria.
- DS explores data feasibility.
- ML engineers build prototypes with DS.
- PM and DS evaluate offline results.
- Engineering integrates the model into the UX.
- PM runs online experiments.
- The designated risk owner approves each user-facing test and release against documented criteria.
- MLOps deploys, monitors, and retrains.
Because of that loop, evaluation is part of the normal working week from step 1. The PM reads model numbers the way a SaaS PM reads a funnel: precision (how many of the answers are right), recall (how many of the right cases it catches), response time, how often it makes things up, cost per call and drift.
This raises the competency floor for the whole product team. Nobody outside the specialists needs to train a model, but every product role needs enough grasp of how models hallucinate, how retrieval works and what an evaluation metric hides to make a decision without booking a meeting.
Data literacy as a baseline rather than a specialty is the practical form of that floor: reading a model-performance dashboard or a drift signal well enough to argue with it.
Here is the kind of call a PM makes alone. A new model variant lifts precision from 88% to 91%, but the slowest 5% of answers now take 2.6 seconds instead of 1.2. If the feature sits inside a live chat, the latency wins and the variant waits; if it drafts emails that a person sends later, ship it.
And because AI carries variable cost, PMs also have to hold strategy and unit economics together, modeling inference cost, scaling efficiency, and the accuracy-versus-cost trade-off before those numbers reach a bill.
Now put money on the same call. Variant B lifts precision by 3 points but costs 0.004 USD more per call. At 2 million calls a month that is 8,000 USD. Three points of precision means about 60,000 fewer wrong answers; if 5% of wrong answers turn into a support ticket that costs 4 USD to handle, B saves 3,000 tickets and 12,000 USD. B pays for itself, but only while the ticket rate holds. If wrong answers rarely escalate, the same 8,000 USD buys nothing a customer notices.
Governance has to run without a meeting
Inline governance only works if each checkpoint has something concrete to check against. The checkpoints sit at dataset sourcing, at training and documentation, at prompt-vulnerability testing, at the user-facing AI disclosure and at the human-in-the-loop step. A responsible AI framework supplies the criteria: fairness thresholds, explainability requirements and a risk score per use case that sets how much scrutiny it receives.
Teams that do not want to invent the categories can borrow them. The NIST AI Risk Management Framework, released on January 26, 2023, for voluntary use, organizes the work into four functions: Govern, Map, Measure and Manage. Its value for an org designer is that each function can be given an owner from the failure list above. The policy document then becomes a clear table of who does what and who signs off.
Risk tiers make the scrutiny proportional. One workable scheme: tier 1 (internal summaries, nothing customer-facing) ships on the product team's own sign-off and a 200-case golden set; tier 2 (customer-facing suggestions a person can ignore) adds a safety review and a 1,000-case set; tier 3 (anything that decides on money, credit or eligibility) adds the governance lead's approval and a documented human override.
Write the thresholds down and a team can predict its own release date. Leave them implicit and every launch turns into a negotiation.
Governance that has turned into theater has symptoms too. The median approval for a tier 2 release stretches past 10 working days, teams start splitting one feature into several "tier 1" pieces to avoid review, and the golden sets stop being updated because nobody owns them. Any one of those is a sign the checkpoints have drifted away from the risk they were meant to catch.
Governance does not end at launch either. Every model is registered, monitored for drift and retrained on a schedule set in advance. Retired models need a documented way out, because a deprecated endpoint that still answers 2% of traffic is a model nobody is monitoring. MLOps teams run these guardrails so they happen by default.
The case against a platform team
Small, fast teams have a fair complaint about everything above. A platform team, they argue, becomes a second queue: product teams wait on it, it builds abstractions nobody asked for, and the people closest to the customer lose control of their own stack.
With two product teams and six ML engineers, they are usually right. The whiteboard arithmetic earlier showed a near-zero saving at two teams, and the coordination cost is real.
The picture changes once duplication becomes visible. Three evaluation harnesses that score the same metric differently mean leadership cannot compare model quality across products, and a compliance question gets argued three times.
The way to keep the platform from becoming a queue is to measure it like a product: adoption by app teams, time for a new team to ship its first model on the shared stack, and the number of requests waiting on the platform. If that last number keeps growing for two quarters, the platform is too thick and should hand components back.
How people work together decides whether any structure works. The fastest teams bring data scientists in while the problem is still being defined, and the PM fixes the quality bar before a model is built. Product, data science, engineering and design work on it at the same time, without handing it from one to the next.
Their roadmaps start from what the system must be able to do: what the data pipelines must support, how the model should change, and which platform pieces the work depends on. Cost per call and response time decide what is realistic long before a feature is designed.
And their PMs communicate in ranges: the expected behavior band, the risk scenarios considered and the plan for the next iteration, so stakeholders do not swing between hype and disappointment.
Settle the topology before you start hiring
Agree the operating model before you scale headcount: who owns the platform, how governance runs, which metrics are defined centrally. A reorg after the fact costs far more than a hiring plan built to fit.
The sequence that holds up runs from measurement to structure. Standardize evaluation and cost definitions first, so every team measures with the same ruler; carve shared components out of whichever team is most overloaded; adjust reporting lines last.
Then track one number: the days from an approved hypothesis to its first online experiment, with the share of releases that shipped with a signed evaluation report as the guardrail. If the first number falls while the second holds above 95%, the structure is working. If speed improves only because evaluation reports stop being signed, you have bought velocity with exactly the risk this design was meant to remove.