How should Datadog rethink its observability thesis for AI buyers?

What the AI buyer is actually purchasing

The reason Datadog's existing thesis strains under AI workloads is that the unit of failure changed. In a microservice estate, failure is binary and observable: the endpoint returned 500, the pod OOM-killed, the queue backed up. Every metric Datadog built its franchise on — latency percentiles, error rates, saturation, throughput — assumes the system either did the right thing or visibly failed to. AI systems break the assumption. An LLM call can return HTTP 200, in 340 milliseconds, with well-formed JSON, and still be wrong. The trace is green. The dashboard is green. The customer is furious. That gap between "the service responded" and "the response was correct" is the entire product opportunity, and it is not addressable with the pre-2024 metric vocabulary.

This changes who signs. The classic Datadog motion was developer-led bottoms-up adoption inside Platform Engineering and SRE, expanding into an enterprise platform agreement negotiated by a VP of Infrastructure. That buyer's success metrics were uptime against an SLO, mean time to resolution, alert volume per on-call rotation, and cost per host or per container. The emerging buyer sits somewhere between ML Platform Engineering, a Head of AI Engineering, and increasingly an AI Product Manager who owns a P&L line. Their success metrics are groundedness of generated answers, hallucination rate on a labeled eval set, cost per successfully completed task, end-user perceived latency for streamed responses, and whether legal will sign off on shipping the feature at all.

Those two buyers do not share a vocabulary, a budget line, or a definition of done. The SRE asks "is the service up?" The AI buyer asks "is the model right, is it safe, and is it inside budget?" Datadog can build every technically excellent LLM feature it wants, and still lose the account if the packaging, the demo, and the pricing page all speak SRE.

There is a RevOps dimension here that gets underweighted in the engineering discussion. When the buying center shifts, the entire revenue motion has to shift with it — territory design, quota carrying, discovery questions, the qualification framework, the demo environment, and the compensation plan for the AE who lands the account. A rep who has spent six years selling infrastructure monitoring to platform teams has a muscle memory that actively works against them in an AI observability conversation. They will open with agent installation footprint and host counts. The AI buyer will hear "this is a legacy tool" in the first four minutes and mentally file the vendor. Rethinking the product thesis without rethinking the go-to-market motion produces a good product that nobody in the right org ever sees.

The adjacent lesson is not unique to observability. The same buying-center migration has played out in data infrastructure, where the buyer moved from DBA to analytics engineer, and in security, where the buyer moved from network admin to application security. In every case, the incumbent with the larger platform eventually won the category — but only after building a dedicated product organization, a separate positioning surface, and a sales team that did not lead with the legacy story. The incumbents that lost were the ones that treated the new category as a feature checkbox on the existing SKU.

The pillars a credible AI observability product has to cover

Four capability areas define whether a platform is taken seriously by an AI engineering team. Missing any one of them turns the deal into a point-solution purchase alongside Datadog rather than instead of a competitor.

LLM call observability. The atomic record is the prompt-response pair, enriched with the model identifier and version, the system prompt template used, temperature and other sampling parameters, input and output token counts, cost derived from those counts, time-to-first-token and total generation latency, and some quality signal — a groundedness score, a retrieval-relevance score, or a downstream user thumbs-down. Datadog shipped LLM Observability in 2024, so the foundation exists; the question is depth relative to purpose-built tools.

Agent workflow monitoring. Multi-step agent systems — LangChain-style agents, OpenAI Assistants, tool-using Claude deployments, custom orchestration — require a fundamentally different trace shape. Each run branches, loops, and carries state forward. Instrumentation must capture step order, tool invocations with their arguments and returned payloads, the reasoning or plan text when the model exposes it, retry and escalation behavior, and cost attribution per step so a team can find the one sub-call burning 60% of the run's budget.

Cost attribution across providers. Real deployments are multi-provider by the second year: an OpenAI model for one workload, Anthropic for another, Google or Azure OpenAI where an enterprise agreement already exists, AWS Bedrock for procurement convenience, and self-hosted open-weight models on vLLM or TGI for the high-volume cheap path. Cost has to roll up across all of them, and slice down to model version, prompt template, tenant, feature, and user cohort.

Safety and compliance evidence. The EU AI Act entered into force in August 2024 with obligations phasing in over subsequent years, and US state-level AI legislation — Colorado's law being the most cited example — adds parallel requirements. High-risk system operators need retained logs, documented accuracy monitoring, bias testing across protected attributes, and PII handling proof. Most teams have none of this beyond an ad-hoc table of prompt logs.

Note what these four have in common: none of them is a host metric, and all of them are things a CFO or a General Counsel can be made to care about. That is the strategic point. Infrastructure observability is an engineering cost center purchase that gets squeezed in every budget review. AI observability, positioned correctly, is a spend-control product and a legal-risk product simultaneously, which means it can be sold to two budget holders who are not currently sending money to observability vendors at all.

How the instrumentation actually gets built

Teams that succeed at this do not start by buying a tool. They start by deciding what a "unit of work" means in their AI system, because everything downstream — cost attribution, quality scoring, compliance retention — hangs off that definition. A support-deflection assistant might define the unit as a resolved conversation. A document-extraction pipeline might define it as a processed document. A coding agent might define it as a merged pull request. The unit of work is the denominator in every metric that matters, and getting it wrong means producing dashboards that are technically accurate and operationally useless.

The practical sequence runs roughly like this. First, wrap the model client at the boundary — not inside application logic, where instrumentation rots, but at the SDK or gateway layer where every call must pass. Teams increasingly route through an internal LLM gateway precisely because it gives one chokepoint for logging, rate limiting, key rotation, and cost control. Second, emit a span per model call carrying the model name and version, the prompt template identifier (not just the rendered prompt — you want to group by template to compare versions), token counts in and out, and computed cost. Third, tag every span with business dimensions: tenant, feature area, user cohort, environment. Without these tags, cost data is a single undifferentiated number and no one can act on it.

Fourth, establish the parent-child relationship for agent runs so the tree structure survives into the trace viewer. Fifth, attach a quality signal, accepting that it will be imperfect. The three practical sources are automated scoring (a groundedness check against retrieved context, or an LLM-as-judge evaluation on a sample), explicit human feedback, and implicit behavioral signals like retry rate, escalation to a human agent, or session abandonment. The implicit signals are underrated — they cost nothing to collect and correlate surprisingly well with quality.

Sixth, sample deliberately. Full prompt-and-response retention on a high-volume system produces log volume that dwarfs the application logs, and observability pricing is volume-based. Most mature teams keep 100% of metadata (tokens, cost, latency, model, tags, scores) and sample payloads — often 1-10% for routine traffic, 100% for anything flagged as an error, a low quality score, or a regulated workload. That split is the difference between an affordable program and a bill that triggers an internal review.

The same instrumentation discipline shows up in adjacent domains and is worth borrowing from. Payment systems learned to log the authorization decision and its inputs separately from the transaction result. Recommendation systems learned to log the candidate set alongside the chosen item. Both exist because debugging a probabilistic system requires the inputs to the decision, not just the output, and both were retrofitted painfully after the fact.

What this costs and how long it takes

Budget conversations go badly when they assume AI observability cost scales like APM cost. It does not, because the payload sizes are orders of magnitude larger and the retention requirements are longer.

On raw data volume: a typical microservice trace span is a few hundred bytes. A single LLM span with a full prompt, retrieved context chunks, and a generated response can run 10-50 KB, and RAG applications with large context windows push higher. At a million calls a day with full payload retention, that is tens of gigabytes daily of high-cardinality searchable data. This is exactly why sampling policy is a first-class design decision rather than a tuning knob, and why any vendor's pricing model has to be modeled against your actual payload profile before signing. Ask for a bill simulation against a week of your real traffic — not a per-GB list price.

On the LLM spend itself, which is the number the AI buyer is actually trying to control: frontier-model API pricing is published per million tokens by every major provider and changes frequently enough that any figure quoted here would age badly — check the provider pricing pages directly. The structural facts that hold: input tokens cost meaningfully less than output tokens, prompt caching can cut input costs substantially on repeated-context workloads, smaller models in the same family are typically an order of magnitude cheaper than flagship models, and self-hosted open-weight inference trades API spend for GPU capacity and MLOps headcount. That last trade is where most cost analyses go wrong — the GPU bill is visible and the engineering time is not.

The economics that make observability an easy purchase: an unattributed AI budget routinely hides a 2-10x cost reduction available through mundane changes. Routing simple requests to a cheaper model, trimming a bloated system prompt that ships on every call, enabling prompt caching on a stable context prefix, deduplicating retrieval chunks, and fixing the retry loop that silently triples the cost of failed requests. None of these are exotic. All of them are invisible without per-template, per-tenant attribution. This is the ROI story, and it is far more persuasive than any dashboard demo.

On timeline: instrumenting a single production AI feature end-to-end is typically a one-to-three-week engineering effort if a gateway layer already exists, and longer if calls are scattered across services. Getting to trustworthy quality scoring takes longer — usually a quarter — because it requires building a labeled evaluation set, and building that set requires domain experts who have day jobs. Compliance evidence generation is the longest pole, often two quarters, because it involves legal defining what "adequate" means before engineering can build to it. Teams that sequence these in parallel rather than in series ship faster, but they need three distinct workstreams and three owners.

For a vendor rethinking its packaging, the timeline asymmetry matters. Cost attribution delivers value in week two and can be sold as a self-serve motion. Compliance delivers value in month six and must be sold as an enterprise motion with legal in the room. Putting both behind the same SKU and the same sales play means one of them is always mispriced.

Where teams and vendors get this wrong

The most common engineering mistake is instrumenting the model call and stopping there. A prompt-and-response log tells you what happened but not why, because in a retrieval-augmented system the response quality is usually determined upstream — by which documents the retriever returned, how they were chunked, and whether the reranker put the relevant passage inside the context window. Teams debug for weeks at the generation layer for problems that live in the retrieval layer. The instrumentation fix is to log retrieval as a first-class span with the query, the candidate set, the scores, and the final selected context.

The second mistake is treating agent traces as linear. A microservice trace is a directed path; an agent run is a decision tree with loops. Flattening it into a span list destroys the information the AI buyer came for — namely which branch the agent took and why. A tool that renders agent runs as flat span lists will demo fine and fail in real debugging, and engineers notice within a week.

The third mistake is quality scoring theater. Adding an LLM-as-judge scorer without validating it against human labels produces a metric that moves, gets dashboarded, gets alerted on, and correlates with nothing. Before trusting an automated scorer, sample 100-200 examples, have humans label them, and measure agreement. If agreement is poor, the scorer is noise and the alerts built on it are worse than no alerts, because they train the on-call to ignore the channel.

The fourth mistake is on the commercial side and it is the one most relevant to a vendor rethinking its thesis. Bolting AI features onto the existing platform SKU, selling them through the existing SRE-facing rep, and demoing them inside an infrastructure dashboard produces a predictable outcome: the AI team evaluates a purpose-built tool, likes it better because it was designed for their workflow, and buys it as a line item alongside the incumbent. The incumbent keeps the infrastructure spend and loses the growth budget. Multiple point solutions have raised substantial venture funding specifically to occupy this position — Arize, Fiddler, WhyLabs, Helicone, LangSmith, and Langfuse all compete here — and several of them have a multi-year head start on product built exclusively for this buyer.

The fifth mistake is underestimating the hyperscaler default. AWS Bedrock, Azure OpenAI, and Google Vertex AI all ship native observability for models running on their own platforms. That native tooling is usually shallower, but it is already enabled and already in the enterprise agreement. The credible counter-position is multi-provider neutrality: a team running three model providers and a self-hosted fleet cannot get one dashboard from any single hyperscaler. That is a real differentiator and it should be the lead message, not a footnote.

The sixth mistake — the quiet killer — is building for the demo instead of the audit. Compliance features that produce a pretty dashboard but cannot export a defensible evidence package fail at exactly the moment they are needed. The test is unglamorous: can a compliance officer, with no engineering help, produce a report covering a specified date range, showing every flagged output, the mitigation applied, and a tamper-evident record of the log's integrity? If that takes an engineer and a weekend, the feature does not exist.

Choosing an approach: build, buy, or consolidate

Most organizations face a three-way decision, and the right answer depends on scale, regulatory exposure, and how much of the estate already runs through one platform.

Reading the framework in practice: regulatory exposure dominates every other input, because it converts an optimization purchase into a gating purchase. If legal has flagged the workload, the evaluation criteria collapse to evidence generation and retention, and the vendor with the best dashboard loses to the vendor with the best export.

Absent regulation, spend materiality is the next fork. There is a rough threshold where dedicated cost attribution pays for itself immediately — when monthly model spend exceeds the fully loaded cost of the engineer who would otherwise be manually reconciling provider invoices against feature usage. Below that, provider-native billing dashboards plus a spreadsheet genuinely work, and buying tooling is premature.

The multi-provider question is where an incumbent platform's structural advantage shows up. A single-provider shop can plausibly live inside that provider's native tooling. The moment there are two providers plus self-hosted inference, no native tool gives a unified view, and the neutral platform wins by default. This is the wedge Datadog should be leaning on hardest, because it is the one thing the hyperscalers structurally cannot copy.

For the consolidation question — whether to push an incumbent for depth or go best-of-breed — the honest calculus includes integration cost that rarely makes the initial comparison. A separate AI observability tool means a second alerting surface, a second on-call integration, a second access-control model, and correlation work every time an incident spans the application and the model. Teams routinely underweight this and discover it during their first cross-boundary incident, when the application trace lives in one tool and the model trace lives in another and nothing joins them. A platform that can put the LLM span in the same trace as the database query it triggered has a durable advantage — but only if it actually does that, rather than putting AI data in a separate tab of the same product.

The RevOps read on all of this: the buying committee for AI observability is genuinely cross-functional in a way infrastructure monitoring never was. ML Platform owns the instrumentation, AI Engineering owns the quality bar, Finance owns the spend question, and Legal owns the compliance gate. A sales motion designed for a single technical champion will stall in a committee that has four champions with four different definitions of value. The vendors that win this category will be the ones that build four distinct value narratives and let the champion pick the one that unlocks their budget.

Related questions

Is LLM observability a separate product from APM or a feature of it?

Structurally it is a separate product with shared plumbing. The data model, the trace shape, the quality metrics, and the buyer all differ. The shared infrastructure — ingestion, storage, query, alerting — is what makes an incumbent platform's version cheaper to build than a startup's.

What should a team instrument first if resources are limited?

Token counts, cost, model version, and prompt template identifier on every call, tagged by tenant and feature. This is the cheapest instrumentation and it unlocks the highest-ROI action — cost attribution. Quality scoring and full payload capture can follow once the spend picture is clear.

How do you measure hallucination rate in production without human labelers?

You approximate it. Groundedness scoring against retrieved context catches unsupported claims in RAG systems. Implicit signals — retries, escalations, abandonment — correlate with poor answers. Both need validation against a human-labeled sample before you trust the number or alert on it.

Does routing everything through an internal LLM gateway create a bottleneck?

It creates a single point of instrumentation, which is the point. Latency overhead is typically small relative to generation time. The real risks are availability and key management, both solved with standard proxy patterns. Most teams at scale conclude the observability and cost control are worth it.

Which matters more to an AI buyer: cost visibility or safety monitoring?

Cost visibility sells faster because the pain is immediate and quantifiable. Safety monitoring sells bigger because it gates whether a feature ships at all in regulated contexts. A vendor should lead with cost to land and expand into compliance.

FAQ

What makes the AI buyer structurally different from the traditional observability buyer?

The traditional buyer optimizes for system availability and measures success in uptime, MTTR, and alert volume. The AI buyer optimizes for output correctness and measures success in answer quality, cost per completed task, and regulatory defensibility. They report through different orgs, hold different budgets, and evaluate tools against different demos. Selling the second buyer with the first buyer's motion is the most common failure mode.

Does an established observability platform already have most of what AI observability requires?

It has the hard parts of the infrastructure — high-volume ingestion, distributed tracing, correlation across telemetry types, alerting, and enterprise access controls. What it typically lacks is the AI-specific data model: prompt template versioning, agent decision trees, groundedness scoring, per-token cost attribution, and compliance evidence export. Those are meaningful product investments, not configuration changes, and they need a dedicated product team rather than a feature squad.

How much data does AI observability actually generate compared to standard APM?

Substantially more per event. A microservice span is measured in hundreds of bytes; an LLM span carrying prompt, retrieved context, and response can be tens of kilobytes. That ratio drives the entire cost and sampling conversation. Practical programs retain full metadata on every call and sample payloads heavily, keeping 100% of payloads only for errors, low-quality outputs, and regulated workloads.

What regulatory requirements are actually driving AI observability purchases?

The EU AI Act, which entered into force in August 2024 with obligations phasing in over subsequent years, imposes logging, accuracy, and documentation duties on high-risk systems. US state-level AI legislation adds parallel requirements. In practice, the trigger is an internal legal review that asks whether the organization can produce a defensible record of what a model output and what was done about it — and discovers that it cannot.

Can hyperscaler-native model monitoring replace a dedicated observability platform?

For single-provider deployments, often yes. Native tooling is already enabled, already covered by the existing agreement, and adequate for basic visibility. It stops working the moment the estate spans multiple providers or includes self-hosted inference, because no hyperscaler builds a good view of a competitor's models. Multi-provider neutrality is the structural argument for an independent platform.

Where does RevOps fit into an AI observability program?

Two places. Internally, RevOps owns the cost-per-outcome metric that connects model spend to revenue-generating activity — the denominator work that turns a token bill into a unit economics story. On the vendor side, RevOps has to redesign territories, qualification criteria, demo environments, and comp plans for a buying committee that now includes Finance and Legal alongside engineering. Product repositioning without go-to-market repositioning does not convert.

Sources

• Datadog LLM Observability: https://www.datadoghq.com/product/llm-observability/
• Arize AI: https://arize.com/
• Fiddler AI: https://www.fiddler.ai/
• WhyLabs: https://whylabs.ai/
• Helicone: https://www.helicone.ai/
• LangSmith: https://www.langchain.com/langsmith
• Langfuse: https://langfuse.com/
• EU AI Act overview: https://artificialintelligenceact.eu/
• OpenTelemetry: https://opentelemetry.io/
• Amazon Bedrock: https://aws.amazon.com/bedrock/

Related on PULSE

• [How should Salesloft rethink its sequencing thesis for AI buyers?](/knowledge/q1828)
• [How should Outreach rethink its sequencing thesis for AI buyers?](/knowledge/q1769)
• [How should ServiceNow rethink its workflow thesis for AI buyers?](/knowledge/q1649)
• [Should Datadog acquire Honeycomb to win observability?](/knowledge/q1716)
• [How does Datadog compete against AI-native observability tools?](/knowledge/q1675)
• [Will Datadog beat Splunk in observability by 2027?](/knowledge/q1670)