Most teams building retrieval augmented generation (RAG) systems for high stakes classification make the same architectural bet: Route every ambiguous case straight to the language model and trust the retrieved context to sort it out. This works fine in a demo. It falls apart the moment the system has to survive an audit, a regulator, or a compliance officer asking why a specific decision was made six months ago.
I have spent the last year building RAG based classification systems in regulated enterprise settings, where the cost of a wrong answer is not a bad chatbot reply. A decision has to hold up to scrutiny long after the model produced it. This environment forces a different design philosophy than most AI engineering content assumes.
Here is what changes when you cannot afford to be probabilistic about everything, and how a cascade architecture solves it.
The appeal of routing everything through a large language model (LLM) is obvious: Fewer moving parts, faster iteration, the model handles unanticipated edge cases. The problem shows up later, in three places.
First, auditability. “The model decided based on retrieved context” is not an acceptable answer. You need a decision path a human can reconstruct without rerunning inference and hoping for the same output.
Second, cost at scale. If your system processes tens of thousands of cases a day and every one hits an LLM call with several retrieved documents in context, your inference bill and latency both scale with volume in a way that rule based logic does not.
Third, and least discussed, model drift on the easy cases. LLMs are excellent at nuanced judgment calls. They are inconsistent, in ways that are hard to detect, on cases that should have a deterministic answer. A clear structured match against known criteria should never depend on a language model’s mood.
The fix: Stop treating the LLM as the front line and start treating it as the escalation path. In practice this means a three stage pipeline.
Stage one is deterministic. Exact matches, structured field comparisons, and anything with a clear rule get resolved here with no model call at all. This stage should clear the majority of volume, often more than half depending on your data quality, and every decision is fully explainable because it is a lookup, not an inference.
Stage two is where retrieval earns its keep. For cases that survive stage one — and I mean survive as in they were not clearly resolved — you build a retrieval layer that pulls the specific evidence relevant to the ambiguity: Prior reviewer decisions on similar cases, contextual documents that explain an apparent conflict, or historical precedent that clarifies an edge case. The retrieval step matters more than the generation step here. If you retrieve the wrong context, even the best language model in the world will produce a confident, well reasoned, wrong answer.
Stage three is the LLM call, and it should only see the residue that stages one and two could not resolve. This is the part people skip when they design their first version, and it is the single biggest lever for both cost and quality. In one system I worked on, routing only the genuinely ambiguous 10 to 15% of cases to the LLM cut inference cost by roughly 6X compared to an all LLM baseline, while improving consistency on the deterministic majority to effectively perfect.
Once a case reaches the LLM stage, most teams default to a neutral prompt: “Assess whether this case should be approved or flagged.” That framing is wrong for high stakes classification because the cost of the two error types is not symmetric. Missing something that genuinely needed attention can mean real harm downstream. Incorrectly flagging something that was fine costs a reviewer’s time and a delay. Those two outcomes are rarely equally bad, yet a neutral prompt asks the model to treat them as if they were.
An asymmetric risk prompt makes that tradeoff explicit to the model rather than letting it guess at your risk tolerance. Concretely, this means instructing the model to treat uncertainty as a reason to escalate rather than clear, providing calibrated examples of both error types with their consequences spelled out, and asking for a confidence score alongside the classification rather than a binary answer. The confidence score becomes your second cascade point: Anything below a certain threshold goes to a human reviewer instead of being auto resolved, no matter what the model’s classification says.
This sounds like a small prompt engineering detail. In practice it is the difference between a system that reduces reviewer workload and one that quietly increases risk while looking like it is working.
Standard RAG evaluation metrics were not built with this use case in mind, and using them without adaptation will give you a false sense of confidence. A few adjustments that matter.
Retrieval quality needs to be measured separately from final classification accuracy. A system can have excellent retrieval ranking scores and still make bad final decisions if the generation step misweights the evidence. Track them independently.
Your evaluation set needs deliberate oversampling of the cases that reach stage three, since that is where your system’s judgment actually gets tested. If your eval set mirrors your production distribution, it will be dominated by the deterministic cases your cascade already handles well, and you will be blind to exactly the failures that matter most.
LLM as judge evaluation works for this domain but only if the judge prompt encodes the same asymmetric risk framing as your production prompt. A judge that treats both error types equally will systematically favor the wrong tradeoff when you are tuning your system.
Finally, build a feedback loop from confirmed outcomes back into your retrieval corpus. When a human reviewer overturns a model decision, that case and its correct resolution should become retrievable context for future similar cases. Without this, your system’s handling of ambiguous cases never improves, it just keeps making the same category of mistake at the same rate.
The instinct to reach for the most capable model for every decision is understandable, but in domains where wrong answers have real consequences, the more valuable engineering work is deciding what should never touch the model at all. Cascade architecture is not a workaround for LLM limitations. It is what a mature RAG system looks like once you have actually had to defend its decisions to someone whose job is to find the flaw in your logic.
If you are building AI systems for any regulated or high stakes domain, the question worth asking before you write a single prompt is not “How do I get the model to handle this well.” It is “Which parts of this decision should never have been the model’s job in the first place.”
Vineet Vijay is a Lead AI and machine learning engineer.
There is a step in the development process for large language model (LLM)-assisted tooling that most teams skip because it’s tedious, time-consuming, and doesn’t produce results visible to end users: Verifying that what the model is saying is actually correct. Not fluent, not coherent, not topically relevant — correct in the sense of accurately identifying the right answer to the specific problem the tool was built to solve.
The gap between “this output sounds right to me” and “this output is verifiably correct” is where most LLM-assisted enterprise tools fail quietly. They pass internal review because the output sounds right. They fail in production because those people weren’t reviewing against ground truth — they were reviewing against their intuition about what a good answer looks like.
This distinction matters more as LLM-assisted tools move from productivity accessories to components that influence real business decisions. If your AI-assisted tool is shaping how an analyst investigates a data quality issue, how a compliance reviewer decides whether to escalate a flagged record, or how an operations team triages a validation failure — the accuracy of its output has real consequences. “Seems reasonable” is not an adequate evaluation standard for that.
The standard evaluation approach for LLM output in enterprise tooling is qualitative: A sample of outputs is reviewed by someone with domain knowledge, judged against a mental model of what a good answer looks like, and the prompt is adjusted if too many outputs seem off.
This catches a specific class of problems: Outputs that are obviously wrong, poorly formatted, or off-topic. These are real issues worth catching. They’re also the easy ones.
What qualitative evaluation consistently misses is the class of outputs that are wrong in ways that are difficult to see without checking against something external. An explanation that confidently identifies the wrong root cause, in language that sounds authoritative, based on reasoning that sounds plausible — this passes qualitative review. It fails the moment someone with the right context checks it against what actually happened.
In a system whose value proposition depends on accuracy, “sounds plausible” is not the same as “correct.” The two can diverge significantly, and qualitative review won’t tell you when they have.
The alternative is building an evaluation harness that scores model output against labeled ground truth — a set of cases where the correct answer is known, against which you can measure accuracy rather than coherence.
I built this while developing a root-cause explainer for data migration drift: A tool that takes a detected drift event and generates a ranked explanation of what most likely caused it. The first prototype produced fluent, specific-sounding explanations that passed qualitative review. When I tested it against cases where I already knew the root cause, the explanation was wrong often enough to matter.
The eval harness I built works in three parts.
First, a synthetic ground truth dataset: Cases where the correct answer is known by construction. This meant introducing specific, controlled causes into a test pipeline — schema changes, transformation logic bugs, source system behavioral shifts — recording exactly what I introduced, and running the model against the resulting drift events. The correct answer for each case was the cause I had deliberately introduced.
Getting the synthetic scenarios realistic enough to be useful required more care than I expected. Early versions were too clean — the drift signal was obvious in ways that real production drift events aren’t. Adding realistic noise, overlapping signals, and cases where multiple plausible causes were present simultaneously was what made the synthetic set actually predictive of real-world performance.
Second, a scoring function that evaluates ranked output. Binary correct/incorrect isn’t sufficient when the model produces a ranked list of likely causes rather than a single answer. An explanation that correctly identifies the root cause as the third most likely candidate is meaningfully different from one that identifies it as the most likely. The scoring function evaluated two dimensions: Presence — did the correct answer appear in the output at all — and rank — how prominently was it featured relative to incorrect candidates. These were combined into a weighted score that rewarded both finding the right answer and ranking it appropriately.
Third, systematic evaluation across the full synthetic dataset rather than spot-checking. Running the harness across the complete set reveals patterns that spot-checking misses: Which categories of problem the model handles reliably, which it consistently gets wrong, and which combinations of signals produce the highest rate of confident incorrect explanations.
The results were more informative than any qualitative review could have been.
Schema change scenarios scored well — the model was reliable at identifying upstream schema changes when the evidence was present and distinctive. Transformation logic bugs were harder — the model consistently identified the right general category but misattributed the specific change that caused the problem, particularly when multiple changes had been made close together. Overlapping-signal scenarios were the hardest — cases where two different causes occurred close in time produced the highest rate of confidently wrong explanations.
That last finding is the one that qualitative review would never have surfaced. The model’s expressed confidence didn’t correlate with its accuracy — it was most confident in the cases where it was most wrong. Without the eval harness measuring against ground truth, that pattern would have been invisible.
For teams deploying LLM-assisted tools in enterprise contexts — particularly tools that influence how people investigate problems, triage alerts, or make routing decisions — the eval harness question to answer before production deployment is: Have we measured accuracy against cases where we know the right answer, or have we only reviewed whether the outputs seem reasonable?
If the answer is the latter, the tool has been tested for fluency and coherence but not for correctness. Those are different properties. For tools that shape business decisions, correctness is the one that matters.
Building the synthetic ground truth dataset is the hard part and the part most worth investing in. It forces you to define precisely what “correct” means for your specific use case — which turns out to be a useful exercise independent of the evaluation itself. The scoring function and the harness infrastructure are relatively straightforward once you have that definition. Without it, you’re measuring something other than what you’re trying to guarantee.
Arun Mishra is an enterprise architect.
Content filters can block unsafe output. They cannot tell you whether an agent was authorized to issue that refund, touch that production system, or commit the company to an external action. Those are different problems, and most enterprises are only s…
If you have built anything with retrieval-augmented generation (RAG) in the last two years, you have lived its central frustration: You chop your documents into chunks, embed them, retrieve the top few that look similar to the question, and hand them t…
The enterprise technology ecosystem is caught in a costly cycle. Over the past two years, millions of dollars have been funneled into generative AI pilots, yet many of these initiatives stall out before ever reaching a live production environment.
When a project fails, the immediate instinct of technical leadership is often to blame the model: The context window was too restrictive, the latency was too high, or the reasoning capabilities simply were not there.
But as data engineers building the scaffolding for these systems, we often see a different reality: The model receives the blame, but the pipeline usually contains the root cause. Production gen AI rarely fails because of model limitations alone. More often, it fails because the enterprise data foundation underneath it is fundamentally unready.
This is what I call the ‘Cleanup Trap’: The false belief that an organization can pipe fragmented, inconsistent, and ungoverned legacy data into a large language model (LLM) orchestrator and simply “clean it up” or patch it at the retrieval layer.
In a standard retrieval-augmented generation (RAG) architecture, the retrieval layer is tasked with pulling relevant business context to ground the model’s responses. Because modern frameworks make it simple to stand up a vector database and a basic embedding pipeline, leadership often assumes that the data engineering problem is solved.
It is not.
When an embedding model receives raw, unvalidated data directly from operational silos, the resulting vector space inherits the structural noise, duplicate records, and conflicting states present in the source systems.
If the core data pipeline suffers from silent degradation — schema drift, missing fields, delayed change-data-capture (CDC) synchronization — that degradation cascades directly into the vector store. An AI model cannot accurately synthesize customer intelligence if the data pipeline behind it is serving stale, contradictory profiles across disparate storage layers.
No amount of prompt engineering, semantic reranking, or vector hyperparameter tuning can compensate for a broken ingestion pipeline. If the foundation is compromised, the downstream application will hallucinate, expose unauthorized context, or fail to deliver deterministic value.
To break out of the ‘Cleanup Trap,’ enterprise data teams must stop treating data quality as a post-processing step. They need to treat data readiness for AI with the same rigor they bring to traditional transaction processing.
This requires a deliberate architectural shift toward zero-trust data ingestion, structured validation frameworks, and automated anomaly detection before data ever reaches an AI orchestration layer.
Data quality checks cannot exist as a nightly batch afterthought. If an enterprise AI application relies on real-time data to assist users, validation must happen inline.
Teams should implement explicit schema validation checks at the earliest ingestion point, such as the streaming ingress layer or the bronze landing layer of a medallion architecture. If an upstream operational database mutates a schema without warning, the pipeline should quarantine anomalous payloads rather than allowing corrupted metadata to pollute downstream AI contexts.
Static row-count validation rules are insufficient for AI readiness. True data health requires a multi-tiered approach.
This means pairing structural verification — null checks, type conformance, and schema validation — with statistical profiling to monitor for data drift. Tracking metric deviations across feature distributions helps ensure that historical context remains stable over time.
If a pipeline suddenly processes an unexpected spike in empty string variables or structurally deviant fields, automated alerts should trigger an immediate pause before vector database updates continue.
An LLM should never be the arbiter of data access control. Trying to enforce row-level security or personal data filtering through system prompts is a compliance risk.
Security must be managed within the data infrastructure tier. Enterprise data foundations should enforce strict access controls, tokenization of sensitive identifiers, and rigorous lineage tracing before information is indexed into vector stores or passed into an agent’s context window.
For technology leaders mapping their infrastructure roadmaps, AI readiness requires evaluating data pipelines against a strict operational checklist.
Can you trace a flawed AI response back to the exact pipeline execution, source record, and transformation step that produced it?
Does your data lake architecture have a programmatic mechanism to segment and quarantine corrupted or non-compliant data before it reaches production feature stores?
Are your operational systems and AI-facing vector databases tightly synchronized, or are your agents making automated decisions based on outdated snapshots?
These questions matter because production AI is not just a model deployment problem. It is a data reliability problem.
The honeymoon phase of gen AI experimentation is ending. Enterprise leaders are demanding measurable, predictable, and secure business outcomes from their AI investments.
If an organization wants to transition from isolated, impressive-looking demos to resilient, production-grade AI systems, it must redirect its focus. Stop looking exclusively at the model tier.
The real competitive differentiator is not only the LLM an organization chooses. It is the engineering discipline, data governance, and pipeline resilience of the infrastructure built to feed it.
In the production era of AI, data engineering is no longer a backend function. It is the control plane for enterprise intelligence.
Naveen Ayalla is a senior data engineer.
DeepSeek’s recent decision to drastically cut pricing on its V4-Pro model by 75% should have been unequivocally good news for enterprise AI vendors and developers. Instead, many are discovering that cheaper models don’t automatically translate into healthier margins.
The reason is simple: While inference costs plummet, agent systems are voraciously consuming tokens faster than prices are declining. For the last 2 decades, software economics was dictated by the same rule. Infra became cheaper every year whereas applications became more capable. AI was initially hypothesized to follow the same pattern. As frontier models improved and token prices dropped, many assumed inference would become a negligible operating expense.That assumption has begun crumbling exponentially.
A chatbot usually turns one user question into one model call. An agent turns it into a chain of planning, retrieval, tool use, verification, summarization, and follow-up decisions. The user sees one answer. The vendor pays for the loop. That is the 100x problem: The same user-visible request can cost a lot more to serve as an agentic workflow than as a chatbot or retrieval-augmented generation (RAG) response. In longer-running workflows, the multiplier is higher. Falling model prices help, but they do not fix a product architecture that turns one prompt into dozens of billable operations.
The scale of what is now at stake is clear in how model providers themselves are pricing developer relationships. OpenAI’s proposed program to give every Y Combinator startup $2 million in API credits — a number that would have funded an entire seed round in any prior tech cycle, and when the same cohort got by on a few thousand dollars of AWS credits — is less a recruiting perk than an admission of what it now costs to run an AI-native company through its first year of product. For established enterprises retrofitting agents into existing product lines, the absolute numbers are larger still.
In a single-turn chatbot, one user message produces roughly one model call. Input-to-billed ratio is about 1:5.
In a multi-step agent rolled out across customer support, sales operations, finance, legal review, and engineering, that ratio routinely lands at 1:700 or higher. Every loop iteration carries forward the cumulative conversation, tool outputs, and reasoning traces. Each step appends; nothing is dropped.
A “simple” agent query like “What did our top customer ask about last week?” typically touches seven priced operations before returning an answer:
User prompt (~50 tokens)
System prompt and tool definitions (~3,000 tokens, repeated on every call)
Retrieval (~5,000 tokens of context)
Model call #1 — tool selection (8,000 in / 200 out)
Tool execution (~4,000 tokens returned)
Model call #2 — summarization (12,000 in / 400 out)
Model call #3 — follow-up decision (12,400 in / 100 out)
One sentence in, roughly 35,000 input tokens billed. Somewhere between $0.10 and $0.40 per query on a frontier model. Multiply that by a million queries a month — the table-stakes volume for any enterprise B2B feature — and the line item is six figures.
The dominant pricing story for enterprise AI has been seat-based SaaS: Pay per-user per-month, deliver agent capability, capture margin. That model assumes a reasonably bounded cost-per-user.
Token amplification breaks the assumption. A power user running 50 agent invocations a day on a $40/seat plan can cost more in inference than the plan charges. Token amplification shatters the traditional SaaS pricing model. When a power user’s daily agent activity costs more in inference than their monthly subscription fee, vendor gross margins turn negative, a paradox that compounds as customers deepen their agent adoption, the very usage curve vendors are selling to their boards. Several vendors are now privately reporting negative gross margins on heavy users, mirroring recent cloud expenditure reports from the Bessemer ‘Supernova’ cohort, where the correlation between AI-agent adoption and gross margin contraction has moved from a theoretical risk to a primary P&L headwind.
The visible symptoms have started leaking into public coverage. Bloomberg this week documented a widening gap between Salesforce’s Agentforce marketing demos and the capabilities actually shipping to customers. This is the kind of gap that opens predictably when promised functionality is technically possible but uneconomical to serve at the price the seat plan implies. Salesforce is the most-watched case, not a unique one.
“For my team, the cost of compute is far beyond the costs of the employees.” — Bryan Catanzaro, VP of Applied Deep Learning, Nvidia
The strategic implication is not “AI is expensive.” It is that the dominant business model assumed by most AI-native company plans does not survive contact with agentic workloads.
Consider an enterprise software vendor charging $40 per-user per-month for an AI-enabled support assistant. A traditional chatbot might cost only a few cents per user per day in inference, leaving healthy gross margins.
Now replace that chatbot with a fully agentic workflow capable of investigating tickets, querying internal systems, drafting responses, validating outputs, and escalating exceptions. If a heavy user executes 50 to 100 agent requests per day, inference consumption can increase by an order of magnitude. What was once a negligible infrastructure cost becomes a material operating expense.
This creates an unusual dynamic: The customers receiving the most value from the product are often the customers generating the highest inference costs. In extreme cases, vendors can find themselves with their most engaged users contributing the least profit. The result is a growing realization across enterprise software that agent adoption and margin expansion are no longer automatically aligned.
The technical responses are known and converging. They are not novel, but they are critical for survival
Cost-aware routing: This technique involves a small classifier model that decides which tier (Haiku, Sonnet, Opus equivalents) handles each query. Well-tuned routers cut inference bills by around 60% without any degradation in quality
Prompt caching: Anthropic, OpenAI, and Google now offer 75 to 90% discounts on cached prefixes.
Context discipline: You can truncate tool outputs, prune reasoning traces, and cap tool depth to prevent your agent from going down a rabbit hole
Speculative decoding: for self-hosted deployments, this technique guarantees 2 to 3X effective throughput on the same GPUs.
“Organizations using orchestration-led governance report stronger productivity gains — a holistic orchestration layer is associated with six times greater productivity impact than compliance‑only approaches” — IBM
The companies building this layer well are starting to look less like microservice operators and more like financial trading systems: Every routing decision priced, every path with its own P&L, every tenant on a metered budget.
Four moves separate the companies that will still have margin in 24 months from the ones that won’t:
Make inference cost a first-class metric. Track it per-feature, per-tenant, per-query class the same way cloud cost was tracked starting in the mid-2010s.
Budget like a media buyer. Set cost-per-thousand-queries ceilings per feature. Cap them. Alert on overruns. Engineering will not enforce this on its own.
Treat the router as core infrastructure, not an optimization. It is the new load balancer.
Audit prompts quarterly. A 4,000-token system prompt that grew organically over six months is a six-figure bill in slow motion. Most teams have never read their own production prompts end to end.
Negotiate volume commits early. Frontier-model vendors now offer reserved-instance-style prepaid commits at substantial discounts. List price is the worst price any enterprise will ever pay.
The structural shift underneath agentic AI is not that it is expensive. As DeepSeek’s price cut today underscores, frontier inference unit costs are dropping roughly 3X per year, and the curve is not slowing.
The shift is that amplification is outrunning the price cuts. Cutting per-token costs 75% does not help a company whose agents are doing 700X more tokens per user query than its pricing model assumed. For the first time since the cloud era began, architecture decisions are again financial decisions in real time. A prompt redesign is a margin event. A poorly bound agent loop is an outage with a credit card attached.
The companies that survive the next 24 months of AI infrastructure pricing will not be the ones running the cheapest model. They will be the ones whose agents are smart and know what they cost to think.
That is the 100X problem. And it is arriving faster than the price cuts can hide it.
Maitreyi Chatterjee is a senior software engineer at a big tech company.
Devansh Agarwal works as an ML engineer at a leading tech company.
Slopsquatting represents an emerging supply chain threat made possible by AI hallucinations. As developers increasingly rely on AI coding assistants, they unknowingly grant cybercriminals access to their software from day one.
Slopsquatting is a new type of supply chain attack that uses large language model (LLM) hallucinations to inject malicious code into development workflows. The term combines “AI slop” and “typosquatting,” a deceptive practice where attackers register misspelled or lookalike versions of popular domains to prey on users who enter URLs incorrectly.
This novel attack vector exploits LLMs’ tendency to generate fictitious software package names, which threat actors can then register and populate with malicious code.
During AI-assisted coding, the model may generate fake open-source packages — bundled collections of files, programs and installation tools. This alone is not necessarily harmful. However, if an attacker registers that fake package name, they can inject malware that gets incorporated directly into a developer’s codebase.
Traditionally, AI safety risks stem from hallucinations, which can adversely affect users who treat misinformation as valid. However, those same hallucinations have evolved into exploitable security vulnerabilities.
Typosquatting is a deceptive practice where a cybercriminal registers a mispelled version of a popular package to trick developers. It has existed for decades, so registries have built protections against it.
However, AI has changed the threat model. It recommends fictitious packages that sound plausible rather than making simple misspellings. Once attackers learn which hallucinated packages models tend to invent, they can register malware-filled packages under those names.
Since the hallucinated packages are not simply typoed versions of popular libraries, there are no protections against this practice at scale. For example, the registry protects against an attacker publishing “crossenv,” a squat of the popular “cross-env” package. However, it would not identify “mpn install cross-env file” or “cross-env-extended” as threats.
Even if many LLMs recommend the same hallucinated package, widespread compromise is still possible. Malicious packages could remain undetected in production for months or even years, allowing threat actors to passively inject malware across countless environments.
One research team analyzed 31,267 vulnerabilities belonging to 14,675 packages across 10 programming languages. They discovered that reported vulnerabilities are increasing at an annual rate of 98%, faster growth than the 25% annual increase in the number of open-source software packages. The team also observed an 85% increase in the average lifespan of vulnerabilities, indicating a decline in security.
Malicious actors can create open-access packages under the same name as commonly hallucinated libraries. Instead of standard code, they are filled with malware. The models believe they are referring to existing packages, so they often repeat the same hallucinated names. Since the hallucinations are not random, attackers could theoretically register packages that trick tens of thousands of developers.
These packages appear legitimate. String similarity to real libraries makes them recognizable. One-character typos suggest simple mistakes rather than malicious intent. Even fully fabricated names remain believable when the AI presents them in proper context. Detection is challenging, as developers trust their coding assistants to recommend valid dependencies.
LLMs generate the statistically most likely answer rather than prioritizing accuracy. Hallucinations are relatively common as a result. One study found hallucination rates range from 50% to 82%, depending on the model and prompting method. Even GPT-4o, the best-performing model, goes no lower than 23%, even with prompt-based mitigation.
Adversarial hallucination attacks could worsen this problem. Threat actors can leverage token-level manipulation or retrieval poisoning to force models to hallucinate in ways they want, increasing the likelihood that models recommend their malicious packages.
While all LLMs are prone to slopsquatting, some are more vulnerable than others. The likelihood of producing hallucinated packages during code generation depends on the model. Proprietary models are four times less likely to generate hallucinated packages than open-source models.
One research group proved this by conducting 30 tests across 30 different systems. Out of the 576,000 code samples and 2.23 million packages it produced, 19.7% were hallucinations. GPT-4.0 Turbo had a hallucination rate of 3.59%, while DeepSeek 1B, the best-performing open-source model, reached 13.63%.
This research suggests that organizations relying on open-source AI tools for code generation are roughly four times more exposed to slopsquatting attacks. That doesn’t necessarily mean proprietary tools will always remain safer, though. Once attackers realize this disparity, they may manipulate proprietary LLMs to take advantage of perceived safety.
Software developers who use AI tools estimate that over 40 percent of the code they commit includes AI assistance. They expect that percentage will increase considerably within the next few years. Already, 72% of those who have tried AI use it daily.
The uptick in vibe coding and AI-assisted coding amplifies the threat surface. As more developers integrate AI tools into their workflows without implementing proper verification processes, the attack surface for slopsquatting continues to expand.
For those using AI to assist with coding, double-checking output is essential. Verifying that recommended packages actually exist in official repositories before incorporating them into projects reduces risk.
Implementing automated checks that validate package names against known registries can help catch hallucinated packages before they enter production code. Security teams should also monitor for unusual package installations and maintain up-to-date threat intelligence on known slopsquatting campaigns.
Zac Amos is the Features Editor at ReHack.
There’s an important distinction between AI that just works today, and AI that lasts at scale. Many companies optimize hard for the first one without ever asking whether they’re building the second.
Velocity without discipline and strategic direction is a liability, not an asset. The hardest part of building AI at scale isn’t getting a model to work once. It’s building systems that continue to work, scale beyond individual teams and use cases, and improve consistently over time.
Today’s AI systems do more than just predict and optimize. They converse, reason, and increasingly take action. An autonomous system making decisions on a traveler’s behalf creates a very different set of expectations around reliability, governance, and accountability. As AI takes on more of those roles, the principles behind how these systems operate matter more than ever.
We have spent years applying AI and machine learning (ML) across the traveler journey — from personalization, ranking, and recommendations, to fraud prevention, customer support, and, more recently, generative and agentic AI experiences. That depth of experience is what led us to develop a set of ML and AI principles to guide how we build, deploy, and evolve AI systems across our company.
The goal is simple: Make sure the systems we build create real business value, scale, and operate safely. These principles define how we measure, design, govern, and operate our systems.
Publishing principles is the easy part. The harder and more important work is turning them into operating mechanisms: Recommendations, requirements, tooling, and release processes that teams actually use.
We have begun using ‘Agentic Release’ tollgates: A set of recommended and, in some cases, required checks before launching agentic AI features. These tollgates translate principles like clear ownership, risk-based governance, evaluation, safe rollout, and monitoring into concrete expectations for teams.
Some of these recommendations and requirements are already being automated and integrated into the software development lifecycle (SDLC). Over time, the goal is for these expectations to become embedded in how we design, evaluate, approve, launch, and monitor AI systems from the start.
The first test for any model is whether it improves a business outcome and, ultimately, the traveler experience — not whether it just improves a technical metric.
Align models to metrics with business impact: Every ML effort must tie directly to a key business outcome or traveler experience metric. Technical optimizations are useful midpoints, not end goals.
Optimize for return on cost: The value a model creates has to justify what it costs to develop, train, and monitor, plus the operational complexity it adds. Favor solutions that deliver lasting impact relative to what they cost to run.
Justify complexity against strong baselines: Complexity should be earned, not assumed. Start with a strong baseline: An existing general model, a simple heuristic, an off-the-shelf solution. Reach for specialized models or more complex architectures only when simpler options genuinely can’t meet the bar.
Require both offline and online evaluation: No model goes to broad deployment on offline validation alone or jumps straight to A/B testing. Every model must perform in both offline and online evaluations. Over time, our offline evaluations should reliably predict what we see online.
Getting a model to work is one challenge. Making its value extend beyond a single team or use case is the harder one.
Build on shared foundations; specialize only when justified: Favor shared, platform-wide foundations for core capabilities, data representations, and model building blocks. Specialization should build on those foundations, not spin up isolated stacks, so when the foundation improves, the gains flow across the organization.
Treat data as a first-class product: A model’s quality is bounded by the quality of its data. We need to maintain robust pipelines, clear lineage, reproducibility, and reusable features built with documented ownership, clear schemas, and SLAs that other teams can rely on.
Prioritize generality over local optimization: When two approaches perform similarly, favor the one whose learnings, assets, and operating patterns can be reused across teams, brands, and use cases. We should optimize not just for local performance, but for how quickly improvements can diffuse across the company and compound over time.
Minimize and sunset manual business rules: Manual rules are sometimes necessary for policy, safety, or compliance, but they should be explicit and reviewed regularly, never silent patches for weak models or a source of permanent maintenance debt.
Reproducibility and traceability by default: Training data, features, configurations, evaluation results, deployment versions, and key decisions should all be documented and recoverable. That’s what lets you debug a production issue months later and hand off ownership without losing institutional knowledge.
The bar for deploying AI isn’t just “does it work?” It’s “can we stand behind it?” Trust isn’t something you add at the end; it’s earned over time and maintained across the full lifecycle of every model we ship.
Assign clear ownership and accountability: Every model needs defined ownership across its lifecycle — a business owner, a product owner, an AI owner, and an operational owner. These don’t need to be four people, but the responsibilities must be explicit. Who’s accountable for outcomes? Who responds if the model drifts? Who answers the incident at 2 a.m.? Without this in place, models become orphaned and problems surface with no one to own them.
Adhere to standards and governance: AI and ML models must use approved platforms and comply with established company standards, release gates, and governance processes. Operating outside these guardrails requires a clear, defined path to remediation or deprecation, rather than an open-ended exception.
Govern proportionally to risk: The level of review, evaluation rigor, and human oversight should scale with a model’s impact. A customer-facing model that affects pricing or availability for millions of travelers demands a far higher bar than an internal tool used by a small team. For high-impact, safety-sensitive, or highly autonomous systems, human-in-the-loop checkpoints are built in from the start.
Design for fairness, privacy, and transparency: We actively test for unintended bias, have strong data guardrails, and favor explainability when decisions meaningfully affect users. These are incorporated from the start, not added on.
Design for safe rollout, rollback, and control: Deployments are progressive, with rollback paths, fallback mechanisms, and circuit breakers ready before launch. The ability to safely undo a deployment matters as much as the ability to ship it.
Monitor continuously and adapt: Once live, teams must actively monitor quality, drift, latency, cost, and business performance and retrain or recalibrate when the data shifts. A team should always be able to explain how its model is performing now, not just how it performed when it launched.
These principles do more than define how we build. They define what we’re willing to ship and how we stand behind it. In a world where AI systems are increasingly consequential and make real decisions for real travelers and partners, these standards matter. Applied consistently, they build responsible AI that lasts.
Xavi Amatriain is Chief AI and Data Officer at Expedia Group
Xavier will share more details about Expedia’s architecture during his session at VB Transform on July 14 at 11:10 am PT. He will discuss: “Expedia’s blueprint for building autonomous agents for high-stakes transactional systems.”
Interested in attending VB Transform 2026? Register here. A select number of complimentary passes are also available to senior technology leaders. Contact us to get yours.
Imagine if you could bring 250 people together in a massive room and have them discuss and debate an important issue, arguing the points and counterpoints, and converging on answers that accurately reflect their collective knowledge, wisdom, values, and sensibilities.
Now imagine that you convened this debate on America’s 250th birthday and asked 250 randomly selected Americans to come up with the top three innovations that America has contributed to the world over the last 250 years. What would they come up with?
I know – this all sounds impossible.
After all, you can’t get more than a dozen people to have a productive conversation on anything. At large scale, nobody would get enough airtime to express their views or respond to others. This is why typical business meetings or focus groups never have more than 8 to 10 people. Thoughtful real-time conversations just don’t scale.
To solve this, a new category of AI technology called “hyper-communication” is greatly expanding the size, scope, and efficiency of large-scale deliberations. It uses specialized AI agents to connect groups in real-time, allowing people to discuss and debate issues at any scale. The goal is to enable hundreds or even thousands of participant to hold thoughtful discussions where they can express their views and argue the merits of any issue.
I first wrote about this emerging technology in VentureBeat two years ago in an article about “Collective Superintelligence.” In that piece, I explain how large human groups can be hyper-connected by AI agents in ways that greatly amplify the group’s collective intelligence. You can check out the science behind hyper-communication in that prior VentureBeat piece. Here I am focusing on the debate among 250 Americans on America’s birthday.
To do this, I asked the team at Unanimous AI to field a randomly selected group of at least 250 Americans (with a broad distribution from every region in the country and diverse mix of political and social demographics) and invite them to a twenty-minute online debate inside a hyper-communication platform called Thinkscape that enables massively scalable discussion by text, voice, or video.
Once connected, we asked the group to come up with the top three contributions that America has made to the world over the last 250 years – not a survey of opinions, but deliberation of ideas, arguments, evidence, and reasoning. The group converged on a set of top answers that surprised me – but on reflection, they were sensible and well-reasoned.
Before getting into the answers, let me show you what the debate looks like behind the scenes. There were 277 people, each of them debating the issues with four or five other people in parallel discussion spaces. The magic is the swarm of AI agents that connect all the small groups together into a single real-time deliberation.This is what it looks like at high speed:
In the debate above, the group of 277 people came up with 94 different ideas and then narrowed it down to a top 10, then a top 3. In the gif above, we just plot the top ten ideas as they emerged and battle for support during the live conversational debate.
The most interesting part of a large debate like this is not the answers, but the reasons that emerge to justify the answers. Here is the group’s reasoning behind the “top three innovations” that America has given to the world over the last 250 years:
#1: The Internet: “Our collective perspective is that America’s greatest contribution to the world over the past 250 years is the internet. It was born exclusively in the U.S. through academic and government research and was scaled globally with profound impact. It transformed communication, democratized information and education, enabled commerce, medicine, research and cultural exchange, and amplified soft power and civic organizing. We also acknowledged significant harms (misinformation, addiction, privacy loss) and arguments that it’s recent, global, or not uniquely American.”
#2 Advances in medicine: “Our collective perspective is that the United States has saved and prolonged hundreds of millions of lives worldwide. American-developed vaccines have successfully eradicated or controlled once-deadly diseases, significantly extending life expectancy and enabling broader societal and technological progress. From major breakthroughs in cancer research and treatments to cutting-edge medical technologies that have revolutionized hospital safety and procedures, U.S. ingenuity has redefined healthcare. Ultimately, while the global diffusion of affordable medicines and vaccines has extended these benefits across borders, the U.S. remains a premier medical destination where people from around the world travel to receive the most advanced treatments.”
#3: Spreading democracy: “Our collective perspective is that one of America’s most significant global contributions is the nation’s system of governance. The US has long demonstrated democracy in practice as an enduring global model. The U.S. Constitution provided a vital blueprint for representative government, inspiring democratic movements and revolutions worldwide while actively promoting human rights and individual liberties internationally. By empowering citizens with the fundamental power to vote and choose their own leaders, this framework has served as a foundational framework for broader societal advances and directly helped establish thriving democracies around the world.”
It’s important to remember, this is 100% human intelligence — a pure reflection of the collective knowledge, wisdom, and values of 277 randomly selected Americans. That’s because the role of the AI agents in a hyper-communication system is to connect people, not replace them. The agents work to enable scalable human deliberation in which every participant is given optimized ability to express their views, respond to others, and converge on solutions based on their merits. The only question left is — what should we ask next?
Louis Rosenberg earned his PhD from Stanford University, was a professor at California State University (Cal Poly) and has been awarded over 300 patents for his work in human-computer interaction, AI, and collective intelligence.
In the past two years, businesses have been trying to fit large language models (LLMs) into support, analytics, development, and internal automation like never before.
Along with the increasing adoption of AI technology, another trend is gaining momentum — cybercriminals are taking advantage of the disconnect between assumptions about LLMs and their actual characteristics.
In 2025 and 2026, several independent sources have highlighted the same trend: Prompt injection remains one of the most impactful and widely demonstrated attack vectors against LLM systems. The OWASP LLM Top 10 (2025) lists prompt injection as LLM01, identifying it as the most critical category of LLM‑specific vulnerabilities, for the second consecutive edition. OWASP’s ranking reflects the fact that LLMs still struggle to reliably separate instructions from data, making them susceptible to manipulation through crafted inputs.
CrowdStrike’s 2026 Global Threat Report — built on frontline intelligence across more than 280 tracked adversaries — documented that threat actors injected malicious prompts into legitimate generative AI tools at more than 90 organizations in 2025. They then used those injections to generate commands that stole credentials and cryptocurrency. The report stated it plainly: “Prompts are the new malware.” AI-enabled adversaries increased their overall attack volume by 89% year-over-year, with prompt injection working as both an entry point and a force multiplier.
Real‑world incidents illustrate the operational impact. In August 2024, researchers at PromptArmor disclosed a prompt injection vulnerability in Slack AI that allowed an attacker to exfiltrate data from private Slack channels they had no access to — including API keys shared in private developer channels — by placing a malicious instruction in a public channel or embedding it in an uploaded document.
In June 2025, researchers at Aim Security disclosed EchoLeak (CVE-2025-32711, CVSS 9.3), the first documented zero-click prompt injection exploit against a production AI system, targeting Microsoft 365 Copilot. By sending a single crafted email, no user interaction required, an attacker could cause Copilot to access internal files and transmit their contents to an attacker-controlled server.
Both vulnerabilities were patched. These incidents underscore the fact that prompt injection is not a theoretical weakness but a practical, repeatable threat organizations must address as they deploy AI systems at scale.
Prompt injection techniques have undergone major evolutions over recent years, now targeting multi-agent architecture, retrieval-augmented generation (RAG) pipelines, model routers, and long-term memory capabilities.
Businesses deploy LLMs to process instructions, summarize information, and trigger automated workflows, but it is difficult for LLMs to tell:
Instructions from data
Information from context
Context from metadata
User intent from metadata
This creates an opportunity for attackers to manipulate and influence the model’s behavior, either directly or indirectly.
Cross-model prompt injection
LLM use is a common practice among enterprises. Attackers corrupt the output of a particular model, knowing well that other models would be processing the content. Hence, the corruption propagates through all AI systems.
RAG supply chain poisoning
Attackers create malicious information — documentation, blog articles, GitHub READMEs. Then they wait until this malicious information is ingested in enterprises’ RAG pipelines, then use it as an attack vector.
Agent hijacking
AI agents have evolved to the point where they can send emails, modify cloud infrastructure, execute code snippets, and interact with internal corporate systems. It takes just a single instruction to make agents act differently in a harmful manner.
Context overflow attacks
With the help of million-token context windows, attackers place malicious code within the document and hope that an LLM will stumble upon it and execute it, thus overriding all previous instructions.
Memory poisoning
Due to the implementation of long-term memory in LLMs, attackers can inject instructions that permanently reconfigure their state.
Model‑router manipulation
Enterprises increasingly use model routers to select between multiple LLMs. Attackers craft prompts that force routing to the weakest or least‑guarded model.
Prompt injection is not a theoretical problem. It directly affects:
Customer‑facing systems (chatbots, support agents)
Internal copilots (developer tools, security assistants)
Automation workflows (ticketing, cloud operations, HR processes)
Data governance (RAG pipelines, knowledge bases)
The risk is no longer limited to “the model said something it shouldn’t.”
In 2026, prompt injection can:
Trigger unauthorized actions
Leak sensitive data
Corrupt internal workflows
Manipulate analytics
Alter business logic
Compromise multi‑agent systems
The attack surface has expanded dramatically.
1. Constrain model permissions
Limit what the model can do, not just what it should do.
2. Segment untrusted content
Treat all external data — including RAG sources — as potentially hostile.
3. Monitor tool invocation
Require human approval for high‑impact actions.
4. Validate content provenance
Ensure RAG pipelines don’t ingest poisoned external content.
5. Harden model routers
Prevent attackers from forcing routing to weaker models.
6. Treat LLMs as untrusted components
This mindset shift is the foundation of modern AI security.
Prompt injection remains the most effective way to compromise enterprise AI systems because it exploits the fundamental way LLMs interpret text. Until organizations treat LLMs as untrusted interpreters — not autonomous decision‑makers — prompt injection will continue to dominate the AI threat landscape.
Julie Brunias is an AI Security Architect.