Ask a large language model why your subscription renewals declined more often last quarter, and it will give you an answer that cites numbers, sounds confident and probably looks reasonable. Whether it is right depends on something the model cannot see: how messy your payments data was before you handed it over.

Andy Barker and Dr. Robert Clewley, both Senior Product Leads at Pagos, took on that gap in a recent PaymentsEd webinar. Andy brings roughly 30 years of merchant-side payments experience, and Robert comes to payments from academic mathematics and the biosciences. Their argument is that making payments data for AI trustworthy is still mostly a data problem, and that most merchants are further from solving it than the hype suggests.

Even One PSP Produces Messy Data

Consider retry optimization for subscription renewals. The first payment is a customer-initiated transaction (CIT), where the cardholder clicks buy; every renewal after that is a merchant-initiated transaction (MIT), where the merchant charges the stored card on the customer’s behalf. When an MIT declines, the merchant has to decide whether to retry, when and how often, and that decision is only as good as the data behind it.

With a single payment service provider (PSP), that data already lives in several places. Decline codes arrive in real time, while scheme fees, descriptors and processor fees often arrive later in separate settlement and fee reports, and disputes follow their own workflow. To calculate an accurate renewal authorization rate, a team has to reconstruct each order’s timeline, deduplicate retries so one order with three attempts doesn’t count as three failures, and adjust for time zones. (See Finding Your True Acceptance Rate for more on getting that number right.)

The reference data underneath also keeps moving. BIN ranges (the leading digits of a card number that identify the issuer and card product) change weekly, product codes update monthly, and the networks revise interchange rates twice a year. A model working from last quarter’s BIN table will draw conclusions about issuers and card types that are no longer true, and it will not warn you.

The Multi-PSP Multiplier

Most growing merchants eventually add a second PSP for failover, routing or geographic expansion. Teams often assume this doubles the work, but it does more than that, because a second PSP almost never structures its data the same way as the first. The merchant now needs a stable customer and order ID across every provider, plus decline code mappings for each PSP that stay consistent with each other. The gotchas rarely appear in the documentation:

  • One PSP reports in Eastern Time and another in UTC, so “approval rate on Tuesday” covers two different windows.
  • One reports amounts in minor units (cents with no decimal point), while another reports whole values.
  • One provides only six-digit BINs, which can lead an AI agent to false assumptions now that the industry has moved to eight-digit BINs.
  • One places the merchant’s order ID in metadata, and another replaces it with its own internal ID.
  • Some omit issuing bank, country or card type, so the merchant needs an accurate BIN lookup of its own.

Processors share no common vocabulary, so raw data from two or more PSPs is not comparable until someone makes it comparable. Volume compounds the issue: a merchant processing millions of transactions a month cannot paste raw reports into a model’s context window and expect it to reconcile them reliably.

Normalized decline codes for US Visa debit MITs, by issuer. Source: Pagos.

Normalization pays off once it is done. When Pagos normalizes decline codes across PSPs and merchants, patterns appear that no single PSP report would show. For US Visa debit MITs, insufficient funds accounts for most declines at Chase and Bank of America, while Wells Fargo returns a large share of “invalid account number” codes and Bancorp Bank leans on “do not honor” and “transaction not allowed.” Retry logic tuned to one issuer’s codes may not fit another’s, and even this chart raises questions a good analyst would test, such as whether some PSPs are overrepresented or whether tokenized and non-tokenized transactions were treated consistently.

Why AI Payments Projects Fail Quietly

Adoption numbers reflect the skepticism. According to industry survey data cited in the webinar, only 8% of businesses actively use AI for payments today, 30% of respondents have not identified a payment task they would trust AI to handle, and another 30% want a human in the loop. Most teams that do use AI rely on tools embedded in their PSP, which stop being useful once a merchant adds a second provider.

The root cause predates generative AI. A 2021 Google Research study, “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI, found that 92% of the AI practitioners it interviewed had experienced at least one data cascade, where a small data problem early in a project compounds into a large, expensive failure downstream. Nearly half had experienced two or more.

LLMs add a specific risk to this pattern. At best, a model working on messy payments data flags a suspicious subset for review; at worst, it misreads the data, trusts its own work and produces results that look perfectly reasonable. Outdated specs, fuzzy data such as BIN mappings and issuer names, and merchant ID (MID) level fee reports that cannot be reconciled one-to-one with transaction-level reports all make silent errors more likely.

A model that is 99% accurate sounds impressive until you ask which 99%. At payments scale, with millions of transactions a month, a 1% error rate shows up constantly, and every failure that customers or finance teams see erodes trust in the next answer.

Build From the Bottom of the Pyramid

Many teams picture analytics maturity as a staircase running from descriptive (what happened) to diagnostic, predictive and finally prescriptive (what to do). That picture encourages the belief that a team falling behind can skip straight to the top. The data, information, knowledge and wisdom (DIKW) pyramid is a more useful model: raw inputs from PSPs, networks, billing, fraud tools, customer data platforms and recovery services enter at the bottom, business strategy sits at the top, and each layer depends on the one beneath it. AI maturity in payments follows the same pyramid, which makes data enablement the real bottleneck.

Each layer of the DIKW pyramid needs its own kind of structure. Source: Pagos.

Each layer calls for its own discipline. Data engineering builds the ontology, a consistent model of what a transaction, order, customer and decline actually are across every source. Data science builds the semantic layer, which harmonizes data into trusted KPIs such as authorization rate. Domain expertise supplies the context layer: business rules, market knowledge and strategy that no dataset contains on its own.

A payments semantic layer is what moves AI past “mostly correct.” It provides a structured taxonomy for codes and fields, logical constraints that encode how payments data is allowed to behave, and continuous updates as BINs, interchange tables and PSP specs change. Without one, every query against payments data for AI starts from ambiguous raw records and inherits all of that ambiguity.

LLMs can help build these layers, but they cannot finish them. One data practitioner quoted in the webinar described LLMs as “remarkably good at producing an initial ontology,” while noting that “every meaningful improvement came from conversations with domain experts.” Pagos works the same way: AI handles a first pass of pattern matching when the team updates its mappings, and expert analysts validate the results. Payments adds traps such as duplicated costs across fee reports or a single order split across processors, which only experienced practitioners know to look for.

Manage Your LLM Like a Junior Analyst

With data readiness as the foundation, the useful framing is to treat the model as a junior analyst you manage directly: enthusiastic and well educated in theory, but short on practical experience. That analyst needs onboarding, context, coaching and review, and should show their work, including citations. You remain responsible for the final output.

Robert keeps his prompts professional and concise, skips praise, and avoids criticizing the model’s mistakes, since a bad result often traces back to how he described the problem. He also proposes rather than asserts, because LLMs tend to accept a user’s challenge even when the user is wrong. The process he recommends combines project management with the scientific method:

  1. Discuss your intent with the model and agree on what success looks like.
  2. State your assumptions and ask the model to challenge them and identify risks and blind spots.
  3. Make observations, form hypotheses, test, review and iterate.
  4. Keep a README summarizing the approach, and save outputs as CSVs and Markdown reports for an audit trail.
  5. Update your templates and system prompts for the next project.

This replaces one long autonomous run with a series of checkpoints. A one-shot prompt fans out into many possible outcomes; a human who reviews at regular milestones narrows that spread at each step. AI owns execution, and the human owns intent, judgment and interpretation.

Human review at regular milestones narrows uncertainty at each step. Source: Pagos.

Pagos builds this into the instruction file its analysts use with Claude. An excerpt:

“When presenting the results of analysis, I want clarity on what are direct observations versus hypothetical explanations. Do not present ‘likely reasons’ as fact. […] For new areas of investigation, let me review your strategy to query the data before you run it.”

Low-Tech Orchestration Works

For merchants getting started, agents and orchestration platforms are unnecessary and can become a distraction. A shared repository with one folder per project, templates and system prompts, a glossary of payments terms and clearly named files gives an LLM the memory it needs, and Model Context Protocol (MCP) connectors can bring in other data sources. Raw data stays in a separate, compliance-appropriate location. (For a related look at turning payments data into insight, see Making Sense of Payments Data with AI.)

Andy’s own BIN analysis shows the payoff. He spent days, then weeks, building a project folder with an instruction file and glossary, and told the model to save important learnings to project memory as he went. That foundational work is what now lets the model handle higher-level analysis, and Robert estimates the approach can take a team to level 3, and partly level 4, of a five-level AI analytics maturity model published by Tellius.

Where This Goes Next

In a live poll, 44% of webinar attendees placed their company’s payments data maturity at the higher end of a five-point scale. That audience likely skews toward data teams, but the question applies to everyone: is your payments data for AI solid enough to support what comes next? Merchants that do the unglamorous work of normalizing codes, refreshing reference data and building a semantic layer will spend more of their time on hard problems, because AI will take on more of the routine ones.


This webinar was presented by Andy Barker, Senior Product Lead at Pagos, and Dr. Robert Clewley, Senior Product Lead at Pagos, as part of the PaymentsEd webinar series. Watch the full recording.