The dashboard said everything was green. Transaction volume normal. API latency healthy. Error rate within SLO. Every metric the team had set up was telling them the system was fine. And the merchant on the phone was telling them that their daily settlement was off by $4,000, and they wanted to know why.
This kind of incident is the defining observability failure of payment systems, and I have debugged variations of it at every company I've ever worked at. The monitoring is tracking what it's easy to track — uptime, latency, error rates — and not what actually matters, which is whether the system is producing correct financial records. Those are different questions, and the techniques for answering the first do not answer the second.
I want to be specific about what payment observability actually requires, because the generic advice — "add metrics, add tracing, add alerts" — is the same advice that produces the dashboards that were green when the merchant was calling.
The three things you're trying to see
Observability in payment systems has to give you visibility into three distinct things, and the techniques for each are different.
Operational health. Is the system running? Are requests getting handled? Are the external dependencies responding? This is what traditional APM covers. It's necessary but nowhere near sufficient.
Business correctness. Is the system producing the right financial outcomes? Are transactions being authorized, captured, settled, and reconciled the way they're supposed to be? Are the numbers we're showing merchants the numbers the money is actually at?
Security and compliance. Is access happening as expected? Are audit trails complete? Is data flowing only where it's supposed to flow?
Most teams invest heavily in the first, lightly in the third, and almost nothing in the second. The second is where most of the real incidents live.
Logs: what to log and what not to log
Logging is where payment systems hit their first observability tension: you need enough logs to debug production issues, but not so many that you accidentally log cardholder data into a non-PCI-scoped system.
The rule I follow is: log everything that isn't cardholder data, and structure logs so that cardholder data can't accidentally get in. This means:
- Never log the full PAN, CVV, expiration date, or magnetic stripe data. Not even in error paths. Not even "just while debugging." Log scrubbers are a defense, not a primary mechanism — the primary mechanism is that this data never touches loggable code paths in the first place.
- Log tokens, not raw card data. The tokens are safe to log; the raw data isn't. If your application is processing raw card data somewhere, that's a PCI scope problem, not a logging problem.
- Log transaction references, not card references. The transaction ID is what you use to look up a transaction, not the card number. Always.
- Log the merchant and location context. Every log line should be associable with a merchant and, when applicable, a location and terminal. Unscoped logs are almost useless at diagnosis time.
- Log the processor and attempt number. When multiple processors are in play, knowing which one handled a specific request, and whether this was a retry, is essential.
Structure matters as much as content. Unstructured free-text logs are not queryable at scale. Use structured logging (JSON or similar), with consistent field names across services, and use a centralized log aggregator that supports querying those fields. The difference between grep-able logs and queryable structured logs is the difference between debugging in ten minutes and debugging in ten hours.
Metrics that actually matter
The standard web-service metrics — request rate, latency percentiles, error rate — are necessary but far from sufficient for payment systems. The metrics that actually tell you something about payment health:
Authorization rate. The fraction of authorization requests that succeeded. This is not the same as HTTP success rate — a 200 response containing a declined authorization is a business failure, not a technical failure. Track authorization success as a separate metric, broken down by processor, by card brand, by merchant, by transaction size bucket.
Decline rate by reason. A 5% decline rate across your platform might be completely normal or completely broken depending on the mix of decline reasons. "Insufficient funds" is expected. "Invalid merchant ID" means your processor configuration is broken. "Fraud suspected" trending upward means something upstream has changed.
Settlement-to-authorization ratio. In a healthy system, nearly every authorization results in a corresponding settlement entry within a predictable window. Drift between authorization volume and settlement volume — by merchant, by processor, by day — is an early warning of reconciliation problems. If you authorized $100,000 on Monday and only $97,000 settled by Wednesday, you need to know before the merchant does.
Reconciliation drift. The running delta between "money we think we have" and "money the processor reports we have." This should be near zero and should trend to zero as settlement catches up with authorizations. Persistent drift, especially in a specific direction, indicates a class of bugs that no other metric will surface.
Time to reconciliation. How long does it take from authorization to fully reconciled? This metric doesn't appear in any dashboard framework out of the box, but it's one of the most valuable signals you can have. A growing time-to-reconciliation means your reconciliation pipeline is falling behind, and reconciliation behind is a leading indicator of discovered discrepancies piling up.
Chargeback rate. Chargebacks as a percentage of transaction volume, tracked per merchant. Exceeds thresholds trigger processor-level consequences (higher fees, account review, termination). You want to see it approaching the threshold well before it crosses.
Processor latency and availability per processor. Each processor you integrate with is a separate dependency with its own health curve. Aggregate metrics hide per-processor degradation.
Distributed tracing across processor boundaries
A single payment transaction passes through your gateway, your payment service, your abstraction layer, your processor integration, the processor's API, the card network, and possibly back through webhook handlers into your ledger and reconciliation pipeline. When something goes wrong somewhere in that chain, you need to be able to trace the entire flow.
Standard distributed tracing (OpenTelemetry, Jaeger, Zipkin) handles the part inside your systems. The part that crosses into the processor is harder — you can't instrument their code, but you can capture trace context in the fields you send and the responses they send back. Most processors don't support W3C trace context headers, but most have an optional "metadata" or "reference" field that lets you propagate your own trace ID. Using it lets you correlate the processor's logs (when you get them) with your traces.
For webhook-driven flows, the trace context lives in the event itself. The webhook handler picks up the trace ID from the event payload and continues the trace in your system. If the trace ID is missing or malformed, you have a gap — and gaps are exactly what you want to know about, because they indicate flows where visibility has been lost.
Trace sampling matters. For payment systems, I lean toward higher sampling rates than typical web services — often 100% for payment-related requests. The volume is manageable because payment request volume is lower than web request volume for most POS platforms, and the value of having a complete trace when something goes wrong dwarfs the storage cost.
The silent failure problem
The hardest bugs to find in payment systems are the ones that don't produce errors. A transaction succeeds technically but fails semantically — it was recorded, but in the wrong state. The metrics look fine. The logs show success. The merchant's reporting eventually shows a discrepancy and nobody knows why.
The defense against silent failures is invariant monitoring. You define the invariants that should hold across your system and continuously verify them. When an invariant fails, that's an alert — even if every other metric is green.
Examples of payment-system invariants:
- The sum of all ledger entries for a merchant equals the merchant's balance.
- Every authorized transaction either has a corresponding capture, a corresponding void, or is within its authorization validity window.
- Every captured amount equals the sum of corresponding settlement entries, within the settlement window.
- Every refund has a parent transaction, and the refund amount does not exceed the captured amount.
- Every settlement record is either matched to an internal transaction or explicitly in investigation.
- Every merchant's chargeback rate is below the processor's threshold.
Each of these is a SQL query you can run continuously. The result should be zero violations. Anything nonzero is a signal — either a real problem or a gap in the invariant definition that needs to be tightened.
The hardest part is writing the invariants. They are domain-specific, they evolve as the product evolves, and they're invisible to the monitoring tools you buy off the shelf. They have to be crafted by engineers who understand both the payment domain and the specific architecture. This is work that nobody wants to do, and it's the work that catches the most important bugs.
Synthetic transactions
Real production traffic does not exercise every code path you care about. Some paths are rare — partial captures, specific refund scenarios, webhook delivery failures, edge-case cards. If you wait for production traffic to hit them, you might not know they're broken until a real merchant does.
Synthetic transactions are the solution. You run a small, controlled volume of real transactions through production using test merchant accounts, test cards, and predetermined scenarios. The synthetic transactions exercise the paths you care about — including edge cases — and their outcomes are verified end-to-end. If a synthetic transaction fails or produces the wrong result, that's an immediate alert, independent of whether any real merchant has been affected yet.
The discipline is:
- Test merchant accounts live in production, not in staging.
- Test cards are funded with real money (a small amount) so they exercise the same authorization and settlement pipeline as real transactions.
- The synthetic scenarios cover the high-value paths: authorization, capture, partial capture, void, refund, partial refund, chargeback (via test card schemes that support this).
- Synthetic transactions run on a regular cadence (every few minutes for the most critical paths, every hour for others) and feed their success/failure into alerting.
This is operationally more expensive than mocking, but it's the only way to verify that the production system — with its real processors, real network, real quirks — is actually working. When the synthetic transaction alarm goes off, you know with high confidence that something real is broken, and you know where.
Alerting that doesn't cry wolf
Alert fatigue is the failure mode of most observability systems. If you wake up operators for every minor fluctuation, they stop responding, and the one time it's real, nobody shows up.
The principle for payment system alerts: alert on impact, not on metrics. A latency spike is not an alert. A latency spike that correlates with declined transactions is an alert. A single error is not an alert. An error rate that exceeds historical bounds for this merchant, this processor, this hour of day is an alert.
This requires baseline-aware alerting rather than threshold alerting. A fixed threshold ("alert if error rate > 5%") is either too sensitive (fires constantly during normal variation) or too loose (doesn't fire until the system is well into failure mode). Baseline-aware alerts compare current behavior to historical behavior at the same time of day, day of week, and for the same segment (merchant, processor, etc.).
For payment systems, specific alerts that I find valuable:
- Authorization rate deviation below expected baseline for a specific processor — indicates that processor has degraded or changed behavior.
- Decline reason distribution shift — sudden appearance of a decline reason that wasn't common before indicates something changed upstream.
- Reconciliation lag growth — time between transaction and reconciliation is increasing, reconciliation pipeline is falling behind.
- Invariant violation — any invariant check failing, with context about which one and which merchant is affected.
- Synthetic transaction failure — a controlled test failed, and the nature of the failure indicates what's broken.
- Settlement file anomaly — settlement file arrived late, was missing expected records, or had records that didn't match any transaction.
Each of these alerts maps to a runbook. The alert says "this happened"; the runbook says "here's what to check, here's how to confirm, here's what to do about it." Runbooks are part of the observability investment — an alert without a runbook is just noise.
The audit trail
For payment systems, observability bleeds into audit. Every action that moves money, changes configuration, or accesses sensitive data has to be auditable after the fact. This is a compliance requirement for PCI and many other regimes, and it's also a forensics requirement — when something goes wrong, you need to know what happened, in what order, caused by whom.
An audit log is different from a metric or an application log. It's a structured record of specific events, retained for years, protected from tampering, and queryable by compliance and security personnel who are not engineers.
The audit log has its own storage (often a write-once database or append-only stream), its own retention (usually years, often decades), and its own access controls (only specific roles can read it, nobody can modify it). Building an audit log right is work; building one wrong looks identical to building one right, until you have an incident and the log turns out to have gaps.
The dashboard problem
A dashboard that shows you the system's state is useful. A dashboard that shows you everything is useless.
The dashboards I actually use for payment systems are small. They show the metrics that matter most — authorization rate, reconciliation lag, chargeback rate, invariant status — for the dimensions that matter (by processor, by merchant tier). They update in real time. They flag anything outside normal ranges with visual cues, not subtle color changes.
The dashboards that teams usually build are dense collections of every metric anyone thought of, most of which are green most of the time, and when something's wrong, nobody can tell because the broken metric is indistinguishable from the forty others. A dashboard should tell you the state of the system at a glance, not require you to scan for the anomaly.
For payment systems, I'd rather have three well-chosen dashboards than thirty comprehensive ones. Focus on the questions you'd ask if someone called right now and said something was wrong, and build the dashboard that answers those questions.
The deeper principle
The goal of observability in payment systems is not to have visibility into everything — it's to have visibility into the questions that matter. "Is money moving correctly?" is the question that matters. Every other metric is in service of that one.
A team that's optimizing its observability for payment systems should be asking, continuously: "If money were being lost right now, would our current observability tell us?" Most teams' observability would not. The monitoring is tracking operational health, and operational health can be fine while financial health is failing.
Build the observability around the failures you're trying to prevent. In payments, that's money moving the wrong way, money disappearing, money being double-counted, or records diverging from reality. None of these show up as errors. All of them show up as invariant violations, if you've defined the invariants.
The team that catches these bugs first is the team that invested in the right observability. The team that catches them last is the one where the merchant called support.
This is part of a series on payment systems architecture. See also the hardest part of payment systems is reconciliation and testing payment systems is nothing like normal software.