High-Availability Banking Architectures

High banking availability: end-to-end continuity, not just duplicate servers

Central principle

The critical service needs to survive process, node, zone, region, vendor, change, and human error failures.

In-depth edition - study material and professional reference

radio_button_unchecked Chapter presentation

At a financial institution, unavailability doesn't just mean a slow screen. It can prevent payments, block transfers, interrupt settlements, leave customers without access to balances, delay reconciliations and increase exposure to fraud. Therefore, the high availability banking architecture needs to be built based on critical business operations and the consequences of their interruption, and not just the duplication of servers.

The Basel Committee defines operational resilience as a bank's ability to deliver critical operations during a disruption. This perspective is broader than technical availability: it includes people, processes, technology, facilities, suppliers and external dependencies. The architecture must assume that failures will occur and establish tolerances for disruption, protection mechanisms, response, adaptation, recovery and learning.

In Brazil, the topic relates to risk management structures, cybersecurity policy, the contracting of processing and cloud services and the requirements of payment arrangements. Standards and manuals do not prescribe a single topology, but require governance, continuity, controls, testing, recoverability and third-party management. The technical design needs to transform these obligations into verifiable properties.

This chapter connects API Gateways, microservices, messaging, Kubernetes, observability, and Zero Trust to a banking vision of continuity. RTO , RPO , SLOs, fault domains, active-active , active-passive , consistency, replication, idempotence , reconciliation, capacity, security, disaster recovery and testing of severe but plausible scenarios will be studied.

How to study this chapter For each architecture, identify the critical operation, the dependency that could fail, the customer-observed effect, the accepted tolerance, the recovery strategy, and the evidence produced by testing. High availability without measurable criteria is just an intention.

Learning Objectives

• Differentiate between availability, reliability, business continuity, disaster recovery and operational resilience.
• Define critical operations, impact, disruption tolerance , RTO , RPO , SLI and SLO.
• Design redundancy by process, node, zone, region, network, vendor, and team.
• Compare active-active , active-passive , warm standby and pilot light topologies.
• Understand synchronous and asynchronous replication, quorum, split-brain and consistency.
• Apply idempotence , ledger, Outbox, reconciliation and delivery semantics to financial transactions.
• Design DNS, balancers, API Gateways, egress and connectivity with controlled failover.
• Relate security, HSM, PKI, secret management and suppliers to availability.
• Plan capacity, graceful degradation, load shedding , and protection against cascading failures.
• Build DR, chaos engineering, observability and auditable incident response tests.

Chapter structure

• 37.1 High availability and operational resilience
• 37.2 Critical operations, impact and tolerance to disruption
• 37.3 Metrics: availability, SLI, SLO, RTO and RPO
• 37.4 Fault domains and true redundancy
• 37.5 Active-active , active-passive and multi-region
• 37.6 Data: replication, quorum and consistency
• 37.7 Transactional integrity, idempotency and reconciliation
• 37.8 Network, DNS, balancers and API Gateways
• 37.9 Security, HSM, PKI, cloud and third parties
• 37.10 Capacity, backpressure and controlled degradation
• 37.11 Observability and incident response
• 37.12 Changes, DR, testing and chaos engineering
• 37.13 Hybrid architectures, mainframe, Pix and instant payments
• 37.14 Troubleshooting and case studies
• Summary, checklist, exercises, glossary and references

radio_button_unchecked 37.1 High availability and operational resilience

High availability is the ability of a service to remain accessible within an agreed level. Reliability deals with the probability of performing correctly over a period of time; business continuity deals with maintaining essential activities; disaster recovery focuses on restoring technology after a disaster; operational resilience integrates prevention, absorption, response, recovery and learning. In banks, these concepts overlap, but are not synonymous.

An application can have 99.99% uptime and still be operationally fragile if a failure produces incorrect balances, duplicate payments or incomplete reconciliation. Therefore, the quality of recovery is as important as its speed. The architecture must preserve integrity, traceability and the ability to explain the final state of each transaction.

Resilience must be assessed by critical operation, not the aggregate status of the infrastructure. The mobile application may be available while payment initiation fails due to unavailability of an HSM or an external participant. The chain needs to be modeled from channel to ledger, including authentication, gateway, services, queues, banks, networks, providers and human processes.

Layered resilience: every dependency needs a failure response

radio_button_unchecked 37.2 Critical operations, impact and tolerance to disruption

The first step is to identify critical operations, such as balance inquiry, card authorization, settlement, Pix, bill issuance, authentication, fraud prevention and partner access. The classification needs to consider customers, financial stability, regulatory obligations, financial losses, reputation and systemic effect. Internal services can also be critical when supporting multiple external journeys.

Business Impact Analysis relates operations, dependencies, peak periods, volumes, consequences and recovery priorities. Disruption tolerance expresses how much impact the institution accepts under severe but plausible scenarios. This tolerance is not just time: it can involve number of affected customers, financial volume, settlement delay, tolerable divergence and manual contingency capacity.

Mapping dependencies requires going beyond the logical diagram. It is necessary to register certificates, HSMs, DNS, queues, banks, legacy systems, cloud providers, telecommunications, on-call teams and support contracts. A shared dependency can invalidate seemingly independent redundancies. Two data centers that use the same circuit, the same certificate authority, or the same configuration pipeline have common risks.

radio_button_unchecked 37.3 Metrics: availability, SLI, SLO, RTO and RPO

Availability can be measured as the proportion of time or requests that are successful. For APIs, the indicator must consider semantics: an HTTP 200 response with incorrect data is not success. SLIs can measure success rate, latency, data freshness, completeness, processing time, and backlog. The SLO defines the objective and the error budget allows you to decide how much change or operational risk fits into the period.

RTO is the maximum time expected to restore operation after an interruption. RPO is the maximum amount of data that can be lost, expressed as the temporal distance between the last recoverable point and failure. These values guide replication, backup, failover, automation, and testing frequency. Zero RPO typically requires synchronous mechanisms or a distributed ledger with strong coordination, increasing latency and complexity.

MTTR and MTBF help observe recovery and failure frequency, but are not a substitute for experience metrics. In distributed systems, partial degradations are more common than total failures. Therefore, dashboards must show availability by operation, region, channel and dependency, as well as integrity and reconciliation indicators.

radio_button_unchecked 37.4 Fault domains and true redundancy

A fault domain is a set of components that can be affected by the same event. Processes on the same host share the node; nodes in the same rack can share power; zones can share a region; Regions can share provider, control plane, identity, or wide area network. True high availability requires identifying the biggest event that the architecture must tolerate.

Redundancy eliminates single points only when copies are independent and tested. Two firewalls with the same faulty configuration can fail simultaneously. Two clusters receiving the same incorrect change do not protect against logical error. Operational diversity, pipeline separation, staged rollout and the ability to freeze changes during an incident are part of the architecture.

Independence also needs to include people and procedures. The same team may not be able to operate two environments during a crisis. Emergency credentials, out-of-band communication, vendor contacts, and runbook access must survive the unavailability of normal systems.

Test of Independence Ask: What common cause brings down all the replicas? Whether the response includes identity, DNS, configuration repository, HSM, route, telecom, vendor, or team, redundancy still has a shared point that needs to be addressed.

radio_button_unchecked 37.5 Active-active , active-passive and multi-region

In active-active , two or more locations serve traffic simultaneously. The capacity is already warmed up and the fault can be absorbed more quickly. However, the model requires data distribution, routing, split-brain prevention, configuration consistency and cross-region reconciliation. Gaining RTO may be accompanied by greater complexity and ongoing cost.

In active-passive , one primary location responds and another remains ready to take over. Standby can be hot, warm or cold, depending on capacity, data and active components. This model simplifies some consistency decisions, but failover needs to be automated and tested. Passive environments that never receive traffic tend to accumulate drift, expired certificates, and unvalidated dependencies.

Multi-region architectures need to define failover granularity. Moving the entire institution at once increases the blast radius ; moving operations by domain allows control but requires decoupled dependencies. Returning to the original region, or failback , must be treated as another high-risk change, with its own synchronization and criteria.

radio_button_unchecked 37.6 Data: replication, quorum and consistency

Data is typically the hardest part of high availability. Synchronous replication confirms writing only after reaching the required number of replicas, reducing RPO , but increasing latency and network sensitivity. Asynchronous replication allows for faster response, but may lose not-yet-replicated transactions during failure. The decision must be made by data class, not by a single rule for the entire platform.

Quorum-based systems require that reads and writes reach a sufficient number of nodes to preserve desired properties. However, quorum does not automatically eliminate conflicts. It is necessary to understand leader election, fencing tokens, clocks, linearizable consistency, eventual consistency and behavior under partitions. Split-brain occurs when isolated sides accept incompatible operations; prevention and reconciliation need to be explicit.

Financial ledgers typically require logical immutability, double entry, auditable trail, and reconstructability. Caches and materialized views can eventually be consistent, as long as the authoritative system remains clear. The architecture must differentiate decision data, derived data and presentation data, preventing a late replica from being treated as an official source.

radio_button_unchecked 37.7 Transactional integrity, idempotency and reconciliation

Failures in financial transactions generate ambiguous states: the customer received a timeout, but the payment was accepted; the gateway repeated the request; the service recorded the ledger, but did not publish the event; the message was consumed twice. The architecture needs to resolve ambiguity through unique identifiers, idempotency keys, transactional states, status query, and reconciliation.

Idempotence allows you to repeat an operation without producing additional financial effects. The key needs to be linked to the consumer, the operation and the relevant payload. The server must persist result or state before responding so that competing retries do not bypass protection. For messaging, Inbox, deduplication and transactional processing reduce delivery effects at least once.

The Transactional Outbox pattern records the business change and event in the same local transaction; a separate publisher sends the event later. This avoids dual writing between bank and broker. Still, consumers must tolerate duplicates. Reconciliation compares authoritative sources, events, settlement, and statements to identify differences that online controls did not resolve.

Conceptual flow of idempotent processing POST/payments HTTP/1.1 Idempotency-Key: 98f4d0b2-...

1. Reserve the key and validate the command 2. Record the transaction in the ledger 3. Record event in Outbox 4. Respond with identifier and status 5. Publish event and reconcile asynchronously

radio_button_unchecked 37.8 Network, DNS, balancers and API Gateways

The path of a banking API goes through DNS, edge protection, balancers, API Gateways, private networks, proxies and backends. Each layer needs health checks that reflect real capacity. A TCP check can declare healthy an instance unable to access the HSM or the bank. Readiness must consider essential dependencies without creating a cascade in which all replicas are removed simultaneously.

DNS and GSLB can direct traffic between regions, but TTL, caches, and persistent clients delay convergence. L4 and L7 balancers can maintain existing connections to the previous region. In HTTP/2, gRPC, and WebSocket, long-lasting connections require controlled draining and reconnection. Failover needs to be tested with real customers and not just administrative queries.

API Gateways must distribute configuration securely, maintain critical policies during control plane loss, and have low-latency dependencies for rate limiting, authentication, and routing. An architecture can choose fail-open or fail-closed by control, but the decision must consider fraud, unavailability and integrity. Key and policy caching needs expiration, revocation, and observability.

radio_button_unchecked 37.9 Security, HSM, PKI, cloud and third parties

Security is dependent on availability. If the identity authority, HSM, PKI, or secret service fails, healthy applications may stop authenticating and signing operations. These components need redundancy, capacity, safe rotation, and emergency procedures. Contingency certificates should not be improvised during the incident.

Hiring cloud and external processing requires knowing regions, responsibility models, support, portability, backup, administrative access and subcontractors. Multi-cloud can reduce vendor lock-in, but increases complexity, operational differences, and error surface. In many cases, a well-tested architecture at one provider with data portability is more resilient than two unevenly maintained clouds.

Zero Trust remains applicable during contingency. Break-glass accounts must be heavily secured, audited and tested. The pressure to recover does not justify deactivating timeless controls and compensations. The response needs to record who authorized, what restrictions were relaxed and when the normal configuration will be restored.

Hidden Dependency Risk Critical systems may share the same HSM, identity tenant, secret vault, pipeline, or telecommunications provider. These components need to appear in the dependency map and test scenarios, even when they are not part of the visible functional flow.

radio_button_unchecked 37.10 Capacity, backpressure and controlled degradation

Failover increases load on surviving infrastructure. If two regions operate at 70% each, the loss of one cannot be absorbed by the other. Capacity planning needs to consider peaks, maintenance, growth, retries, and dependency degradation. N+1 means that the operation continues after losing a component; in multi-region, the institution must decide whether to require full capacity or degraded service.

Backpressure prevents producers from overcharging consumers. Queues, concurrency limits, rate limiting and circuit breakers must keep the system in a recoverable range. Retries need backoff, jitter and budget; otherwise, they multiply traffic during the failure. Load shedding rejects lower priority work to preserve critical operations.

Graceful degradation must be planned in advance. The bank can maintain balance inquiry and payments while suspending non-essential reporting, recommendations and enrichment. Degraded responses need to be explicit to avoid decisions based on outdated data. Contingency feature flags must be tested, versioned and protected against improper activation.

radio_button_unchecked 37.11 Observability and incident response

Observability needs to respond to whether the critical operation is available, correct, and within tolerance. Infrastructure metrics are insufficient without business signals: payments accepted, settled, pending, duplicated, reversed, and reconciled. Distributed traces help locate dependency, while audit logs explain decisions and changes.

Alerts must be actionable and SLO-driven. A small error rate can be critical in a high-value operation; high latency may be tolerable in reporting. Burn-rate alerts detect accelerated error budget consumption. Synthetic transactions and external probes verify the complete journey, but they need to use controlled data and do not create real financial effects.

During the incident, the command needs to combine technology, business, security, communications and suppliers. Runbooks describe actions, preconditions and rollback, but are not a substitute for judgment. After recovery, reconciliation and root cause analysis must occur before declaring definitive closure.

Operational resilience cycle

radio_button_unchecked 37.12 Changes, disaster recovery and chaos engineering

Changes are one of the main sources of unavailability. Progressive delivery, canary, blue-green, feature flags, automatic validation and rollback reduce risk. Schema changes and data migrations require expand-migrate-contract to enable coexistence. Automation should block promotion when SLOs, reconciliation, or security checks fail.

Disaster recovery needs to be exercised with real objectives. Just testing machine creation does not prove that the critical operation works. Testing should include enabling traffic, identity, keys, data, queues, providers, channels, and feedback. Evidence must record times, data loss, manual interventions, deviations and corrective actions.

Chaos engineering introduces controlled failures to validate hypotheses. In banks, the scope needs to respect risk and environment: pod loss, network delay, zone unavailability, certificate expiration, backlog or HSM failure can be simulated progressively. The objective is not to cause chaos, but to discover weaknesses before a real event.

radio_button_unchecked 37.13 Hybrid architectures, mainframe, Pix and instant payments

Banks often combine mainframes, middleware, distributed banking, cloud and gateways. The mainframe can be the authoritative ledger, while pipelines and APIs operate on modern platforms. High availability depends on the coupling between these worlds: queues, transactions, adapters, batch windows, host capacity and connection limits need to be observed together.

Instant payments increase the requirement for continuous operation and low latency. The institution needs to treat connectivity, certificates, signature, fraud prevention, limits, messaging, reconciliation and contingency as a chain. A customer response should distinguish outright rejection, pending processing and unknown status, allowing for later consultation without repeating the financial effect.

In hybrid architectures, migration to the cloud must occur across capabilities and domains, not just by copying machines. Strangler, events, anti-corruption APIs, and controlled replication enable gradual transition. Coexistence needs clear data ownership and legacy disconnection criteria to avoid two authoritative systems.

Pix Case A communication failure after sending an order should not be automatically treated as a business failure. The system needs to query the status, use unique identifiers, and reconcile before allowing another attempt that could duplicate the payment.

radio_button_unchecked 37.14 Troubleshooting and case studies

High availability troubleshooting starts with the timeline. Determine what change or failure occurred, what region or zone was affected, when the health check reacted, when routing converged, and what data state was promoted. Logs without synchronized clocks and correlation IDs make reconstruction difficult.

For intermittent failovers, investigate flapping health checks, TTLs, connection pools, sessions, leader elections, and dependencies with insufficient capacity. In data divergence, discontinue conflicting new writings, preserve evidence, and establish the authoritative source before reconciling. Regaining availability at the expense of integrity can compound the loss.

Case Study: A primary region loses connectivity to the database. The balancer maintains instances as healthy because the technical endpoint responds. Clients receive errors for several minutes. The fix involves operation-oriented readiness, circuit breaker, gradual removal of traffic, replica promotion, lag checking and subsequent reconciliation.

Case study: after a deployment, the two active-active clusters start to reject tokens due to incorrect issuer configuration. Redundancy did not protect against common configuration failure. The answer requires independent rollback, staged rollout, synthetic validation, and pipeline separation between deployment waves.

radio_button_unchecked Chapter summary

High-availability banking architectures need to start from critical operations and their tolerances to disruption. Technical availability alone does not guarantee resilience: integrity, reconciliation, security, people, suppliers and communication are part of the result.

RTO , RPO , SLOs, and tolerances drive topology, replication, capacity, and testing. Active-active and active-passive have different trade-offs; Neither option eliminates the need to understand data, common dependencies, and behavior under partition.

Financial transactions require idempotence , authoritative ledger, Outbox, status query and reconciliation. Networking, DNS, gateways, HSMs, identity, and cloud are also critical dependencies. Operational resilience is proven through exercises, evidence and continuous learning.

Next step of the course Chapter 38 will delve deeper into troubleshooting APIs and Gateways, consolidating a methodology for locating flaws in DNS, TCP, TLS, authentication, policies, backends, data and distributed platforms.

radio_button_unchecked Architecture checklist

• Critical operations, owners, dependencies and tolerances are documented.
• SLIs and SLOs measure actual success, latency, integrity, and freshness per operation.
• RTO and RPO have compatible technical mechanisms and tests.
• Fault domains and shared dependencies were identified.
• Active-active or active-passive topology has promotion and failback criteria.
• Replication, quorum, fencing and split-brain prevention are defined.
• Transactions use idempotence , ledger, Outbox and reconciliation.
• DNS, balancers, gateways, persistent connections, and clients were tested on failover.
• HSM, PKI, identity, secrets, cloud and third parties participate in the exercises.
• Survivable capability supports failure with margin and explicit priorities.
• Backpressure, retries, circuit breakers and load shedding prevent cascades.
• DR, restore, chaos tests and runbooks produce evidence and corrective actions.

radio_button_unchecked Exercises

• Differentiate high availability, disaster recovery and operational resilience.
• Define RTO and RPO for balance consultation, Pix and monthly report, justifying differences.
• Design an active-active architecture and identify consistency risks.
• Explain why replication does not replace backup.
• Describe how to prevent duplicate payments after timeout.
• Propose health checks for gateway, service and ledger.
• List common dependencies that can bring down two regions simultaneously.
• Define a graceful degradation strategy for a digital bank.
• Create a DR test that validates full journey and reconciliation.
• Analyze a scenario in which the service returns quickly, but the data is divergent.

radio_button_unchecked Glossary

radio_button_unchecked Technical references

• Basel Committee on Banking Supervision. Principles for Operational Resilience. 2021.
• Basel Committee on Banking Supervision. Revisions to the Principles for the Sound Management of Operational Risk. 2021.
• Central Bank of Brazil. CMN Resolution No. 4,557, of February 23, 2017 - risk and capital management structure.
• Central Bank of Brazil. CMN Resolution No. 4,893, of February 26, 2021, with subsequent amendments - cybersecurity and contracting of processing, storage and cloud.
• Central Bank of Brazil. CMN Resolution No. 5,274, of December 18, 2025 - changes to cybersecurity requirements and technology services.
• Central Bank of Brazil. Pix regulations and current technical and safety manuals.
• NIST. SP 800-34 Rev. 1 - Contingency Planning Guide for Federal Information Systems.
• NIST. Cybersecurity Framework 2.0.
• Google. Site Reliability Engineering - Availability, SLOs and error budgets.
• OpenTelemetry. Specifications and Semantic Conventions for distributed observability.

Update Note Banking regulations, payment arrangement manuals and provider requirements evolve. Before applying controls or declaring compliance, validate the current Central Bank text, critical service manuals and the actual capabilities of the deployed version.