What If the Money Moves, But the Transaction Doesn't?

Note: Guys, For better explaining of the scenario in learning purpose and analysis, I'm using an bank name lets say Sober Bank(SB)

Imagine in our Sober Bank the database is completely correct… and we're still missing ৳10,000.

Sober Bank says the money moved. The customer says they paid. The payment gateway says SUCCESS.

But our database has no transaction record !!!!.

What should a fintech system do now?

---

Abstraction

I personally found most engineers explain payments as a single event: "the customer paid." In reality, a payment is a chain of independent systems — the customer's bank, the payment gateway, our microservice application, our database, our ledger and the merchant settlement account — each of which can succeed or fail independently of the others.

There is no distributed transaction that spans a customer's bank, a third-party payment gateway, and our own database. we cannot wrap all of them in one BEGIN TRANSACTION ... COMMIT. Which means the real engineering question in fintech is never "how do I charge a card?" It's:

"How do I design a system that never truly loses track of money, even when one link in the chain fails after the money has already moved?"

That is the problem I'm gonna explain in this post.

---

The Actual Scenario

1. A customer pays ৳10,000 to sober bank.
2. then Sober bank debits the customer's account.
3. The payment gateway receives confirmation and returns SUCCESS.
4. and then our fintech application crashes — let's say due to pod restart, or unhandled exception — before it writes the transaction record to our database.

Now four different systems hold four different "truths":

Bank: debited ✅        Gateway: SUCCESS ✅        sober bank DB: ❌ (no record)
        └────────────────────┬────────────────────┘
                    Where does ৳10,000 belong?

The money isn't with the customer anymore. It isn't reflected in the merchant's ledger yet. And our system — the one piece of software that's supposed to be the source of truth — has no idea any of this happened.

This is not a bug in the traditional sense. No line of code was "wrong." It's a gap in a system that was never designed to assume partial failure as normal.

---

Why This Isn't a Bug — It's a Distributed Systems Reality

In a monolith with one database, "crash before commit" usually just means "nothing happened, retry." But In fintech, that assumption is false, because the side effect already happened outside our database — at the sober bank, at the gateway. our database is not the only source of truth anymore; it's one participant in a distributed, eventually-consistent system.

A production-grade fintech system has to be built on a different assumption from day one:

Failures will happen between steps, not just within them. Design for recovery, not just for the happy path.

That means idempotency, event sourcing of intent, and reconciliation aren't "nice to have" — they are the core domain logic, as important as the payment calculation itself.

---

Solutions

Below is how a system built with Clean Architecture + DDD + microservices — by following clean architecture — can guarantee that ৳10,000 is never actually lost, only ever temporarily untracked.

1. Persist Intent Before we Call the Gateway (Idempotency First)

The fix starts before the crash even has a chance to matter. Generate a TransactionReference and persist a Pending payment row before calling the gateway — not after. Now there is always a row to reconcile against, no matter when the crash happens.

// Payments.Domain/Entities/PaymentTransaction.cs
namespace Payments.Domain.Entities;

public class PaymentTransaction
{
    public Guid Id { get; private set; }
    public string TransactionReference { get; private set; } = null!; // idempotency key
    public decimal Amount { get; private set; }
    public PaymentStatus Status { get; private set; }
    public DateTime CreatedAtUtc { get; private set; }

    private PaymentTransaction() { }

    public static PaymentTransaction Initiate(decimal amount, string transactionReference)
    {
        return new PaymentTransaction
        {
            Id = Guid.NewGuid(),
            TransactionReference = transactionReference,
            Amount = amount,
            Status = PaymentStatus.Initiated,
            CreatedAtUtc = DateTime.UtcNow
        };
    }

    public void MarkAuthorized() => TransitionTo(PaymentStatus.Authorized);
    public void MarkCaptured()   => TransitionTo(PaymentStatus.Captured);
    public void MarkSettled()    => TransitionTo(PaymentStatus.Settled);
    public void MarkFailed() => TransitionTo(PaymentStatus.Failed);
    public void Reverse()  => TransitionTo(PaymentStatus.Reversed);

    private void TransitionTo(PaymentStatus next)
    {
        if (!PaymentStatus.IsValidTransition(Status, next))
            throw new InvalidOperationException($"Cannot move payment {TransactionReference} from {Status} to {next}.");

        Status = next;
    }
}

2. Payment State as a First-Class Domain Model

A payment is never just "success" or "failure" — it moves through a state machine. Modeling that explicitly in the domain (not as a loose string status column) means invalid states are structurally impossible.

// Payments.Domain/Entities/PaymentStatus.cs
namespace Payments.Domain.Entities;

public sealed class PaymentStatus
{
    public static readonly PaymentStatus Initiated  = new(nameof(Initiated));
    public static readonly PaymentStatus Authorized = new(nameof(Authorized));
    public static readonly PaymentStatus Captured   = new(nameof(Captured));
    public static readonly PaymentStatus Settled    = new(nameof(Settled));
    public static readonly PaymentStatus Failed     = new(nameof(Failed));
    public static readonly PaymentStatus Reversed   = new(nameof(Reversed));

    private static readonly Dictionary<string, string[]> AllowedTransitions = new()
    {
        [nameof(Initiated)]  = [nameof(Authorized), nameof(Failed)],
        [nameof(Authorized)] = [nameof(Captured), nameof(Failed)],
        [nameof(Captured)]   = [nameof(Settled), nameof(Reversed)],
    };

    public string Value { get; }
    private PaymentStatus(string value) => Value = value;

    public static bool IsValidTransition(PaymentStatus from, PaymentStatus to) =>
        AllowedTransitions.TryGetValue(from.Value, out var next) && next.Contains(to.Value);

    public override string ToString() => Value;
}

3. The Transactional Outbox Pattern (No Event Is Ever Lost)

The dangerous gap is: "gateway call succeeded, but publishing the follow-up event failed." The outbox pattern closes it — the PaymentTransaction row and its domain event are written in the same database transaction. A separate dispatcher then reliably delivers the event to Kafka/RabbitMQ, retrying until it's acknowledged.

// Payments.Application/Payments/Commands/InitiatePaymentCommandHandler.cs
public class InitiatePaymentCommandHandler : IRequestHandler<InitiatePaymentCommand, PaymentTransaction>
{
    private readonly IPaymentsDbContext _db;

    public InitiatePaymentCommandHandler(IPaymentsDbContext db) => _db = db;

    public async Task<PaymentTransaction> Handle(InitiatePaymentCommand request, CancellationToken ct)
    {
        var transaction = PaymentTransaction.Initiate(request.Amount, request.TransactionReference);

        await using var dbTransaction = await _db.Database.BeginTransactionAsync(ct);

        _db.PaymentTransactions.Add(transaction);
        _db.OutboxMessages.Add(OutboxMessage.From(new PaymentInitiatedEvent(transaction.TransactionReference, transaction.Amount)));

        await _db.SaveChangesAsync(ct);   // transaction row + outbox row commit atomically
        await dbTransaction.CommitAsync(ct);

        return transaction; // gateway call happens AFTER this is safely persisted
    }
}

// Payments.Infrastructure/Outbox/OutboxDispatcher.cs
public class OutboxDispatcher : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var pending = await _db.OutboxMessages
                .Where(m => m.ProcessedAtUtc == null)
                .OrderBy(m => m.CreatedAtUtc)
                .Take(50)
                .ToListAsync(ct);

            foreach (var message in pending)
            {
                await _eventBus.PublishAsync(message.Type, message.Payload, ct); // Kafka / RabbitMQ
                message.MarkProcessed();
            }

            await _db.SaveChangesAsync(ct);
            await Task.Delay(TimeSpan.FromSeconds(2), ct);
        }
    }
}

4. Idempotent Webhook Consumer

The gateway's webhook is the system's only external confirmation. It can arrive late, out of order, or more than once — the consumer must be safe to run twice with the same payload, using TransactionReference as the idempotency key.

// Payments.Api/Webhooks/GatewayWebhookController.cs
[ApiController]
[Route("webhooks/gateway")]
public class GatewayWebhookController : ControllerBase
{
    private readonly ISender _mediator;

    [HttpPost("payment-confirmed")]
    public async Task<IActionResult> OnPaymentConfirmed(GatewayWebhookPayload payload, CancellationToken ct)
    {
        // TransactionReference is the idempotency key — replays are safe.
        await _mediator.Send(new ReconcileGatewayConfirmationCommand(
            payload.TransactionReference,
            payload.GatewayStatus,
            payload.Amount), ct);

        return Ok();
    }
}

public class ReconcileGatewayConfirmationCommandHandler
    : IRequestHandler<ReconcileGatewayConfirmationCommand>
{
    public async Task Handle(ReconcileGatewayConfirmationCommand request, CancellationToken ct)
    {
        var transaction = await _repository.FindByReference(request.TransactionReference);

        if (transaction is null)
        {
            // The crash scenario: gateway says SUCCESS, but we never persisted a row.
            // Recreate it into a Suspense state instead of dropping the confirmation.
            await _repository.AddAsync(PaymentTransaction.FromSuspenseRecovery(request), ct);
            return;
        }

        if (transaction.Status == PaymentStatus.Captured) return; // already processed — idempotent no-op

        transaction.MarkCaptured();
        await _repository.SaveAsync(transaction, ct);
    }
}

5. Double-Entry Ledger — Money Is Never "Missing," Only Unclassified

Every movement of money is recorded as a balanced pair: a debit and a credit. If the destination account isn't confirmed yet, the amount goes into a suspense/clearing account — never nowhere.

// Payments.Domain/Entities/LedgerEntry.cs
public class LedgerEntry
{
    public Guid TransactionReferenceId { get; private set; }
    public string DebitAccount { get; private set; } = null!;
    public string CreditAccount { get; private set; } = null!;
    public decimal Amount { get; private set; }

    public static LedgerEntry ToSuspense(string transactionReference, decimal amount) =>
        new()
        {
            TransactionReferenceId = Guid.Parse(transactionReference),
            DebitAccount = "Customer:PendingClearing",
            CreditAccount = "Bank:SuspenseAccount", // money is parked here, never lost
            Amount = amount
        };

    public static LedgerEntry ToMerchant(string transactionReference, decimal amount, string merchantAccount) =>
        new()
        {
            TransactionReferenceId = Guid.Parse(transactionReference),
            DebitAccount = "Bank:SuspenseAccount",
            CreditAccount = merchantAccount,
            Amount = amount
        };
}

Until reconciliation confirms where the money belongs, it sits in Bank:SuspenseAccount — visible, auditable, and flagged, instead of silently absent from every report (the same failure mode as [[NULL-vs-Zero]] — an unrecorded state should never be indistinguishable from "nothing happened").

6. Reconciliation as Its Own Microservice

A dedicated Reconciliation Service periodically diffs the gateway's settlement report against the internal ledger and auto-heals mismatches — replaying missing webhooks, re-emitting outbox events, or raising an alert for manual review.

Payment Gateway
      ↓
Transaction Reference
      ↓
Webhook / Event
      ↓
Payment Service
      ↓
Ledger
      ↓
Reconciliation
      ↓
Settlement