← All posts

Logging and Observability

One request, one correlation id, every step on record.** PayNexa writes structured logs with Serilog from every service. Each log goes to three places at once: the console, a rolling JSON file, and Seq, a searchable log server running in Docker. OpenTelemetry traces go to Seq as well. Every line from the first byte of a request to the last Kafka message it causes carries the same `CorrelationId`: HTTP, validation, SQL Server, MongoDB, Redis, the outbox, Kafka and calls to other services. Personal data and secrets are masked before anything is written. This document uses the **Customer API** as the worked example. Every other PayNexa service gets the same behavior from the shared building blocks with no extra code.

SerilogSeqOpenTelemetryPII maskingKafkaLifecycle logging

Contents

  1. Github Open Source
  2. The big picture
  3. Where Serilog runs and where logs go
  4. Prerequisites
  5. Startup: how logging is switched on
  6. The HTTP pipeline: what wraps every request
  7. Inside a command: behaviors and steps
  8. Data stores, cache and broker
  9. After the response: outbox, projection, Kafka
  10. Correlation: how one id follows the whole journey
  11. Service-to-service calls, retries and outages
  12. Errors and exceptions
  13. Anatomy of a log event
  14. Masking personal data and secrets
  15. Levels, event ids and noise control
  16. Reading the logs: console, file, Seq
  17. What this gives the business and the code
  18. Configuration reference
  19. Known limitations
  20. Real console output, end to end
  21. Seeing it in action: screenshots

0. GitHub Open Source

Item Details
Repository PayNexa
Branch feature/logging_and_observability

1. The big picture

flowchart TB
    Client["Client / Swagger"] -->|"HTTPS + X-Correlation-Id"| Gateway["YARP API Gateway<br/>service-apigateway"]
    Gateway -->|"forwards X-Correlation-Id"| Customer["Customer API<br/>service-customer-api"]

    subgraph CustomerProcess["Customer API process"]
        direction TB
        Pipeline["HTTP pipeline<br/>correlation · request logs · body logs"]
        Mediator["Mediator pipeline<br/>LoggingBehavior · ValidationBehavior"]
        Handler["Command / query handler<br/>logger.BeginStep(...)"]
        Stores["SQL · MongoDB · Redis · Kafka<br/>interceptors and steps"]
        Serilog["Serilog logger<br/>enrichers + masking"]
        Pipeline --> Mediator --> Handler --> Stores
        Pipeline -.-> Serilog
        Mediator -.-> Serilog
        Handler -.-> Serilog
        Stores -.-> Serilog
    end

    Customer --- CustomerProcess

    Serilog -->|"text"| Console["Console<br/>terminal / docker logs"]
    Serilog -->|"compact JSON, daily"| File["Rolling file<br/>%TEMP%/paynexa/logs"]
    Serilog -->|"HTTP batches :5341"| Seq[("Seq<br/>infrastructure-seq")]
    OTel["OpenTelemetry tracing"] -->|"OTLP :5341"| Seq
    CustomerProcess -.-> OTel

    Developer["Developer / support"] -->|"browser :5342"| Seq

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    classDef store fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class Client,Gateway,Customer,Pipeline,Mediator,Handler,Stores,Serilog,Console,File,OTel,Developer box
    class Seq store

Four ideas carry the whole design:

Idea What it means
Structured, not text Every value (Operation, DurationMs, StatusCode, CustomerId, …) is a searchable property, not just words in a sentence.
Lifecycle everywhere Every unit of work logs started → succeeded/failed (with duration) → ended: requests, commands, validation, steps, SQL, MongoDB, Redis, Kafka, outbox.
One correlation id Created or accepted at the edge, attached to every log line, trace, outbox row and Kafka header.
Safe by default Secrets are redacted and personal data is masked by one shared rule set before a log is written anywhere.

2. Where Serilog runs and where logs go

Serilog is not a server. It is a library that runs inside each service process (Customer API, Payment API, gateway, …). Each process formats its own log events and hands them to its sinks. Seq is the server: a separate container that receives events from all services and lets you search them.

flowchart LR
    subgraph Host["My machine"]
        direction TB
        subgraph Docker["Docker: compose project paynexa"]
            direction TB
            API["service-customer-api<br/>(Serilog inside)"]
            GW["service-apigateway<br/>(Serilog inside)"]
            Other["service-payment-api …<br/>(Serilog inside)"]
            SeqC[("infrastructure-seq<br/>datalust/seq<br/>volume paynexa_seq_data")]
            API -->|"http://seq:5341"| SeqC
            GW -->|"http://seq:5341"| SeqC
            Other -->|"http://seq:5341"| SeqC
        end
        Local["Customer.API run from<br/>Visual Studio / dotnet run<br/>(Serilog inside)"] -->|"http://localhost:5341"| SeqC
        Browser["Browser"] -->|"http://localhost:5342"| SeqC
        DockerLogs["docker logs service-customer-api"] -.->|"reads console"| API
        TempFile["%TEMP%\paynexa\logs\customer-service\*.json"] -.->|"written by"| Local
    end

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class API,GW,Other,Local,Browser,DockerLogs,TempFile box
Sink Format Where Why
Console Human-readable text (or compact JSON with ConsoleFormat = Json) Terminal / Visual Studio Output / docker logs Immediate feedback while developing; container log collection later
Rolling file Compact JSON (CLEF), one file per day, 100 MB max, 7 files kept %TEMP%\paynexa\logs\<service>\<service>-yyyyMMdd.json (or PayNexaLogging:FileDirectory) Durable fallback when Seq is down; can be replayed into Seq
Seq Structured events (and OTLP traces) http://localhost:5341 locally, http://seq:5341 from containers Search, filter, correlate across services, dashboards, alerts

Port 5341 is Seq's ingestion endpoint (services write there). Port 5342 is its web UI (people read there).


3. Prerequisites

Prerequisite Why How
.NET 10 SDK Builds and runs the services dotnet --version
Docker Desktop Runs Seq (and the rest of the infrastructure) docker compose up -d from the repository root
src/.env Holds SEQ_ADMIN_PASSWORD for Seq's first login Copy src/.env.example → src/.env
Seq container running Receives logs and traces docker ps --filter name=infrastructure-seq
Service:Name in appsettings.json Every event is stamped with it; startup fails fast without it e.g. "Service": { "Name": "customer-service" }
PayNexaLogging:SeqServerUrl Tells Serilog where Seq is appsettings.Development.json (local) / docker-compose.override.yml (Docker)
Seq licence Seq is free for individual use No key needed locally

Nothing else is required. Serilog, its sinks and OpenTelemetry are NuGet packages pinned in Directory.Packages.props, and all of them are free and open source (Apache-2.0).


4. Startup: how logging is switched on

Logging is live before the host exists, so a failure while building the app is still recorded.

sequenceDiagram
    autonumber
    participant P as Program.cs
    participant B as Bootstrap logger<br/>(console only)
    participant DI as AddPresentation →<br/>AddPayNexaServiceDefaults →<br/>AddPayNexaLogging
    participant S as Full Serilog logger
    participant I as InitializeInfrastructureAsync

    P->>B: Log.Logger = PayNexaLogging.CreateBootstrapLogger()
    P->>DI: builder.Services.AddPresentation(...).AddApplication().AddInfrastructure(...)
    DI->>DI: ServiceIdentity.From(config) → name, version, environment
    DI->>S: services.AddSerilog(...) configure sinks, enrichers, masking
    P->>P: builder.Build()
    Note over B,S: The bootstrap logger is "frozen" and replaced in place:<br/>from here on every log reaches console + file + Seq
    P->>P: app.UsePresentation() → UsePayNexaServiceDefaults()
    P->>I: migrations, indexes, Kafka topics, seeding
    I->>S: "Startup: Initialize … started / succeeded / ended" (own correlation id)
    P->>P: app.RunAsync()
    Note over P,S: If anything throws: Log.Fatal("… terminated unexpectedly during startup")<br/>finally: Log.CloseAndFlushAsync() so nothing is lost

The code, in src/Services/Customer/Customer.API/Program.cs:

Log.Logger = PayNexaLogging.CreateBootstrapLogger();

try
{
    var builder = WebApplication.CreateBuilder(args);
    {
        builder.Services
            .AddPresentation(builder.Configuration, builder.Environment)
            .AddApplication()
            .AddInfrastructure(builder.Configuration);
    }

    var app = builder.Build();
    {
        app.UsePresentation();
        await app.InitializeInfrastructureAsync();
        await app.RunAsync();
    }

    return 0;
}
catch (Exception exception) when (exception is not HostAbortedException)
{
    Log.Fatal(exception, "Customer service terminated unexpectedly during startup");
    return 1;
}
finally
{
    await Log.CloseAndFlushAsync();
}

What AddPayNexaLogging configures, in src/BuildingBlocks/PayNexa.Logging/PayNexaLogging.cs:

configuration
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
    .MinimumLevel.Override("Microsoft.EntityFrameworkCore", LogEventLevel.Warning)
    .ReadFrom.Configuration(appConfiguration)
    .Enrich.FromLogContext()
    .Enrich.WithProperty("ServiceName", identity.Name)
    .Enrich.WithProperty("ServiceVersion", identity.Version)
    .Enrich.WithProperty("Environment", identity.Environment)
    .Enrich.WithMachineName()
    .Enrich.WithProcessId()
    .Enrich.WithThreadId()
    .Enrich.With<SensitiveDataMaskingEnricher>();

configuration.WriteTo.Async(sink => sink.Console(outputTemplate: TextTemplate));
configuration.WriteTo.Async(sink => sink.File(new CompactJsonFormatter(), path, rollingInterval: RollingInterval.Day, ...));
configuration.WriteTo.Seq(options.SeqServerUrl);
  • Async sinks: writing a log never blocks the request thread.
  • Unhandled exceptions: RegisterProcessLevelExceptionLogging also hooks AppDomain.UnhandledException and TaskScheduler.UnobservedTaskException, so even a crash outside a request is logged and flushed.

5. The HTTP pipeline: what wraps every request

UsePayNexaServiceDefaults() puts the logging middleware first, so it wraps everything else, including the exception handler.

flowchart TB
    In(["Request in"]) --> C1
    C1["1 · CorrelationIdMiddleware<br/>accept valid X-Correlation-Id or generate one<br/>push CorrelationId + RequestId into LogContext<br/>tag the trace, echo the header on the response"]
    C1 --> C2["2 · RequestStartedLoggingMiddleware<br/>HTTP POST /api/v1/customers started"]
    C2 --> C3{"Development and<br/>HttpBodies:Enabled?"}
    C3 -->|"yes"| C4["3 · HttpBodyLoggingMiddleware<br/>request body (masked)<br/>… response body (status, masked)"]
    C3 -->|"no"| C5
    C4 --> C5["4 · Serilog RequestLoggingMiddleware<br/>HTTP POST … responded 201 in 421.02 ms<br/>+ RequestHost, ClientIp, UserId, EndpointName"]
    C5 --> C6["5 · Exception handler → Problem Details"]
    C6 --> C7["6 · Controllers → Mediator"]
    C7 --> Out(["Response out<br/>X-Correlation-Id header"])

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class C1,C2,C4,C5,C6,C7 box
Middleware File Logs
CorrelationIdMiddleware PayNexa.Logging/Correlation/CorrelationIdMiddleware.cs A Warning when a malformed id is replaced
RequestStartedLoggingMiddleware PayNexa.Logging/RequestLogging/RequestStartedLoggingMiddleware.cs HTTP {RequestMethod} {RequestPath} started
HttpBodyLoggingMiddleware PayNexa.Logging/RequestLogging/HttpBodyLoggingMiddleware.cs Masked JSON bodies (Development only)
Serilog request logging PayNexaLogging.UsePayNexaRequestLogging HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed} ms

The completion level follows the outcome: Error for 5xx or an exception, Warning for 4xx, Information for success, and Debug for /health, /openapi and /swagger, so polling does not flood the logs.


6. Inside a command: behaviors and steps

Every command and query goes through the Mediator pipeline. Logging is a pipeline behavior, so handlers never write "started" or "ended" themselves.

flowchart LR
    Ctrl["CustomersController<br/>CreateAsync(request.ToCommand())"] --> LB
    LB["LoggingBehavior<br/>Command X started<br/>… succeeded / failed / slow / cancelled<br/>Command X ended"] --> VB
    VB["ValidationBehavior<br/>X: Validation started<br/>succeeded / failed (N errors, field names only)<br/>ended"] --> DR
    DR["DomainRuleBehavior<br/>DomainException → Result"] --> H
    H["Handler<br/>logger.BeginStep('Email uniqueness check')<br/>logger.BeginStep('Persist customer to write store')"]

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class Ctrl,LB,VB,DR,H box

6.1 LoggingBehavior, the operation lifecycle

Source: src/BuildingBlocks/PayNexa.Common/Behaviors/LoggingBehavior.cs.

using var operationScope = OperationContext.Begin(OperationName);
using var logScope = logger.BeginScope(new Dictionary<string, object?>
{
    ["Operation"] = OperationName,
    ["OperationKind"] = OperationKind,
});

OperationLog.Started(logger, OperationKind, OperationName);
var startedAt = Stopwatch.GetTimestamp();
try
{
    var response = await next(message, cancellationToken);
    if (response is Result { IsFailure: true } failed) LogFailedResult(failed.Error, durationMs);
    else OperationLog.Succeeded(logger, OperationName, durationMs);
    if (durationMs >= options.Value.SlowOperationThresholdMs) OperationLog.Slow(...);
    return response;
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { OperationLog.Cancelled(...); throw; }
catch (Exception exception) { OperationLog.Faulted(...); exception.MarkAsLogged(); throw; }
finally { OperationLog.Ended(logger, OperationName); }
stateDiagram-v2
    [*] --> Started : Command X started
    Started --> Succeeded : Result success
    Started --> FailedResult : Result failure, Warning or Error by error type
    Started --> Faulted : exception, Error with stack trace
    Started --> Cancelled : client went away
    Succeeded --> Slow : duration over 500 ms
    Succeeded --> Ended
    Slow --> Ended
    FailedResult --> Ended
    Faulted --> Ended
    Cancelled --> Ended
    Ended --> [*] : X ended

6.2 Steps: named work inside a handler

logger.BeginStep(...) (PayNexa.Common/Logging/OperationStep.cs) gives any block the same lifecycle, prefixed with the current operation name.

using (var step = logger.BeginStep("Email uniqueness check"))
{
    if (await Customers.EmailExistsAsync(email, cancellationToken))
    {
        step.Failed(CustomerErrorCodes.EmailAlreadyRegistered);
        return Errors.Customer.EmailAlreadyRegistered;
    }

    step.Succeeded();
}

Produces:

RegisterCustomerCommand: Email uniqueness check started
RegisterCustomerCommand: Email uniqueness check failed in 7 ms. Reason: Customer.EmailAlreadyRegistered
RegisterCustomerCommand: Email uniqueness check ended
  • step.WithProperty("CustomerVersion", customer.Version) attaches data to the outcome line.
  • A step disposed without Succeeded() or Failed() (an exception or an early return) logs a Warning: "Step ended without an explicit outcome". A forgotten outcome can't go unnoticed.

7. Data stores, cache and broker

These are logged automatically by the building blocks, so no handler code is needed.

Component Mechanism Example
SQL Server EF Core SqlCommandLoggingInterceptor SQL Server INSERT on CustomerDb succeeded in 2.42 ms (SaveChanges)
MongoDB Driver command events → MongoCommandLogger (skips hello, ping, auth…) MongoDB update on customers succeeded in 14.88 ms (paynexa_customer)
Redis RedisCacheService steps with CacheKey, CacheHit, TtlSeconds Redis GET succeeded in 0.8 ms (CacheHit = true)
Kafka KafkaEventBus step with EventId, EventType, Topic, MessageKey, Partition, Offset Kafka publish customer.created to paynexa.customer succeeded in 23.75 ms
Domain events DomainEventDispatcher steps Handle domain event CustomerRegisteredDomainEvent succeeded in 30.22 ms

SQL text is never logged, only the operation (SELECT, INSERT, …), the database, the duration and the source (LinqQuery, SaveChanges). The outbox's own 1-second polling query is tagged BackgroundPolling and logged at Debug, so it doesn't bury real traffic.


8. After the response: outbox, projection, Kafka

The client gets its 201 as soon as SQL Server commits. The rest happens in the background OutboxProcessor, and it keeps the request's correlation id, because the id is stored with the outbox row.

sequenceDiagram
    autonumber
    participant C as Client
    participant API as Customer API
    participant SQL as SQL Server<br/>CustomerDb
    participant OP as OutboxProcessor<br/>(background, every 1 s)
    participant M as MongoDB<br/>paynexa_customer
    participant R as Redis
    participant K as Kafka<br/>paynexa.customer

    C->>API: POST /api/v1/customers (X-Correlation-Id: doc-demo-27493)
    API->>SQL: INSERT customer + INSERT OutboxMessage<br/>(CorrelationId, TraceParent) in ONE transaction
    API-->>C: 201 Created (X-Correlation-Id: doc-demo-27493)
    Note over API: logs so far: HTTP started → body → command → validation →<br/>steps → SQL → responded 201
    OP->>SQL: claim pending rows (UPDLOCK, READPAST)
    OP->>OP: restore CorrelationId + trace parent from the row
    OP->>M: Project customer to read store (version-checked upsert)
    OP->>R: Redis DEL customer:{id}:profile
    OP->>K: Kafka publish customer.created<br/>headers: correlation-id, traceparent, event-id …
    OP->>SQL: mark processed
    Note over OP: logs: Outbox.CustomerCreatedIntegrationEvent: Dispatch … started →<br/>Project … → Redis DEL … → Kafka publish … → succeeded / ended<br/>all with CorrelationId = doc-demo-27493

A failure here is visible too. A failed dispatch logs "failed on attempt N; retrying in S s". After Outbox:MaxAttempts (20) the row is dead-lettered and logged at Critical: "needs manual attention".


9. Correlation: how one id follows the whole journey

flowchart LR
    Client["Client<br/>may send X-Correlation-Id"] -->|"header"| GW["Gateway<br/>accept or generate<br/>forward header"]
    GW -->|"X-Correlation-Id"| API["Customer API<br/>LogContext.CorrelationId<br/>trace tag correlation.id"]
    API -->|"column CorrelationId<br/>+ TraceParent"| OB[("Outbox row")]
    OB -->|"restored by processor"| OP["OutboxProcessor logs"]
    OP -->|"Kafka header correlation-id<br/>+ traceparent"| K{{"Kafka"}}
    K -->|"consumer restores id<br/>(future consumers)"| N["Notification / Transaction"]
    API -->|"typed service client<br/>forwards X-Correlation-Id"| P["Other service"]
    API -->|"response header +<br/>Problem Details correlationId"| Client

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class Client,GW,API,OP,N,P box

Rules, from CorrelationContext and CorrelationIdMiddleware:

  • Accepting an id: a client id is accepted if it is 1–64 characters of letters, digits, -, _, . or :. Anything else is replaced, with a Warning.
  • New ids are UUID v7 without dashes (01a0dcb81b387b88…). They sort by time.
  • Where it appears: every log line, every trace (correlation.id), every error body ("correlationId": "…") and the response header.

How support uses it: a customer reports "it failed, reference 01a0dc9e…". Paste that into Seq, and the full story appears across every service.


10. Service-to-service calls, retries and outages

When a service calls another (for example Payment → Customer payment-eligibility), the typed client from AddServiceClient<…> logs the call and every resilience event.

sequenceDiagram
    autonumber
    participant Pay as payment-service
    participant H as ServiceCallLoggingHandler<br/>+ resilience (retry, timeout, circuit)
    participant Cus as customer-service

    Pay->>H: GetPaymentEligibility(customerId)
    H->>H: "payment-service -> customer-service GetPaymentEligibility started"
    H->>Cus: GET /api/v1/customers/{id}/payment-eligibility (X-Correlation-Id)
    Cus--xH: 503 / timeout
    H->>H: Warning "attempt 1 failed with 503, retrying in 400 ms"
    H->>Cus: retry
    Cus--xH: 503
    H->>H: Error "Circuit OPENED for customer-service … considered down"
    H-->>Pay: Error "failed: … customer-service is unavailable after N attempts"
    Note over H: later: "Circuit HALF-OPEN … probing" → "Circuit CLOSED … calls flow normally again"
  • Unsafe methods are never retried automatically: POST and PUT go out once, and duplicates are prevented with idempotency keys.
  • Metrics: paynexa.service_call.retries and paynexa.service_call.circuit_opened are OpenTelemetry metrics, ready for dashboards.

11. Errors and exceptions

flowchart TB
    E{"What went wrong?"}
    E -->|"invalid input"| V["ValidationBehavior: Validation failed (N errors)<br/>LoggingBehavior: failed with Validation.Failed<br/>Warning · HTTP 400"]
    E -->|"business rule<br/>(duplicate email, closed customer)"| B["Step failed · Reason: Customer.EmailAlreadyRegistered<br/>LoggingBehavior: failed with … (Conflict)<br/>Warning · HTTP 409 / 404 / 422"]
    E -->|"unexpected exception"| X["LoggingBehavior: failed with unhandled SqlException<br/>Error + stack trace (logged once)<br/>GlobalExceptionHandler: Previously logged … mapped to HTTP 503"]
    E -->|"client disconnected"| A["HTTP … aborted by the client<br/>Information · 499"]
    E -->|"process crash"| F["Fatal · flushed before exit"]

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class V,B,X,A,F box

No duplicate stack traces. LoggingBehavior logs the exception with its stack and calls exception.MarkAsLogged(). GlobalExceptionHandler then logs only a one-line mapping ("Previously logged SqlException mapped to HTTP 503") instead of the same stack twice.


12. Anatomy of a log event

What Seq stores for one line (for example "RegisterCustomerCommand succeeded in 342.62 ms"):

classDiagram
    class LogEvent {
        Timestamp 2026-09-26T07:57:28.521Z
        Level Information
        MessageTemplate Operation succeeded in DurationMs ms
        EventId 1001
        SourceContext PayNexa.Common.Behaviors.LoggingBehavior
    }
    class Service {
        ServiceName customer-service
        ServiceVersion 1.0.0
        Environment Development
        MachineName, ProcessId, ThreadId
    }
    class Correlation {
        CorrelationId doc-demo-27493
        RequestId
        TraceId, SpanId
    }
    class Operation {
        Operation RegisterCustomerCommand
        OperationKind Command
        DurationMs 342.62
    }
    class Http {
        RequestMethod POST
        RequestPath /api/v1/customers
        StatusCode, ClientIp, UserId, EndpointName
    }
    LogEvent --> Service : enrichers
    LogEvent --> Correlation : LogContext + Activity
    LogEvent --> Operation : scope + template
    LogEvent --> Http : request logging
Property Source
ServiceName, ServiceVersion, Environment ServiceIdentity (config Service:Name, assembly version)
MachineName, ProcessId, ThreadId Serilog enrichers
CorrelationId, RequestId CorrelationIdMiddleware → LogContext
TraceId, SpanId Current OpenTelemetry Activity
Operation, OperationKind, Step LoggingBehavior / OperationStep scopes
DurationMs, ErrorCode, ErrorType, CacheHit, Topic, Offset, … The message template and WithProperty(...)

Every message is declared once with [LoggerMessage] source generation. That makes it fast (no boxing and no parsing at runtime), strongly typed, and consistent: the same template always yields the same property names.


13. Masking personal data and secrets

One rule set, SensitiveFields, is applied in two places:

  • SensitiveDataMaskingEnricher masks every structured property of every event, including nested objects and arrays.
  • JsonBodyMasker masks every field of a logged request or response body.
flowchart TB
    N["Field / property name"] --> S{"contains password, secret, token,<br/>apikey, authorization, credential,<br/>cardnumber, cvv, cvc,<br/>connectionstring, privatekey?"}
    S -->|"yes"| R["***REDACTED***<br/>(whole value, even objects and numbers)"]
    S -->|"no"| Em{"contains email?"}
    Em -->|"yes"| EM["r***@example.com"]
    Em -->|"no"| Ph{"contains phone?"}
    Ph -->|"yes"| PH["*********1234"]
    Ph -->|"no"| Pe{"firstName, lastName, fullName,<br/>dateOfBirth, line1, line2, postalCode?"}
    Pe -->|"yes"| PE["S***"]
    Pe -->|"no"| K["kept as is<br/>(ids, status, city, country, dates…)"]

    classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
    class R,EM,PH,PE,K box
request body: {"firstName":"N***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********3000",
               "dateOfBirth":"1***","address":{"line1":"G***","line2":null,"city":"Dhaka","postalCode":"1***","countryCode":"BD"}}

Body logging is Development only: it needs the environment to be Development and PayNexaLogging:HttpBodies:Enabled = true. Bodies longer than 4096 characters are truncated. Non-JSON bodies and health, OpenAPI and Swagger calls are not logged.


14. Levels, event ids and noise control

Level Used for
Debug Health, OpenAPI and Swagger requests; outbox polling queries
Information Lifecycle (started, succeeded, ended), request completion, business events, bodies
Warning Expected failures (validation, not found, conflict), slow operations, retries, rejected correlation ids
Error Unexpected exceptions, 5xx, downstream unavailable, circuit opened
Critical / Fatal Dead-lettered outbox messages, process crash
Event id range Area
1000–1006 Operation lifecycle (LoggingBehavior)
1100–1104 Steps (OperationStep)
2000–2010 HTTP: request started (2000), request body (2001), response body (2002), rejected correlation id (2010)
3000+ Service-to-service calls and resilience
4000 / 4100 / 4200 / 4300 MongoDB / Redis / SQL Server and outbox / Kafka
5000–5002 Exceptions and aborted requests
10001–10003 Customer business events: registered, changed, seeded

Framework noise is trimmed by overrides: Microsoft.AspNetCore, Microsoft.EntityFrameworkCore, System.Net.Http.HttpClient and Polly log at Warning or above. Microsoft.Hosting.Lifetime stays at Information, which keeps "Now listening on …". For a deep dive, set Serilog:MinimumLevel:Default to Debug in appsettings.Development.json.


15. Reading the logs: console, file, Seq

15.1 Console

[07:57:28.521 INF] customer-service doc-demo-27493 PayNexa.Common.Behaviors.LoggingBehavior
    RegisterCustomerCommand succeeded in 342.62 ms

Format: [time level] service correlationId source, then the message indented on the next line.

  • Visual Studio / dotnet run: the terminal or the Output window.
  • Docker: docker logs -f service-customer-api.

15.2 Log file

Get-Content "$env:TEMP\paynexa\logs\customer-service\customer-service-$(Get-Date -Format yyyyMMdd).json" -Wait -Tail 20

15.3 Seq

Open http://localhost:5342 and sign in as admin with SEQ_ADMIN_PASSWORD from src/.env.

Question Seq query
Everything for one request CorrelationId = 'doc-demo-27493'
Everything for one trace id (from an error response) @TraceId = '930f34d0f6df392c8f6b90b604f25b61'
Only the Customer service ServiceName = 'customer-service'
All errors today @Level = 'Error' (then choose the time range)
Failed commands by error code Operation like '%Command' and ErrorCode is not null
Slow operations DurationMs > 500
Duplicate-email attempts ErrorCode = 'Customer.EmailAlreadyRegistered'
Request / response bodies EventId.Id in [2001, 2002]
Kafka publishes Topic = 'paynexa.customer'
A downstream outage TargetService = 'customer-service' and @Level = 'Error'

The lines in Seq with a duration on the right (for example GET api/v{version:apiVersion}/customers 10 ms, outbox process CustomerCreatedIntegrationEvent 17 ms, paynexa.customer publish 11 ms) are OpenTelemetry spans, exported to Seq's OTLP endpoint. Click one to see the trace tree, and use Trace on any log line to jump between logs and spans.


16. What this gives the business and the code

16.1 Business

Benefit How the logging delivers it
Faster incident resolution The correlation id in every error response lets support find the full story in one Seq search, instead of guessing across machines.
Outage detection "Circuit OPENED for customer-service … considered down" and dead-letter Critical logs are ready-made alert triggers.
Performance and SLA evidence Every request, command, query and store call carries DurationMs; slow operations raise Warnings automatically.
Compliance and privacy PII is masked and secrets redacted before any sink (GDPR data minimisation; PCI DSS forbids card data in logs), and there is still a complete audit trail of who did what, when, with what outcome.
Fraud and abuse signals Repeated Customer.EmailAlreadyRegistered, validation storms or KYC rejections are queryable by ErrorCode, ClientIp and UserId.
Product insight Business events (Customer … registered, KYC changes, payment-eligibility checks) can be counted over time without a separate analytics pipeline.
Money safety The outbox, dispatch and dead-letter logs show whether an event such as a payment completion actually left the system. Nothing is lost silently.

16.2 Code and engineering

Benefit How
Zero logging boilerplate in handlers Lifecycle, validation, SQL, MongoDB, Redis, Kafka, outbox and HTTP are logged by building blocks; handlers only name meaningful steps.
Consistency across services The same templates, properties and levels everywhere; a new service gets it all through AddPayNexaServiceDefaults().
Performance [LoggerMessage] source generation, async sinks, and Debug level for noisy polling.
Debuggability Development-only masked request/response bodies, explicit step outcomes, and a Warning for steps that end without one.
Testability Logging is plain ILogger; unit tests assert on it with FakeLogger.
Vendor freedom Serilog sinks and OTLP are standards; Seq can be swapped for Elasticsearch, Grafana Loki or a cloud backend by changing configuration, not code.

17. Configuration reference

appsettings.json (all environments):

"Service": { "Name": "customer-service" },
"OperationLogging": { "SlowOperationThresholdMs": 500 },
"PayNexaLogging": {
    "ConsoleFormat": "Text",
    "SeqServerUrl": "",
    "RetainedFileCount": 7,
    "HttpBodies": {
      "Enabled": false,
      "MaxLoggedLength": 4096,
      "MaxCapturedBytes": 1048576
    }
  },
  "Observability": {
    "OtlpTracesEndpoint": null,
    "OtlpMetricsEndpoint": null
  },
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft.AspNetCore": "Warning",
        "Microsoft.EntityFrameworkCore": "Warning"
      }
    }
  },

appsettings.Development.json (local runs):

"PayNexaLogging": { "SeqServerUrl": "http://localhost:5341", "HttpBodies": { "Enabled": true } },
"Observability": { "OtlpTracesEndpoint": "http://localhost:5341/ingest/otlp/v1/traces" }

docker-compose.override.yml (containers):

PayNexaLogging__SeqServerUrl: http://seq:5341
PayNexaLogging__ConsoleFormat: Text
PayNexaLogging__HttpBodies__Enabled: "true"
Observability__OtlpTracesEndpoint: http://seq:5341/ingest/otlp/v1/traces
Key Default Purpose
Service:Name required Name stamped on every event and trace
PayNexaLogging:SeqServerUrl empty (Seq off) Seq ingestion URL
PayNexaLogging:SeqApiKey empty Seq API key when Seq requires one
PayNexaLogging:ConsoleFormat Text Text or Json
PayNexaLogging:FileDirectory %TEMP%/paynexa/logs/<service> Rolling file location
PayNexaLogging:RetainedFileCount 7 Days of files kept
PayNexaLogging:HttpBodies:* off Development-only body logging
OperationLogging:SlowOperationThresholdMs 500 "is slow" Warning threshold
Observability:OtlpTracesEndpoint / OtlpMetricsEndpoint off Where traces and metrics are exported
Serilog:MinimumLevel:* Information Standard Serilog level switches

18. Known limitations

  • Validation messages are masked in logged 400 bodies. Problem Details puts messages under field names ("errors": { "email": [...] }), and the name rule masks them (***REDACTED***). The client still receives the real messages; only the log copy is masked. The Validation step still logs which fields failed (InvalidFields).
  • Seq outage. Seq outages lose nothing. The Seq sink buffers in memory and retries, and the rolling file keeps every event regardless.
  • Kafka consumers will restore the correlation-id header when the first consuming service (Notification or Transaction) is built.

19. Real console output, end to end

The three traces below were captured from docker logs service-customer-api. Each one is filtered by its correlation id, and the [time level] service correlationId source prefix is shown only on the first trace.

19.1 Successful registration: 201 Created

[14:52:44.821 INF] customer-service postman-1790412764065-33571 PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB find on customers succeeded in 0.76 ms (paynexa_customer)
[14:52:44.822 INF] customer-service postman-1790412764065-33571 PayNexa.Common.Behaviors.LoggingBehavior
    ListCustomersQuery succeeded in 3.64 ms
[14:52:44.822 INF] customer-service postman-1790412764065-33571 PayNexa.Common.Behaviors.LoggingBehavior
    ListCustomersQuery ended
[14:52:44.822 INF] customer-service postman-1790412764065-33571 Serilog.AspNetCore.RequestLoggingMiddleware
    HTTP GET /api/v1/customers responded 200 in 10.07 ms
[14:52:44.822 INF] customer-service postman-1790412764065-33571 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
    HTTP GET /api/v1/customers response body (200): {"items":[{"id":"01a0dce9-cbcd-71c3-be12-275da823c84a","firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********8259","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:51:44.717Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:51:44.717Z","updatedBy":"anonymous"},{"id":"01a0dce9-7792-7b94-956a-e893b53ac7e0","firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********2821","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:51:23.155Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:51:23.155Z","updatedBy":"anonymous"},{"id":"01a0dce4-56fd-7e28-a31f-eff26f632c2e","firstName":"N***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********0999","dateOfBirth":"1***","address":{"line1":"B***","line2":"F***","city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Closed","statusReason":"Customer requested account closure","kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:45:47.337Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:46:11.616Z","updatedBy":"anonymous"},{"id":"189dc8dc-990f-48e0-a37b-e6f2b60b9d7d","firstName":"T***","lastName":"C***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"S***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:46:06.84Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:46:06.84Z","updatedBy":"system"},{"id":"58c49479-ec65-4de2-86e7-033c546291aa","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"M***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:43:20.347Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:43:20.347Z","updatedBy":"system"},{"id":"01a0d92c-1476-7bf2-a97e-c02723b33032","firstName":"N***","lastName":"R***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"B***","line2":"F***","city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Closed","statusReason":"Customer request","kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-25T15:25:39.992Z","createdBy":"anonymous","updatedAtUtc":"2026-09-25T15:25:41.559Z","updatedBy":"anonymous"},{"id":"01a0d775-8b75-72de-bcab-3c690216124b","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":null,"status":"Active","statusReason":null,"kycStatus":null,"kycRejectionReason":null,"createdAtUtc":"2026-09-25T07:26:39.989Z","createdBy":null,"updatedAtUtc":"2026-09-25T07:27:40.173Z","updatedBy":null}],"page":1,"pageSize":20,"totalCount":7,"totalPages":1,"hasPreviousPage":false,"hasNextPage":false}
[14:54:12.322 INF] customer-service postman-1790412850987-45508 PayNexa.Logging.RequestLogging.RequestStartedLoggingMiddleware
    HTTP POST /api/v1/customers started
[14:54:12.322 INF] customer-service postman-1790412850987-45508 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
    HTTP POST /api/v1/customers request body: {"firstName":"M***","lastName":"H***","email":"m***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"}}
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.LoggingBehavior
    Command RegisterCustomerCommand started
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.ValidationBehavior
    RegisterCustomerCommand: Validation started
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.ValidationBehavior
    RegisterCustomerCommand: Validation succeeded in 0.16 ms
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.ValidationBehavior
    RegisterCustomerCommand: Validation ended
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    RegisterCustomerCommand: Email uniqueness check started
[14:54:12.324 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
    SQL Server SELECT on CustomerDb started (LinqQuery)
[14:54:12.327 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
    SQL Server SELECT on CustomerDb succeeded in 3.43 ms (LinqQuery)
[14:54:12.327 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    RegisterCustomerCommand: Email uniqueness check succeeded in 4.55 ms
[14:54:12.327 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    RegisterCustomerCommand: Email uniqueness check ended
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    RegisterCustomerCommand: Persist customer to write store started
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Common.DomainEvents.DomainEventDispatcher
    RegisterCustomerCommand: Handle domain event CustomerRegisteredDomainEvent started
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Common.DomainEvents.DomainEventDispatcher
    RegisterCustomerCommand: Handle domain event CustomerRegisteredDomainEvent succeeded in 0.11 ms
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Common.DomainEvents.DomainEventDispatcher
    RegisterCustomerCommand: Handle domain event CustomerRegisteredDomainEvent ended
[14:54:12.329 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
    SQL Server INSERT on CustomerDb started (SaveChanges)
[14:54:12.332 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
    SQL Server INSERT on CustomerDb succeeded in 3.58 ms (SaveChanges)
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    RegisterCustomerCommand: Persist customer to write store succeeded in 5.41 ms
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    RegisterCustomerCommand: Persist customer to write store ended
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
    Customer 01a0dcec-0c67-774c-aee8-a3cc9951822f registered
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.LoggingBehavior
    RegisterCustomerCommand succeeded in 10.69 ms
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.LoggingBehavior
    RegisterCustomerCommand ended
[14:54:12.334 INF] customer-service postman-1790412850987-45508 Serilog.AspNetCore.RequestLoggingMiddleware
    HTTP POST /api/v1/customers responded 201 in 12.07 ms
[14:54:12.334 INF] customer-service postman-1790412850987-45508 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
    HTTP POST /api/v1/customers response body (201): {"id":"01a0dcec-0c67-774c-aee8-a3cc9951822f","firstName":"M***","lastName":"H***","email":"m***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:54:12.3282326Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:54:12.3282326Z","updatedBy":"anonymous"}
[14:54:12.848 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Outbox.OutboxProcessor
    Outbox.CustomerCreatedIntegrationEvent: Dispatch CustomerCreatedIntegrationEvent started
[14:54:12.849 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Infrastructure.MongoDB.CustomerProjectionHandler
    Outbox.CustomerCreatedIntegrationEvent: Project customer to read store started
[14:54:12.849 INF] customer-service postman-1790412850987-45508 PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB update on customers started (paynexa_customer)
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB update on customers succeeded in 2.57 ms (paynexa_customer)
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Infrastructure.MongoDB.CustomerProjectionHandler
    Outbox.CustomerCreatedIntegrationEvent: Project customer to read store succeeded in 3.53 ms
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Infrastructure.MongoDB.CustomerProjectionHandler
    Outbox.CustomerCreatedIntegrationEvent: Project customer to read store ended
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.Caching.RedisCacheService
    Outbox.CustomerCreatedIntegrationEvent: Redis DEL started
[14:54:12.854 INF] customer-service postman-1790412850987-45508 PayNexa.Caching.RedisCacheService
    Outbox.CustomerCreatedIntegrationEvent: Redis DEL succeeded in 1.44 ms
[14:54:12.854 INF] customer-service postman-1790412850987-45508 PayNexa.Caching.RedisCacheService
    Outbox.CustomerCreatedIntegrationEvent: Redis DEL ended
[14:54:12.854 INF] customer-service postman-1790412850987-45508 PayNexa.Messaging.Kafka.KafkaEventBus
    Outbox.CustomerCreatedIntegrationEvent: Kafka publish customer.created to paynexa.customer started
[14:54:12.867 INF] customer-service postman-1790412850987-45508 PayNexa.Messaging.Kafka.KafkaEventBus
    Outbox.CustomerCreatedIntegrationEvent: Kafka publish customer.created to paynexa.customer succeeded in 12.89 ms

How to read it:

  1. request body is masked.
  2. The command starts, validation passes, and the email check runs one SELECT.
  3. Persisting raises the domain event, which queues the integration event in the outbox. SQL Server then INSERTs the customer and the outbox row in one transaction.
  4. The command succeeds in 342 ms and the client gets 201 after 421 ms.
  5. About 80 ms later, the outbox processor projects the customer to MongoDB, clears the Redis cache entry and publishes customer.created to Kafka. All of it still carries doc-demo-27493.

19.2 Invalid input: 400 Bad Request

    HTTP POST /api/v1/customers started
[14:59:02.751 INF] customer-service postman-1790413141501-51236 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
    HTTP POST /api/v1/customers request body: {"firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********6701","dateOfBirth":"2***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"}}
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.LoggingBehavior
    Command RegisterCustomerCommand started
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.ValidationBehavior
    RegisterCustomerCommand: Validation started
[14:59:02.754 WRN] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.ValidationBehavior
    RegisterCustomerCommand: Validation failed in 0.4 ms. Reason: 1 validation error(s)
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.ValidationBehavior
    RegisterCustomerCommand: Validation ended
[14:59:02.754 WRN] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.LoggingBehavior
    RegisterCustomerCommand failed in 0.65 ms with Validation.Failed (Validation)
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.LoggingBehavior
    RegisterCustomerCommand ended
[14:59:02.755 WRN] customer-service postman-1790413141501-51236 Serilog.AspNetCore.RequestLoggingMiddleware
    HTTP POST /api/v1/customers responded 400 in 3.74 ms
[14:59:02.755 INF] customer-service postman-1790413141501-51236 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
    HTTP POST /api/v1/customers response body (400): {"type":"https://tools.ietf.org/html/rfc9110#section-15.5.1","title":"Validation failed","status":400,"detail":"One or more validation errors occurred.","instance":"/api/v1/customers","errors":{"dateOfBirth":["\u0027***"]},"traceId":"930f34d0f6df392c8f6b90b604f25b61","correlationId":"postman-1790413141501-51236","errorCode":"Validation.Failed"}

No database is touched: validation stops the request before the handler runs.

19.3 Business rule: duplicate email → 409 Conflict

    HTTP POST /api/v1/customers started
    HTTP POST /api/v1/customers request body: {"firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***"}
    Command RegisterCustomerCommand started
    RegisterCustomerCommand: Validation started
    RegisterCustomerCommand: Validation succeeded in 0.33 ms
    RegisterCustomerCommand: Validation ended
    RegisterCustomerCommand: Email uniqueness check started
    SQL Server SELECT on CustomerDb started (LinqQuery)
    SQL Server SELECT on CustomerDb succeeded in 1.51 ms (LinqQuery)
    RegisterCustomerCommand: Email uniqueness check failed in 7 ms. Reason: Customer.EmailAlreadyRegistered
    RegisterCustomerCommand: Email uniqueness check ended
    RegisterCustomerCommand failed in 8.78 ms with Customer.EmailAlreadyRegistered (Conflict)
    RegisterCustomerCommand ended
    HTTP POST /api/v1/customers responded 409 in 25.77 ms
    HTTP POST /api/v1/customers response body (409): {"type":"https://tools.ietf.org/html/rfc9110#section-15.5.10","title":"Request conflicts with the current state","status":409,"detail":"A customer with this email address already exists.",...}

The exact step that failed and why (Customer.EmailAlreadyRegistered) is on record. No stack trace is logged, because it is an expected business outcome, not a bug.

19.4 Successful data fetch → response body (200)

[14:35:11.404 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.ValidationBehavior
    ListCustomersQuery: Validation succeeded in 16.23 ms
[14:35:11.404 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.ValidationBehavior
    ListCustomersQuery: Validation ended
[14:35:11.435 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB aggregate on customers started (paynexa_customer)
[14:35:11.442 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB aggregate on customers succeeded in 7.26 ms (paynexa_customer)
[14:35:11.489 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB find on customers started (paynexa_customer)
[14:35:11.490 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
    MongoDB find on customers succeeded in 1.94 ms (paynexa_customer)
[14:35:11.533 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.LoggingBehavior
    ListCustomersQuery succeeded in 145.4 ms
[14:35:11.536 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.LoggingBehavior
    ListCustomersQuery ended
[14:35:11.568 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d Serilog.AspNetCore.RequestLoggingMiddleware
    HTTP GET /api/v1/customers responded 200 in 272.19 ms
[14:35:11.579 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
    HTTP GET /api/v1/customers response body (200): {"items":[{"id":"189dc8dc-990f-48e0-a37b-e6f2b60b9d7d","firstName":"T***","lastName":"C***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"S***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:46:06.84Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:46:06.84Z","updatedBy":"system"},{"id":"58c49479-ec65-4de2-86e7-033c546291aa","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"M***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:43:20.347Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:43:20.347Z","updatedBy":"system"},{"id":"01a0d92c-1476-7bf2-a97e-c02723b33032","firstName":"N***","lastName":"R***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"B***","line2":"F***","city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Closed","statusReason":"Customer request","kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-25T15:25:39.992Z","createdBy":"anonymous","updatedAtUtc":"2026-09-25T15:25:41.559Z","updatedBy":"anonymous"},{"id":"01a0d775-8b75-72de-bcab-3c690216124b","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":null,"status":"Active","statusReason":null,"kycStatus":null,"kycRejectionReason":null,"createdAtUtc":"2026-09-25T07:26:39.989Z","createdBy":null,"updatedAtUtc":"2026-09-25T07:27:40.173Z","updatedBy":null}],"page":1,"pageSize":20,"totalCount":4,"totalPages":1,"hasPreviousPage":false,"hasNextPage":false}

Note: Sensitive data are masked, and it is completely configurable whether we want to display http request/response body in the logger/console or not


20. Seeing it in action: screenshots

These screenshots come from a real session: Customer.API running locally, the rest of the stack in Docker, requests sent from the Postman collection in postman/.

20.1 The running stack in Docker Desktop

Docker Desktop showing the paynexa stack

Everything runs as one Docker Compose project, paynexa, with container names grouped by prefix:

  • database-*: SQL Server (1433), MongoDB (27017) and Redis (6379).
  • infrastructure-*:
    • infrastructure-seq receives logs on 5341 and serves the UI on 5342.
    • infrastructure-kafka is the event broker on 9092.
    • infrastructure-kafka-ui is the Kafka UI on 8090.
  • service-*: the API gateway and the five APIs. service-customer-api is stopped here because Customer.API was running from Visual Studio on the same ports (5001 / 6001). The local process still sends its logs to the same Seq container.

20.2 Seq: masked request and response bodies

Seq showing masked request and response bodies

Seq lists events newest first, so read each request from the bottom up:

  1. Now listening on …, Outbox processor … started, then Startup: Seed Customers … succeeded: the service started, and the lines above them are its first requests.
  2. HTTP POST /api/v1/customers started: the request arrives.
  3. request body: {"firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********6701","dateOfBirth":"2***",...}: the body is logged with every personal field masked.
  4. Command RegisterCustomerCommand started, then Validation started, then Validation failed … Reason: 1 validation error(s): the lifecycle from LoggingBehavior and ValidationBehavior. The orange dots mark Warning-level lines.
  5. RegisterCustomerCommand failed … with Validation.Failed (Validation), then ended.
  6. responded 400 in 3.74 ms and response body (400): {"type":…,"title":"Validation failed",…}: the outcome and the Problem Details that went back to the client.
  7. POST api/v{version:apiVersion}/customers 5 ms with a duration on the right: the OpenTelemetry span for the same request, stored in Seq next to the logs.

The two outlined blocks are two separate attempts. Each has its own correlation id, so filtering on one shows only that request.

20.3 Postman: a bad request with traceId and correlationId

Postman showing a 400 response with traceId and correlationId

The collection's Register customer request was sent with a date of birth that makes the customer under 18. The API answered 400 Bad Request with Problem Details:

  • errors.dateOfBirth: the exact rule that failed ("must make the customer at least 18 years old").
  • errorCode: Validation.Failed, a stable code clients can branch on.
  • correlationId: postman-1790413141501-51236, the value Postman sent in X-Correlation-Id and the API echoed back.
  • traceId: 930f34d0f6df392c8f6b90b604f25b61, the W3C trace id of this request.

Both ids can be searched in Seq and lead to the same request:

Id from the response Seq query Finds
correlationId CorrelationId = 'postman-1790413141501-51236' Every log line of the request, plus background work it started (outbox, Kafka), even across services
traceId @TraceId = '930f34d0f6df392c8f6b90b604f25b61' Every log line and the OpenTelemetry spans of the request, shown as a trace tree

The trace id is stored on every event ("@tr":"930f34d0f6df392c8f6b90b604f25b61" in the log file; ten events for this request). Seq exposes it as the built-in @TraceId property. Clicking any event in Seq and choosing Trace does the same search for you.

The correlation id is the business-friendly reference to give support. The trace id is the technical reference for following timing across services and spans.

20.4 Postman: a successful query

Postman showing a successful List customers response

List customers uses only environment variables: {{baseUrl}}/api/{{apiVersion}}/customers?page={{page}}&page-size={{pageSize}}. The optional filters (search, status, kyc-status) are unticked until needed. The response is 200 OK with a paged result read from the MongoDB read model, and the Test Results 3/3 show the collection's checks passed: status code, correlation id echoed, and a paged body.

The client receives the real data (names, emails, phone numbers). Masking applies only to what is written to the logs.

20.5 Console: the same response, masked in the log

Console showing the masked response body

This is the Customer.API console for that same request (postman-1790414119486-19708):

  • ListCustomersQuery succeeded in 129.35 ms, then ended.
  • HTTP GET /api/v1/customers responded 200 in 187.12 ms.
  • response body (200): {"items":[{"id":"01a0dcec-…","firstName":"M***","lastName":"H***","email":"m***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"K***",…,"city":"Dhaka",…,"postalCode":"1***","countryCode":"BD"},"status":"Active",…}

Compare it with the Postman response in 20.4:

Field Postman (client) Console / Seq (log)
firstName Mahbuba Sultana M***
email mahbuba.sultana.himu@gmail.com m***@gmail.com
phoneNumber +8801300000000 *********0000
address.line1 Kazipara, Dhaka-1207, Bangladesh K***
city, countryCode, status, ids, timestamps unchanged unchanged

The log keeps enough to debug (ids, statuses, cities, the last four phone digits, the email domain) without storing anyone's identity in plain text.

No comments yet.

Sign in to leave a comment.