Logging and Observability
One request, one correlation id, every step on record.** PayNexa writes structured logs with Serilog from every service. Each log goes to three places at once: the console, a rolling JSON file, and Seq, a searchable log server running in Docker. OpenTelemetry traces go to Seq as well. Every line from the first byte of a request to the last Kafka message it causes carries the same `CorrelationId`: HTTP, validation, SQL Server, MongoDB, Redis, the outbox, Kafka and calls to other services. Personal data and secrets are masked before anything is written. This document uses the **Customer API** as the worked example. Every other PayNexa service gets the same behavior from the shared building blocks with no extra code.
Contents
- Github Open Source
- The big picture
- Where Serilog runs and where logs go
- Prerequisites
- Startup: how logging is switched on
- The HTTP pipeline: what wraps every request
- Inside a command: behaviors and steps
- Data stores, cache and broker
- After the response: outbox, projection, Kafka
- Correlation: how one id follows the whole journey
- Service-to-service calls, retries and outages
- Errors and exceptions
- Anatomy of a log event
- Masking personal data and secrets
- Levels, event ids and noise control
- Reading the logs: console, file, Seq
- What this gives the business and the code
- Configuration reference
- Known limitations
- Real console output, end to end
- Seeing it in action: screenshots
0. GitHub Open Source
| Item | Details |
|---|---|
| Repository | PayNexa |
| Branch | feature/logging_and_observability |
1. The big picture
flowchart TB
Client["Client / Swagger"] -->|"HTTPS + X-Correlation-Id"| Gateway["YARP API Gateway<br/>service-apigateway"]
Gateway -->|"forwards X-Correlation-Id"| Customer["Customer API<br/>service-customer-api"]
subgraph CustomerProcess["Customer API process"]
direction TB
Pipeline["HTTP pipeline<br/>correlation · request logs · body logs"]
Mediator["Mediator pipeline<br/>LoggingBehavior · ValidationBehavior"]
Handler["Command / query handler<br/>logger.BeginStep(...)"]
Stores["SQL · MongoDB · Redis · Kafka<br/>interceptors and steps"]
Serilog["Serilog logger<br/>enrichers + masking"]
Pipeline --> Mediator --> Handler --> Stores
Pipeline -.-> Serilog
Mediator -.-> Serilog
Handler -.-> Serilog
Stores -.-> Serilog
end
Customer --- CustomerProcess
Serilog -->|"text"| Console["Console<br/>terminal / docker logs"]
Serilog -->|"compact JSON, daily"| File["Rolling file<br/>%TEMP%/paynexa/logs"]
Serilog -->|"HTTP batches :5341"| Seq[("Seq<br/>infrastructure-seq")]
OTel["OpenTelemetry tracing"] -->|"OTLP :5341"| Seq
CustomerProcess -.-> OTel
Developer["Developer / support"] -->|"browser :5342"| Seq
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
classDef store fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class Client,Gateway,Customer,Pipeline,Mediator,Handler,Stores,Serilog,Console,File,OTel,Developer box
class Seq store
Four ideas carry the whole design:
| Idea | What it means |
|---|---|
| Structured, not text | Every value (Operation, DurationMs, StatusCode, CustomerId, …) is a searchable property, not just words in a sentence. |
| Lifecycle everywhere | Every unit of work logs started → succeeded/failed (with duration) → ended: requests, commands, validation, steps, SQL, MongoDB, Redis, Kafka, outbox. |
| One correlation id | Created or accepted at the edge, attached to every log line, trace, outbox row and Kafka header. |
| Safe by default | Secrets are redacted and personal data is masked by one shared rule set before a log is written anywhere. |
2. Where Serilog runs and where logs go
Serilog is not a server. It is a library that runs inside each service process (Customer API, Payment API, gateway, …). Each process formats its own log events and hands them to its sinks. Seq is the server: a separate container that receives events from all services and lets you search them.
flowchart LR
subgraph Host["My machine"]
direction TB
subgraph Docker["Docker: compose project paynexa"]
direction TB
API["service-customer-api<br/>(Serilog inside)"]
GW["service-apigateway<br/>(Serilog inside)"]
Other["service-payment-api …<br/>(Serilog inside)"]
SeqC[("infrastructure-seq<br/>datalust/seq<br/>volume paynexa_seq_data")]
API -->|"http://seq:5341"| SeqC
GW -->|"http://seq:5341"| SeqC
Other -->|"http://seq:5341"| SeqC
end
Local["Customer.API run from<br/>Visual Studio / dotnet run<br/>(Serilog inside)"] -->|"http://localhost:5341"| SeqC
Browser["Browser"] -->|"http://localhost:5342"| SeqC
DockerLogs["docker logs service-customer-api"] -.->|"reads console"| API
TempFile["%TEMP%\paynexa\logs\customer-service\*.json"] -.->|"written by"| Local
end
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class API,GW,Other,Local,Browser,DockerLogs,TempFile box
| Sink | Format | Where | Why |
|---|---|---|---|
| Console | Human-readable text (or compact JSON with ConsoleFormat = Json) |
Terminal / Visual Studio Output / docker logs |
Immediate feedback while developing; container log collection later |
| Rolling file | Compact JSON (CLEF), one file per day, 100 MB max, 7 files kept | %TEMP%\paynexa\logs\<service>\<service>-yyyyMMdd.json (or PayNexaLogging:FileDirectory) |
Durable fallback when Seq is down; can be replayed into Seq |
| Seq | Structured events (and OTLP traces) | http://localhost:5341 locally, http://seq:5341 from containers |
Search, filter, correlate across services, dashboards, alerts |
Port 5341 is Seq's ingestion endpoint (services write there). Port 5342 is its web UI (people read there).
3. Prerequisites
| Prerequisite | Why | How |
|---|---|---|
| .NET 10 SDK | Builds and runs the services | dotnet --version |
| Docker Desktop | Runs Seq (and the rest of the infrastructure) | docker compose up -d from the repository root |
src/.env |
Holds SEQ_ADMIN_PASSWORD for Seq's first login |
Copy src/.env.example → src/.env |
| Seq container running | Receives logs and traces | docker ps --filter name=infrastructure-seq |
Service:Name in appsettings.json |
Every event is stamped with it; startup fails fast without it | e.g. "Service": { "Name": "customer-service" } |
PayNexaLogging:SeqServerUrl |
Tells Serilog where Seq is | appsettings.Development.json (local) / docker-compose.override.yml (Docker) |
| Seq licence | Seq is free for individual use | No key needed locally |
Nothing else is required. Serilog, its sinks and OpenTelemetry are NuGet packages pinned in Directory.Packages.props, and all of them are free and open source (Apache-2.0).
4. Startup: how logging is switched on
Logging is live before the host exists, so a failure while building the app is still recorded.
sequenceDiagram
autonumber
participant P as Program.cs
participant B as Bootstrap logger<br/>(console only)
participant DI as AddPresentation →<br/>AddPayNexaServiceDefaults →<br/>AddPayNexaLogging
participant S as Full Serilog logger
participant I as InitializeInfrastructureAsync
P->>B: Log.Logger = PayNexaLogging.CreateBootstrapLogger()
P->>DI: builder.Services.AddPresentation(...).AddApplication().AddInfrastructure(...)
DI->>DI: ServiceIdentity.From(config) → name, version, environment
DI->>S: services.AddSerilog(...) configure sinks, enrichers, masking
P->>P: builder.Build()
Note over B,S: The bootstrap logger is "frozen" and replaced in place:<br/>from here on every log reaches console + file + Seq
P->>P: app.UsePresentation() → UsePayNexaServiceDefaults()
P->>I: migrations, indexes, Kafka topics, seeding
I->>S: "Startup: Initialize … started / succeeded / ended" (own correlation id)
P->>P: app.RunAsync()
Note over P,S: If anything throws: Log.Fatal("… terminated unexpectedly during startup")<br/>finally: Log.CloseAndFlushAsync() so nothing is lost
The code, in src/Services/Customer/Customer.API/Program.cs:
Log.Logger = PayNexaLogging.CreateBootstrapLogger();
try
{
var builder = WebApplication.CreateBuilder(args);
{
builder.Services
.AddPresentation(builder.Configuration, builder.Environment)
.AddApplication()
.AddInfrastructure(builder.Configuration);
}
var app = builder.Build();
{
app.UsePresentation();
await app.InitializeInfrastructureAsync();
await app.RunAsync();
}
return 0;
}
catch (Exception exception) when (exception is not HostAbortedException)
{
Log.Fatal(exception, "Customer service terminated unexpectedly during startup");
return 1;
}
finally
{
await Log.CloseAndFlushAsync();
}
What AddPayNexaLogging configures, in src/BuildingBlocks/PayNexa.Logging/PayNexaLogging.cs:
configuration
.MinimumLevel.Information()
.MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
.MinimumLevel.Override("Microsoft.EntityFrameworkCore", LogEventLevel.Warning)
.ReadFrom.Configuration(appConfiguration)
.Enrich.FromLogContext()
.Enrich.WithProperty("ServiceName", identity.Name)
.Enrich.WithProperty("ServiceVersion", identity.Version)
.Enrich.WithProperty("Environment", identity.Environment)
.Enrich.WithMachineName()
.Enrich.WithProcessId()
.Enrich.WithThreadId()
.Enrich.With<SensitiveDataMaskingEnricher>();
configuration.WriteTo.Async(sink => sink.Console(outputTemplate: TextTemplate));
configuration.WriteTo.Async(sink => sink.File(new CompactJsonFormatter(), path, rollingInterval: RollingInterval.Day, ...));
configuration.WriteTo.Seq(options.SeqServerUrl);
- Async sinks: writing a log never blocks the request thread.
- Unhandled exceptions:
RegisterProcessLevelExceptionLoggingalso hooksAppDomain.UnhandledExceptionandTaskScheduler.UnobservedTaskException, so even a crash outside a request is logged and flushed.
5. The HTTP pipeline: what wraps every request
UsePayNexaServiceDefaults() puts the logging middleware first, so it wraps everything else, including the exception handler.
flowchart TB
In(["Request in"]) --> C1
C1["1 · CorrelationIdMiddleware<br/>accept valid X-Correlation-Id or generate one<br/>push CorrelationId + RequestId into LogContext<br/>tag the trace, echo the header on the response"]
C1 --> C2["2 · RequestStartedLoggingMiddleware<br/>HTTP POST /api/v1/customers started"]
C2 --> C3{"Development and<br/>HttpBodies:Enabled?"}
C3 -->|"yes"| C4["3 · HttpBodyLoggingMiddleware<br/>request body (masked)<br/>… response body (status, masked)"]
C3 -->|"no"| C5
C4 --> C5["4 · Serilog RequestLoggingMiddleware<br/>HTTP POST … responded 201 in 421.02 ms<br/>+ RequestHost, ClientIp, UserId, EndpointName"]
C5 --> C6["5 · Exception handler → Problem Details"]
C6 --> C7["6 · Controllers → Mediator"]
C7 --> Out(["Response out<br/>X-Correlation-Id header"])
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class C1,C2,C4,C5,C6,C7 box
| Middleware | File | Logs |
|---|---|---|
CorrelationIdMiddleware |
PayNexa.Logging/Correlation/CorrelationIdMiddleware.cs |
A Warning when a malformed id is replaced |
RequestStartedLoggingMiddleware |
PayNexa.Logging/RequestLogging/RequestStartedLoggingMiddleware.cs |
HTTP {RequestMethod} {RequestPath} started |
HttpBodyLoggingMiddleware |
PayNexa.Logging/RequestLogging/HttpBodyLoggingMiddleware.cs |
Masked JSON bodies (Development only) |
| Serilog request logging | PayNexaLogging.UsePayNexaRequestLogging |
HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed} ms |
The completion level follows the outcome: Error for 5xx or an exception, Warning for 4xx, Information for success, and Debug for /health, /openapi and /swagger, so polling does not flood the logs.
6. Inside a command: behaviors and steps
Every command and query goes through the Mediator pipeline. Logging is a pipeline behavior, so handlers never write "started" or "ended" themselves.
flowchart LR
Ctrl["CustomersController<br/>CreateAsync(request.ToCommand())"] --> LB
LB["LoggingBehavior<br/>Command X started<br/>… succeeded / failed / slow / cancelled<br/>Command X ended"] --> VB
VB["ValidationBehavior<br/>X: Validation started<br/>succeeded / failed (N errors, field names only)<br/>ended"] --> DR
DR["DomainRuleBehavior<br/>DomainException → Result"] --> H
H["Handler<br/>logger.BeginStep('Email uniqueness check')<br/>logger.BeginStep('Persist customer to write store')"]
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class Ctrl,LB,VB,DR,H box
6.1 LoggingBehavior, the operation lifecycle
Source: src/BuildingBlocks/PayNexa.Common/Behaviors/LoggingBehavior.cs.
using var operationScope = OperationContext.Begin(OperationName);
using var logScope = logger.BeginScope(new Dictionary<string, object?>
{
["Operation"] = OperationName,
["OperationKind"] = OperationKind,
});
OperationLog.Started(logger, OperationKind, OperationName);
var startedAt = Stopwatch.GetTimestamp();
try
{
var response = await next(message, cancellationToken);
if (response is Result { IsFailure: true } failed) LogFailedResult(failed.Error, durationMs);
else OperationLog.Succeeded(logger, OperationName, durationMs);
if (durationMs >= options.Value.SlowOperationThresholdMs) OperationLog.Slow(...);
return response;
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { OperationLog.Cancelled(...); throw; }
catch (Exception exception) { OperationLog.Faulted(...); exception.MarkAsLogged(); throw; }
finally { OperationLog.Ended(logger, OperationName); }
stateDiagram-v2
[*] --> Started : Command X started
Started --> Succeeded : Result success
Started --> FailedResult : Result failure, Warning or Error by error type
Started --> Faulted : exception, Error with stack trace
Started --> Cancelled : client went away
Succeeded --> Slow : duration over 500 ms
Succeeded --> Ended
Slow --> Ended
FailedResult --> Ended
Faulted --> Ended
Cancelled --> Ended
Ended --> [*] : X ended
6.2 Steps: named work inside a handler
logger.BeginStep(...) (PayNexa.Common/Logging/OperationStep.cs) gives any block the same lifecycle, prefixed with the current operation name.
using (var step = logger.BeginStep("Email uniqueness check"))
{
if (await Customers.EmailExistsAsync(email, cancellationToken))
{
step.Failed(CustomerErrorCodes.EmailAlreadyRegistered);
return Errors.Customer.EmailAlreadyRegistered;
}
step.Succeeded();
}
Produces:
RegisterCustomerCommand: Email uniqueness check started
RegisterCustomerCommand: Email uniqueness check failed in 7 ms. Reason: Customer.EmailAlreadyRegistered
RegisterCustomerCommand: Email uniqueness check ended
step.WithProperty("CustomerVersion", customer.Version)attaches data to the outcome line.- A step disposed without
Succeeded()orFailed()(an exception or an earlyreturn) logs a Warning: "Step ended without an explicit outcome". A forgotten outcome can't go unnoticed.
7. Data stores, cache and broker
These are logged automatically by the building blocks, so no handler code is needed.
| Component | Mechanism | Example |
|---|---|---|
| SQL Server | EF Core SqlCommandLoggingInterceptor |
SQL Server INSERT on CustomerDb succeeded in 2.42 ms (SaveChanges) |
| MongoDB | Driver command events → MongoCommandLogger (skips hello, ping, auth…) |
MongoDB update on customers succeeded in 14.88 ms (paynexa_customer) |
| Redis | RedisCacheService steps with CacheKey, CacheHit, TtlSeconds |
Redis GET succeeded in 0.8 ms (CacheHit = true) |
| Kafka | KafkaEventBus step with EventId, EventType, Topic, MessageKey, Partition, Offset |
Kafka publish customer.created to paynexa.customer succeeded in 23.75 ms |
| Domain events | DomainEventDispatcher steps |
Handle domain event CustomerRegisteredDomainEvent succeeded in 30.22 ms |
SQL text is never logged, only the operation (SELECT, INSERT, …), the database, the duration and the source (LinqQuery, SaveChanges). The outbox's own 1-second polling query is tagged BackgroundPolling and logged at Debug, so it doesn't bury real traffic.
8. After the response: outbox, projection, Kafka
The client gets its 201 as soon as SQL Server commits. The rest happens in the background OutboxProcessor, and it keeps the request's correlation id, because the id is stored with the outbox row.
sequenceDiagram
autonumber
participant C as Client
participant API as Customer API
participant SQL as SQL Server<br/>CustomerDb
participant OP as OutboxProcessor<br/>(background, every 1 s)
participant M as MongoDB<br/>paynexa_customer
participant R as Redis
participant K as Kafka<br/>paynexa.customer
C->>API: POST /api/v1/customers (X-Correlation-Id: doc-demo-27493)
API->>SQL: INSERT customer + INSERT OutboxMessage<br/>(CorrelationId, TraceParent) in ONE transaction
API-->>C: 201 Created (X-Correlation-Id: doc-demo-27493)
Note over API: logs so far: HTTP started → body → command → validation →<br/>steps → SQL → responded 201
OP->>SQL: claim pending rows (UPDLOCK, READPAST)
OP->>OP: restore CorrelationId + trace parent from the row
OP->>M: Project customer to read store (version-checked upsert)
OP->>R: Redis DEL customer:{id}:profile
OP->>K: Kafka publish customer.created<br/>headers: correlation-id, traceparent, event-id …
OP->>SQL: mark processed
Note over OP: logs: Outbox.CustomerCreatedIntegrationEvent: Dispatch … started →<br/>Project … → Redis DEL … → Kafka publish … → succeeded / ended<br/>all with CorrelationId = doc-demo-27493
A failure here is visible too. A failed dispatch logs "failed on attempt N; retrying in S s". After Outbox:MaxAttempts (20) the row is dead-lettered and logged at Critical: "needs manual attention".
9. Correlation: how one id follows the whole journey
flowchart LR
Client["Client<br/>may send X-Correlation-Id"] -->|"header"| GW["Gateway<br/>accept or generate<br/>forward header"]
GW -->|"X-Correlation-Id"| API["Customer API<br/>LogContext.CorrelationId<br/>trace tag correlation.id"]
API -->|"column CorrelationId<br/>+ TraceParent"| OB[("Outbox row")]
OB -->|"restored by processor"| OP["OutboxProcessor logs"]
OP -->|"Kafka header correlation-id<br/>+ traceparent"| K{{"Kafka"}}
K -->|"consumer restores id<br/>(future consumers)"| N["Notification / Transaction"]
API -->|"typed service client<br/>forwards X-Correlation-Id"| P["Other service"]
API -->|"response header +<br/>Problem Details correlationId"| Client
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class Client,GW,API,OP,N,P box
Rules, from CorrelationContext and CorrelationIdMiddleware:
- Accepting an id: a client id is accepted if it is 1–64 characters of letters, digits,
-,_,.or:. Anything else is replaced, with a Warning. - New ids are UUID v7 without dashes (
01a0dcb81b387b88…). They sort by time. - Where it appears: every log line, every trace (
correlation.id), every error body ("correlationId": "…") and the response header.
How support uses it: a customer reports "it failed, reference 01a0dc9e…". Paste that into Seq, and the full story appears across every service.
10. Service-to-service calls, retries and outages
When a service calls another (for example Payment → Customer payment-eligibility), the typed client from AddServiceClient<…> logs the call and every resilience event.
sequenceDiagram
autonumber
participant Pay as payment-service
participant H as ServiceCallLoggingHandler<br/>+ resilience (retry, timeout, circuit)
participant Cus as customer-service
Pay->>H: GetPaymentEligibility(customerId)
H->>H: "payment-service -> customer-service GetPaymentEligibility started"
H->>Cus: GET /api/v1/customers/{id}/payment-eligibility (X-Correlation-Id)
Cus--xH: 503 / timeout
H->>H: Warning "attempt 1 failed with 503, retrying in 400 ms"
H->>Cus: retry
Cus--xH: 503
H->>H: Error "Circuit OPENED for customer-service … considered down"
H-->>Pay: Error "failed: … customer-service is unavailable after N attempts"
Note over H: later: "Circuit HALF-OPEN … probing" → "Circuit CLOSED … calls flow normally again"
- Unsafe methods are never retried automatically: POST and PUT go out once, and duplicates are prevented with idempotency keys.
- Metrics:
paynexa.service_call.retriesandpaynexa.service_call.circuit_openedare OpenTelemetry metrics, ready for dashboards.
11. Errors and exceptions
flowchart TB
E{"What went wrong?"}
E -->|"invalid input"| V["ValidationBehavior: Validation failed (N errors)<br/>LoggingBehavior: failed with Validation.Failed<br/>Warning · HTTP 400"]
E -->|"business rule<br/>(duplicate email, closed customer)"| B["Step failed · Reason: Customer.EmailAlreadyRegistered<br/>LoggingBehavior: failed with … (Conflict)<br/>Warning · HTTP 409 / 404 / 422"]
E -->|"unexpected exception"| X["LoggingBehavior: failed with unhandled SqlException<br/>Error + stack trace (logged once)<br/>GlobalExceptionHandler: Previously logged … mapped to HTTP 503"]
E -->|"client disconnected"| A["HTTP … aborted by the client<br/>Information · 499"]
E -->|"process crash"| F["Fatal · flushed before exit"]
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class V,B,X,A,F box
No duplicate stack traces. LoggingBehavior logs the exception with its stack and calls exception.MarkAsLogged(). GlobalExceptionHandler then logs only a one-line mapping ("Previously logged SqlException mapped to HTTP 503") instead of the same stack twice.
12. Anatomy of a log event
What Seq stores for one line (for example "RegisterCustomerCommand succeeded in 342.62 ms"):
classDiagram
class LogEvent {
Timestamp 2026-09-26T07:57:28.521Z
Level Information
MessageTemplate Operation succeeded in DurationMs ms
EventId 1001
SourceContext PayNexa.Common.Behaviors.LoggingBehavior
}
class Service {
ServiceName customer-service
ServiceVersion 1.0.0
Environment Development
MachineName, ProcessId, ThreadId
}
class Correlation {
CorrelationId doc-demo-27493
RequestId
TraceId, SpanId
}
class Operation {
Operation RegisterCustomerCommand
OperationKind Command
DurationMs 342.62
}
class Http {
RequestMethod POST
RequestPath /api/v1/customers
StatusCode, ClientIp, UserId, EndpointName
}
LogEvent --> Service : enrichers
LogEvent --> Correlation : LogContext + Activity
LogEvent --> Operation : scope + template
LogEvent --> Http : request logging
| Property | Source |
|---|---|
ServiceName, ServiceVersion, Environment |
ServiceIdentity (config Service:Name, assembly version) |
MachineName, ProcessId, ThreadId |
Serilog enrichers |
CorrelationId, RequestId |
CorrelationIdMiddleware → LogContext |
TraceId, SpanId |
Current OpenTelemetry Activity |
Operation, OperationKind, Step |
LoggingBehavior / OperationStep scopes |
DurationMs, ErrorCode, ErrorType, CacheHit, Topic, Offset, … |
The message template and WithProperty(...) |
Every message is declared once with [LoggerMessage] source generation. That makes it fast (no boxing and no parsing at runtime), strongly typed, and consistent: the same template always yields the same property names.
13. Masking personal data and secrets
One rule set, SensitiveFields, is applied in two places:
SensitiveDataMaskingEnrichermasks every structured property of every event, including nested objects and arrays.JsonBodyMaskermasks every field of a logged request or response body.
flowchart TB
N["Field / property name"] --> S{"contains password, secret, token,<br/>apikey, authorization, credential,<br/>cardnumber, cvv, cvc,<br/>connectionstring, privatekey?"}
S -->|"yes"| R["***REDACTED***<br/>(whole value, even objects and numbers)"]
S -->|"no"| Em{"contains email?"}
Em -->|"yes"| EM["r***@example.com"]
Em -->|"no"| Ph{"contains phone?"}
Ph -->|"yes"| PH["*********1234"]
Ph -->|"no"| Pe{"firstName, lastName, fullName,<br/>dateOfBirth, line1, line2, postalCode?"}
Pe -->|"yes"| PE["S***"]
Pe -->|"no"| K["kept as is<br/>(ids, status, city, country, dates…)"]
classDef box fill:#e3f0fc,stroke:#9cc3ea,color:#0b4a8b
class R,EM,PH,PE,K box
request body: {"firstName":"N***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********3000",
"dateOfBirth":"1***","address":{"line1":"G***","line2":null,"city":"Dhaka","postalCode":"1***","countryCode":"BD"}}
Body logging is Development only: it needs the environment to be Development and PayNexaLogging:HttpBodies:Enabled = true. Bodies longer than 4096 characters are truncated. Non-JSON bodies and health, OpenAPI and Swagger calls are not logged.
14. Levels, event ids and noise control
| Level | Used for |
|---|---|
| Debug | Health, OpenAPI and Swagger requests; outbox polling queries |
| Information | Lifecycle (started, succeeded, ended), request completion, business events, bodies |
| Warning | Expected failures (validation, not found, conflict), slow operations, retries, rejected correlation ids |
| Error | Unexpected exceptions, 5xx, downstream unavailable, circuit opened |
| Critical / Fatal | Dead-lettered outbox messages, process crash |
| Event id range | Area |
|---|---|
| 1000–1006 | Operation lifecycle (LoggingBehavior) |
| 1100–1104 | Steps (OperationStep) |
| 2000–2010 | HTTP: request started (2000), request body (2001), response body (2002), rejected correlation id (2010) |
| 3000+ | Service-to-service calls and resilience |
| 4000 / 4100 / 4200 / 4300 | MongoDB / Redis / SQL Server and outbox / Kafka |
| 5000–5002 | Exceptions and aborted requests |
| 10001–10003 | Customer business events: registered, changed, seeded |
Framework noise is trimmed by overrides: Microsoft.AspNetCore, Microsoft.EntityFrameworkCore, System.Net.Http.HttpClient and Polly log at Warning or above. Microsoft.Hosting.Lifetime stays at Information, which keeps "Now listening on …". For a deep dive, set Serilog:MinimumLevel:Default to Debug in appsettings.Development.json.
15. Reading the logs: console, file, Seq
15.1 Console
[07:57:28.521 INF] customer-service doc-demo-27493 PayNexa.Common.Behaviors.LoggingBehavior
RegisterCustomerCommand succeeded in 342.62 ms
Format: [time level] service correlationId source, then the message indented on the next line.
- Visual Studio /
dotnet run: the terminal or the Output window. - Docker:
docker logs -f service-customer-api.
15.2 Log file
Get-Content "$env:TEMP\paynexa\logs\customer-service\customer-service-$(Get-Date -Format yyyyMMdd).json" -Wait -Tail 20
15.3 Seq
Open http://localhost:5342 and sign in as admin with SEQ_ADMIN_PASSWORD from src/.env.
| Question | Seq query |
|---|---|
| Everything for one request | CorrelationId = 'doc-demo-27493' |
| Everything for one trace id (from an error response) | @TraceId = '930f34d0f6df392c8f6b90b604f25b61' |
| Only the Customer service | ServiceName = 'customer-service' |
| All errors today | @Level = 'Error' (then choose the time range) |
| Failed commands by error code | Operation like '%Command' and ErrorCode is not null |
| Slow operations | DurationMs > 500 |
| Duplicate-email attempts | ErrorCode = 'Customer.EmailAlreadyRegistered' |
| Request / response bodies | EventId.Id in [2001, 2002] |
| Kafka publishes | Topic = 'paynexa.customer' |
| A downstream outage | TargetService = 'customer-service' and @Level = 'Error' |
The lines in Seq with a duration on the right (for example GET api/v{version:apiVersion}/customers 10 ms, outbox process CustomerCreatedIntegrationEvent 17 ms, paynexa.customer publish 11 ms) are OpenTelemetry spans, exported to Seq's OTLP endpoint. Click one to see the trace tree, and use Trace on any log line to jump between logs and spans.
16. What this gives the business and the code
16.1 Business
| Benefit | How the logging delivers it |
|---|---|
| Faster incident resolution | The correlation id in every error response lets support find the full story in one Seq search, instead of guessing across machines. |
| Outage detection | "Circuit OPENED for customer-service … considered down" and dead-letter Critical logs are ready-made alert triggers. |
| Performance and SLA evidence | Every request, command, query and store call carries DurationMs; slow operations raise Warnings automatically. |
| Compliance and privacy | PII is masked and secrets redacted before any sink (GDPR data minimisation; PCI DSS forbids card data in logs), and there is still a complete audit trail of who did what, when, with what outcome. |
| Fraud and abuse signals | Repeated Customer.EmailAlreadyRegistered, validation storms or KYC rejections are queryable by ErrorCode, ClientIp and UserId. |
| Product insight | Business events (Customer … registered, KYC changes, payment-eligibility checks) can be counted over time without a separate analytics pipeline. |
| Money safety | The outbox, dispatch and dead-letter logs show whether an event such as a payment completion actually left the system. Nothing is lost silently. |
16.2 Code and engineering
| Benefit | How |
|---|---|
| Zero logging boilerplate in handlers | Lifecycle, validation, SQL, MongoDB, Redis, Kafka, outbox and HTTP are logged by building blocks; handlers only name meaningful steps. |
| Consistency across services | The same templates, properties and levels everywhere; a new service gets it all through AddPayNexaServiceDefaults(). |
| Performance | [LoggerMessage] source generation, async sinks, and Debug level for noisy polling. |
| Debuggability | Development-only masked request/response bodies, explicit step outcomes, and a Warning for steps that end without one. |
| Testability | Logging is plain ILogger; unit tests assert on it with FakeLogger. |
| Vendor freedom | Serilog sinks and OTLP are standards; Seq can be swapped for Elasticsearch, Grafana Loki or a cloud backend by changing configuration, not code. |
17. Configuration reference
appsettings.json (all environments):
"Service": { "Name": "customer-service" },
"OperationLogging": { "SlowOperationThresholdMs": 500 },
"PayNexaLogging": {
"ConsoleFormat": "Text",
"SeqServerUrl": "",
"RetainedFileCount": 7,
"HttpBodies": {
"Enabled": false,
"MaxLoggedLength": 4096,
"MaxCapturedBytes": 1048576
}
},
"Observability": {
"OtlpTracesEndpoint": null,
"OtlpMetricsEndpoint": null
},
"Serilog": {
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft.AspNetCore": "Warning",
"Microsoft.EntityFrameworkCore": "Warning"
}
}
},
appsettings.Development.json (local runs):
"PayNexaLogging": { "SeqServerUrl": "http://localhost:5341", "HttpBodies": { "Enabled": true } },
"Observability": { "OtlpTracesEndpoint": "http://localhost:5341/ingest/otlp/v1/traces" }
docker-compose.override.yml (containers):
PayNexaLogging__SeqServerUrl: http://seq:5341
PayNexaLogging__ConsoleFormat: Text
PayNexaLogging__HttpBodies__Enabled: "true"
Observability__OtlpTracesEndpoint: http://seq:5341/ingest/otlp/v1/traces
| Key | Default | Purpose |
|---|---|---|
Service:Name |
required | Name stamped on every event and trace |
PayNexaLogging:SeqServerUrl |
empty (Seq off) | Seq ingestion URL |
PayNexaLogging:SeqApiKey |
empty | Seq API key when Seq requires one |
PayNexaLogging:ConsoleFormat |
Text |
Text or Json |
PayNexaLogging:FileDirectory |
%TEMP%/paynexa/logs/<service> |
Rolling file location |
PayNexaLogging:RetainedFileCount |
7 | Days of files kept |
PayNexaLogging:HttpBodies:* |
off | Development-only body logging |
OperationLogging:SlowOperationThresholdMs |
500 | "is slow" Warning threshold |
Observability:OtlpTracesEndpoint / OtlpMetricsEndpoint |
off | Where traces and metrics are exported |
Serilog:MinimumLevel:* |
Information | Standard Serilog level switches |
18. Known limitations
- Validation messages are masked in logged 400 bodies. Problem Details puts messages under field names (
"errors": { "email": [...] }), and the name rule masks them (***REDACTED***). The client still receives the real messages; only the log copy is masked. The Validation step still logs which fields failed (InvalidFields). - Seq outage. Seq outages lose nothing. The Seq sink buffers in memory and retries, and the rolling file keeps every event regardless.
- Kafka consumers will restore the
correlation-idheader when the first consuming service (Notification or Transaction) is built.
19. Real console output, end to end
The three traces below were captured from docker logs service-customer-api. Each one is filtered by its correlation id, and the [time level] service correlationId source prefix is shown only on the first trace.
19.1 Successful registration: 201 Created
[14:52:44.821 INF] customer-service postman-1790412764065-33571 PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB find on customers succeeded in 0.76 ms (paynexa_customer)
[14:52:44.822 INF] customer-service postman-1790412764065-33571 PayNexa.Common.Behaviors.LoggingBehavior
ListCustomersQuery succeeded in 3.64 ms
[14:52:44.822 INF] customer-service postman-1790412764065-33571 PayNexa.Common.Behaviors.LoggingBehavior
ListCustomersQuery ended
[14:52:44.822 INF] customer-service postman-1790412764065-33571 Serilog.AspNetCore.RequestLoggingMiddleware
HTTP GET /api/v1/customers responded 200 in 10.07 ms
[14:52:44.822 INF] customer-service postman-1790412764065-33571 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
HTTP GET /api/v1/customers response body (200): {"items":[{"id":"01a0dce9-cbcd-71c3-be12-275da823c84a","firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********8259","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:51:44.717Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:51:44.717Z","updatedBy":"anonymous"},{"id":"01a0dce9-7792-7b94-956a-e893b53ac7e0","firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********2821","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:51:23.155Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:51:23.155Z","updatedBy":"anonymous"},{"id":"01a0dce4-56fd-7e28-a31f-eff26f632c2e","firstName":"N***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********0999","dateOfBirth":"1***","address":{"line1":"B***","line2":"F***","city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Closed","statusReason":"Customer requested account closure","kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:45:47.337Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:46:11.616Z","updatedBy":"anonymous"},{"id":"189dc8dc-990f-48e0-a37b-e6f2b60b9d7d","firstName":"T***","lastName":"C***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"S***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:46:06.84Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:46:06.84Z","updatedBy":"system"},{"id":"58c49479-ec65-4de2-86e7-033c546291aa","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"M***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:43:20.347Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:43:20.347Z","updatedBy":"system"},{"id":"01a0d92c-1476-7bf2-a97e-c02723b33032","firstName":"N***","lastName":"R***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"B***","line2":"F***","city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Closed","statusReason":"Customer request","kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-25T15:25:39.992Z","createdBy":"anonymous","updatedAtUtc":"2026-09-25T15:25:41.559Z","updatedBy":"anonymous"},{"id":"01a0d775-8b75-72de-bcab-3c690216124b","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":null,"status":"Active","statusReason":null,"kycStatus":null,"kycRejectionReason":null,"createdAtUtc":"2026-09-25T07:26:39.989Z","createdBy":null,"updatedAtUtc":"2026-09-25T07:27:40.173Z","updatedBy":null}],"page":1,"pageSize":20,"totalCount":7,"totalPages":1,"hasPreviousPage":false,"hasNextPage":false}
[14:54:12.322 INF] customer-service postman-1790412850987-45508 PayNexa.Logging.RequestLogging.RequestStartedLoggingMiddleware
HTTP POST /api/v1/customers started
[14:54:12.322 INF] customer-service postman-1790412850987-45508 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
HTTP POST /api/v1/customers request body: {"firstName":"M***","lastName":"H***","email":"m***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"}}
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.LoggingBehavior
Command RegisterCustomerCommand started
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.ValidationBehavior
RegisterCustomerCommand: Validation started
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.ValidationBehavior
RegisterCustomerCommand: Validation succeeded in 0.16 ms
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.ValidationBehavior
RegisterCustomerCommand: Validation ended
[14:54:12.323 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
RegisterCustomerCommand: Email uniqueness check started
[14:54:12.324 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
SQL Server SELECT on CustomerDb started (LinqQuery)
[14:54:12.327 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
SQL Server SELECT on CustomerDb succeeded in 3.43 ms (LinqQuery)
[14:54:12.327 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
RegisterCustomerCommand: Email uniqueness check succeeded in 4.55 ms
[14:54:12.327 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
RegisterCustomerCommand: Email uniqueness check ended
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
RegisterCustomerCommand: Persist customer to write store started
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Common.DomainEvents.DomainEventDispatcher
RegisterCustomerCommand: Handle domain event CustomerRegisteredDomainEvent started
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Common.DomainEvents.DomainEventDispatcher
RegisterCustomerCommand: Handle domain event CustomerRegisteredDomainEvent succeeded in 0.11 ms
[14:54:12.328 INF] customer-service postman-1790412850987-45508 PayNexa.Common.DomainEvents.DomainEventDispatcher
RegisterCustomerCommand: Handle domain event CustomerRegisteredDomainEvent ended
[14:54:12.329 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
SQL Server INSERT on CustomerDb started (SaveChanges)
[14:54:12.332 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Logging.SqlCommandLoggingInterceptor
SQL Server INSERT on CustomerDb succeeded in 3.58 ms (SaveChanges)
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
RegisterCustomerCommand: Persist customer to write store succeeded in 5.41 ms
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
RegisterCustomerCommand: Persist customer to write store ended
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Application.Commands.RegisterCustomer.RegisterCustomerCommandHandler
Customer 01a0dcec-0c67-774c-aee8-a3cc9951822f registered
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.LoggingBehavior
RegisterCustomerCommand succeeded in 10.69 ms
[14:54:12.333 INF] customer-service postman-1790412850987-45508 PayNexa.Common.Behaviors.LoggingBehavior
RegisterCustomerCommand ended
[14:54:12.334 INF] customer-service postman-1790412850987-45508 Serilog.AspNetCore.RequestLoggingMiddleware
HTTP POST /api/v1/customers responded 201 in 12.07 ms
[14:54:12.334 INF] customer-service postman-1790412850987-45508 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
HTTP POST /api/v1/customers response body (201): {"id":"01a0dcec-0c67-774c-aee8-a3cc9951822f","firstName":"M***","lastName":"H***","email":"m***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T08:54:12.3282326Z","createdBy":"anonymous","updatedAtUtc":"2026-09-26T08:54:12.3282326Z","updatedBy":"anonymous"}
[14:54:12.848 INF] customer-service postman-1790412850987-45508 PayNexa.SqlServer.Outbox.OutboxProcessor
Outbox.CustomerCreatedIntegrationEvent: Dispatch CustomerCreatedIntegrationEvent started
[14:54:12.849 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Infrastructure.MongoDB.CustomerProjectionHandler
Outbox.CustomerCreatedIntegrationEvent: Project customer to read store started
[14:54:12.849 INF] customer-service postman-1790412850987-45508 PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB update on customers started (paynexa_customer)
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB update on customers succeeded in 2.57 ms (paynexa_customer)
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Infrastructure.MongoDB.CustomerProjectionHandler
Outbox.CustomerCreatedIntegrationEvent: Project customer to read store succeeded in 3.53 ms
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.Customers.Infrastructure.MongoDB.CustomerProjectionHandler
Outbox.CustomerCreatedIntegrationEvent: Project customer to read store ended
[14:54:12.852 INF] customer-service postman-1790412850987-45508 PayNexa.Caching.RedisCacheService
Outbox.CustomerCreatedIntegrationEvent: Redis DEL started
[14:54:12.854 INF] customer-service postman-1790412850987-45508 PayNexa.Caching.RedisCacheService
Outbox.CustomerCreatedIntegrationEvent: Redis DEL succeeded in 1.44 ms
[14:54:12.854 INF] customer-service postman-1790412850987-45508 PayNexa.Caching.RedisCacheService
Outbox.CustomerCreatedIntegrationEvent: Redis DEL ended
[14:54:12.854 INF] customer-service postman-1790412850987-45508 PayNexa.Messaging.Kafka.KafkaEventBus
Outbox.CustomerCreatedIntegrationEvent: Kafka publish customer.created to paynexa.customer started
[14:54:12.867 INF] customer-service postman-1790412850987-45508 PayNexa.Messaging.Kafka.KafkaEventBus
Outbox.CustomerCreatedIntegrationEvent: Kafka publish customer.created to paynexa.customer succeeded in 12.89 ms
How to read it:
- request body is masked.
- The command starts, validation passes, and the email check runs one
SELECT. - Persisting raises the domain event, which queues the integration event in the outbox. SQL Server then
INSERTs the customer and the outbox row in one transaction. - The command succeeds in 342 ms and the client gets
201after 421 ms. - About 80 ms later, the outbox processor projects the customer to MongoDB, clears the Redis cache entry and publishes
customer.createdto Kafka. All of it still carriesdoc-demo-27493.
19.2 Invalid input: 400 Bad Request
HTTP POST /api/v1/customers started
[14:59:02.751 INF] customer-service postman-1790413141501-51236 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
HTTP POST /api/v1/customers request body: {"firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********6701","dateOfBirth":"2***","address":{"line1":"K***","line2":null,"city":"Dhaka","state":null,"postalCode":"1***","countryCode":"BD"}}
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.LoggingBehavior
Command RegisterCustomerCommand started
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.ValidationBehavior
RegisterCustomerCommand: Validation started
[14:59:02.754 WRN] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.ValidationBehavior
RegisterCustomerCommand: Validation failed in 0.4 ms. Reason: 1 validation error(s)
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.ValidationBehavior
RegisterCustomerCommand: Validation ended
[14:59:02.754 WRN] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.LoggingBehavior
RegisterCustomerCommand failed in 0.65 ms with Validation.Failed (Validation)
[14:59:02.754 INF] customer-service postman-1790413141501-51236 PayNexa.Common.Behaviors.LoggingBehavior
RegisterCustomerCommand ended
[14:59:02.755 WRN] customer-service postman-1790413141501-51236 Serilog.AspNetCore.RequestLoggingMiddleware
HTTP POST /api/v1/customers responded 400 in 3.74 ms
[14:59:02.755 INF] customer-service postman-1790413141501-51236 PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
HTTP POST /api/v1/customers response body (400): {"type":"https://tools.ietf.org/html/rfc9110#section-15.5.1","title":"Validation failed","status":400,"detail":"One or more validation errors occurred.","instance":"/api/v1/customers","errors":{"dateOfBirth":["\u0027***"]},"traceId":"930f34d0f6df392c8f6b90b604f25b61","correlationId":"postman-1790413141501-51236","errorCode":"Validation.Failed"}
No database is touched: validation stops the request before the handler runs.
19.3 Business rule: duplicate email → 409 Conflict
HTTP POST /api/v1/customers started
HTTP POST /api/v1/customers request body: {"firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***"}
Command RegisterCustomerCommand started
RegisterCustomerCommand: Validation started
RegisterCustomerCommand: Validation succeeded in 0.33 ms
RegisterCustomerCommand: Validation ended
RegisterCustomerCommand: Email uniqueness check started
SQL Server SELECT on CustomerDb started (LinqQuery)
SQL Server SELECT on CustomerDb succeeded in 1.51 ms (LinqQuery)
RegisterCustomerCommand: Email uniqueness check failed in 7 ms. Reason: Customer.EmailAlreadyRegistered
RegisterCustomerCommand: Email uniqueness check ended
RegisterCustomerCommand failed in 8.78 ms with Customer.EmailAlreadyRegistered (Conflict)
RegisterCustomerCommand ended
HTTP POST /api/v1/customers responded 409 in 25.77 ms
HTTP POST /api/v1/customers response body (409): {"type":"https://tools.ietf.org/html/rfc9110#section-15.5.10","title":"Request conflicts with the current state","status":409,"detail":"A customer with this email address already exists.",...}
The exact step that failed and why (Customer.EmailAlreadyRegistered) is on record. No stack trace is logged, because it is an expected business outcome, not a bug.
19.4 Successful data fetch → response body (200)
[14:35:11.404 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.ValidationBehavior
ListCustomersQuery: Validation succeeded in 16.23 ms
[14:35:11.404 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.ValidationBehavior
ListCustomersQuery: Validation ended
[14:35:11.435 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB aggregate on customers started (paynexa_customer)
[14:35:11.442 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB aggregate on customers succeeded in 7.26 ms (paynexa_customer)
[14:35:11.489 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB find on customers started (paynexa_customer)
[14:35:11.490 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.MongoDb.Logging.MongoCommandLogger
MongoDB find on customers succeeded in 1.94 ms (paynexa_customer)
[14:35:11.533 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.LoggingBehavior
ListCustomersQuery succeeded in 145.4 ms
[14:35:11.536 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Common.Behaviors.LoggingBehavior
ListCustomersQuery ended
[14:35:11.568 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d Serilog.AspNetCore.RequestLoggingMiddleware
HTTP GET /api/v1/customers responded 200 in 272.19 ms
[14:35:11.579 INF] customer-service 01a0dcdaa3387450866ce2baf74e549d PayNexa.Logging.RequestLogging.HttpBodyLoggingMiddleware
HTTP GET /api/v1/customers response body (200): {"items":[{"id":"189dc8dc-990f-48e0-a37b-e6f2b60b9d7d","firstName":"T***","lastName":"C***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"S***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Pending","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:46:06.84Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:46:06.84Z","updatedBy":"system"},{"id":"58c49479-ec65-4de2-86e7-033c546291aa","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"M***","line2":null,"city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Active","statusReason":null,"kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-26T06:43:20.347Z","createdBy":"system","updatedAtUtc":"2026-09-26T06:43:20.347Z","updatedBy":"system"},{"id":"01a0d92c-1476-7bf2-a97e-c02723b33032","firstName":"N***","lastName":"R***","email":"t***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"B***","line2":"F***","city":"Dhaka","state":"Dhaka","postalCode":"1***","countryCode":"BD"},"status":"Closed","statusReason":"Customer request","kycStatus":"Verified","kycRejectionReason":null,"createdAtUtc":"2026-09-25T15:25:39.992Z","createdBy":"anonymous","updatedAtUtc":"2026-09-25T15:25:41.559Z","updatedBy":"anonymous"},{"id":"01a0d775-8b75-72de-bcab-3c690216124b","firstName":"S***","lastName":"R***","email":"s***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":null,"status":"Active","statusReason":null,"kycStatus":null,"kycRejectionReason":null,"createdAtUtc":"2026-09-25T07:26:39.989Z","createdBy":null,"updatedAtUtc":"2026-09-25T07:27:40.173Z","updatedBy":null}],"page":1,"pageSize":20,"totalCount":4,"totalPages":1,"hasPreviousPage":false,"hasNextPage":false}
Note: Sensitive data are masked, and it is completely configurable whether we want to display http request/response body in the logger/console or not
20. Seeing it in action: screenshots
These screenshots come from a real session: Customer.API running locally, the rest of the stack in Docker, requests sent from the Postman collection in postman/.
20.1 The running stack in Docker Desktop

Everything runs as one Docker Compose project, paynexa, with container names grouped by prefix:
database-*: SQL Server (1433), MongoDB (27017) and Redis (6379).infrastructure-*:infrastructure-seqreceives logs on5341and serves the UI on5342.infrastructure-kafkais the event broker on9092.infrastructure-kafka-uiis the Kafka UI on8090.
service-*: the API gateway and the five APIs.service-customer-apiis stopped here because Customer.API was running from Visual Studio on the same ports (5001/6001). The local process still sends its logs to the same Seq container.
20.2 Seq: masked request and response bodies

Seq lists events newest first, so read each request from the bottom up:
Now listening on …,Outbox processor … started, thenStartup: Seed Customers … succeeded: the service started, and the lines above them are its first requests.HTTP POST /api/v1/customers started: the request arrives.request body: {"firstName":"S***","lastName":"R***","email":"n***@example.com","phoneNumber":"*********6701","dateOfBirth":"2***",...}: the body is logged with every personal field masked.Command RegisterCustomerCommand started, thenValidation started, thenValidation failed … Reason: 1 validation error(s): the lifecycle fromLoggingBehaviorandValidationBehavior. The orange dots mark Warning-level lines.RegisterCustomerCommand failed … with Validation.Failed (Validation), thenended.responded 400 in 3.74 msandresponse body (400): {"type":…,"title":"Validation failed",…}: the outcome and the Problem Details that went back to the client.POST api/v{version:apiVersion}/customers 5 mswith a duration on the right: the OpenTelemetry span for the same request, stored in Seq next to the logs.
The two outlined blocks are two separate attempts. Each has its own correlation id, so filtering on one shows only that request.
20.3 Postman: a bad request with traceId and correlationId

The collection's Register customer request was sent with a date of birth that makes the customer under 18. The API answered 400 Bad Request with Problem Details:
errors.dateOfBirth: the exact rule that failed ("must make the customer at least 18 years old").errorCode:Validation.Failed, a stable code clients can branch on.correlationId:postman-1790413141501-51236, the value Postman sent inX-Correlation-Idand the API echoed back.traceId:930f34d0f6df392c8f6b90b604f25b61, the W3C trace id of this request.
Both ids can be searched in Seq and lead to the same request:
| Id from the response | Seq query | Finds |
|---|---|---|
correlationId |
CorrelationId = 'postman-1790413141501-51236' |
Every log line of the request, plus background work it started (outbox, Kafka), even across services |
traceId |
@TraceId = '930f34d0f6df392c8f6b90b604f25b61' |
Every log line and the OpenTelemetry spans of the request, shown as a trace tree |
The trace id is stored on every event ("@tr":"930f34d0f6df392c8f6b90b604f25b61" in the log file; ten events for this request). Seq exposes it as the built-in @TraceId property. Clicking any event in Seq and choosing Trace does the same search for you.
The correlation id is the business-friendly reference to give support. The trace id is the technical reference for following timing across services and spans.
20.4 Postman: a successful query

List customers uses only environment variables: {{baseUrl}}/api/{{apiVersion}}/customers?page={{page}}&page-size={{pageSize}}. The optional filters (search, status, kyc-status) are unticked until needed. The response is 200 OK with a paged result read from the MongoDB read model, and the Test Results 3/3 show the collection's checks passed: status code, correlation id echoed, and a paged body.
The client receives the real data (names, emails, phone numbers). Masking applies only to what is written to the logs.
20.5 Console: the same response, masked in the log

This is the Customer.API console for that same request (postman-1790414119486-19708):
ListCustomersQuery succeeded in 129.35 ms, thenended.HTTP GET /api/v1/customers responded 200 in 187.12 ms.response body (200): {"items":[{"id":"01a0dcec-…","firstName":"M***","lastName":"H***","email":"m***@gmail.com","phoneNumber":"*********0000","dateOfBirth":"1***","address":{"line1":"K***",…,"city":"Dhaka",…,"postalCode":"1***","countryCode":"BD"},"status":"Active",…}
Compare it with the Postman response in 20.4:
| Field | Postman (client) | Console / Seq (log) |
|---|---|---|
firstName |
Mahbuba Sultana |
M*** |
email |
mahbuba.sultana.himu@gmail.com |
m***@gmail.com |
phoneNumber |
+8801300000000 |
*********0000 |
address.line1 |
Kazipara, Dhaka-1207, Bangladesh |
K*** |
city, countryCode, status, ids, timestamps |
unchanged | unchanged |
The log keeps enough to debug (ids, statuses, cities, the last four phone digits, the email domain) without storing anyone's identity in plain text.
No comments yet.
Sign in to leave a comment.