Noland / Systems & experiments
Application Event Lab
Unreliable events. Reliable application updates.
I built this simulation to explore how a backend can safely process unreliable external events while keeping a customer-facing application updated in real time. The interface is client-side, but the behavior models the validation, idempotency, ordering, persistence, queueing, and reconnection concerns I would account for in a production system.
Architectural simulation. Everything runs in memory in this tab. No real backend, Twilio connection, database, or WebSocket server is running. Refreshing clears the lab. All applicant data is fictional.
Try this: Send a normal event, then send its duplicate. Disconnect live updates, change the target status, send another event, and reconnect.
01 / Customer view
Avery Example Fictional applicant
Passport Renewal DEMO-1042
Live updates connected
Customer view frozen. Live notifications are disconnected. Reconnect to load the latest database state.
Application Submitted
- Application Submitted
- Documents Under Review
- Information Required
- Documents Verified
- Submitted to Agency
- Completed
Your fictional application has been submitted.
02 / Webhook generator
Introduce an event
Test difficult conditions
Inspect latest webhook envelope
Send an event to inspect its simulated envelope.
Ready. Send an event to follow its journey.
03 / Processing pipeline
From delivery to notification
- HTTPS webhook receivedWaiting
- Signature validatedWaiting
- Event deduplicatedWaiting
- Event recordedWaiting
- Background job queuedWaiting
- Application state updatedWaiting
- Real-time event publishedWaiting
- Client notifiedWaiting
04 / System health
Inside the simulated backend
- Webhooks received
- 0
- Events processed
- 0
- Invalid signatures
- 0
- Duplicate deliveries
- 0
- Queue depth
- 0
- Failed jobs
- 0
- Connected clients
- 1
- Application version
- 0
05 / Internal event log
Every delivery has a trace
| Time | Event ID | Event type | Result | Attempts | Version | Latency |
|---|
No matching events yet.
Design notes / Beyond the browser
How I Would Build the Production Version
I would keep receipt, business processing, and delivery separate. A NestJS REST controller accepts Twilio callbacks after a signature guard validates them. Injectable services normalize provider data and enforce application rules. PostgreSQL is the authority for applications, webhook events, pending jobs, and outgoing notifications.
- Validate before trusting. The simulation uses an explicit valid/invalid flag, not cryptography. Production validates
X-Twilio-Signatureusing the official Twilio SDK, the exact public URL including query parameters, and original request data. Form callbacks and JSON callbacks require their appropriate SDK validation paths. See Twilio’s webhook security guidance. - Record before acknowledging. A PostgreSQL unique constraint on
(provider, event_id)prevents duplicate inbox rows, even across server instances. Insert the inbox row and pending job in the same transaction, then acknowledge. A worker claims jobs with a lease and retries failed work with bounded backoff and a dead-letter path. - Order deliberately. The lab assumes one trusted, application-wide source sequence. Older snapshots are recorded for audit but deferred without changing state. Twilio does not supply a universal application version: real adapters need provider-specific ordering and a server-owned application revision. Arrival timestamps alone are insufficient. Independent calls and document streams need their own ordering rules; a production state machine also validates permitted status transitions.
- Separate state from notifications. Redis distributes messages across instances; it is not the application database. Socket.io rooms or GraphQL subscriptions deliver notifications to authorized clients. A Next.js dashboard uses authenticated GraphQL queries and mutations. On reconnect it joins its authorized channel, refetches state, and reconciles overlapping notifications by version.
- Make failures diagnosable. Structured logs, counters, queue age, retry counts, and correlation IDs link receipt to commit to publish. Log identifiers and outcomes, never sensitive personal data, documents, tokens, or full webhook bodies. Authorization applies to every application query, mutation, and room subscription.
The transactional outbox
A database update can succeed while publishing its notification fails. The stored state remains correct, but clients may not hear about it. Writing the state change and an outgoing event in the same database transaction closes that gap. A separate publisher claims outbox records, publishes with retries, and marks them sent only after publication. A crash after publishing can cause a second notification, so consumers still deduplicate by event ID or version. This is at-least-once delivery, not a promise of exactly-once transport.
The lab models that atomic commit and a successful outbox drain. Its worker-failure switch fails before commit; it does not simulate process crashes, disk persistence, network security, or broker outages. Its database is authoritative only within this tab’s lifetime.
| Technology | What I would use it for |
|---|---|
| REST | Inbound Twilio webhooks with provider-defined request bodies and acknowledgements. |
| GraphQL | Flexible dashboard queries and authorized application mutations. |
| Socket.io | Named live events and application rooms; GraphQL subscriptions are an alternative. |
| PostgreSQL | Durable state, unique event identities, and atomic transactions. |
| Redis | Cross-instance event distribution, separate from durable application state. |
| Background queues | Asynchronous work with retries, backoff, leases, and failure inspection. |
Production Sketch
Explanatory TypeScript outlines only; none execute on this site. Imports, DTO definitions, dependency wiring, and repository implementations are omitted. The db methods below describe transaction contracts, not a particular ORM API. Normalized events share the lab’s eventId, provider, eventType, applicationId, sequence, and payload fields. signatureStatus is simulation-only and never trusted from a real request.
1. NestJS webhook controller
@Controller('webhooks/twilio')
export class WebhookController {
constructor(private readonly events: ApplicationEvents) {}
@Post('status')
@UseGuards(TwilioSignatureGuard)
@HttpCode(202)
async receive(@Body() form: TwilioStatusCallback) {
// Adapter validates schema, maps provider IDs to an application,
// and derives a stable identity for this callback, not just CallSid.
const event = await this.events.normalizeTwilio(form);
await this.events.record(event); // inbox + pending job committed
return { accepted: true }; // acknowledgement, not completion
}
}2. Signature guard outline
@Injectable()
export class TwilioSignatureGuard implements CanActivate {
constructor(private readonly config: WebhookConfig) {}
canActivate(context: ExecutionContext): boolean {
const req = context.switchToHttp().getRequest();
// This endpoint accepts form-encoded callbacks only.
// Config contains the exact externally configured callback URL.
const valid = twilio.validateRequest(
this.config.authToken,
req.get('X-Twilio-Signature') ?? '',
this.config.exactPublicCallbackUrl,
req.body, // original form parameters, before DTO transformation
);
if (!valid) throw new ForbiddenException();
return true;
}
}Use the official SDK. Preserve the exact public URL and all original parameters; do not reconstruct it from untrusted forwarded headers. For JSON callbacks, preserve the raw body and use the SDK’s JSON-specific validation method. Secrets belong in server configuration only.
3. Idempotent receipt and worker service
@Injectable()
export class ApplicationEvents {
constructor(private readonly db: Database) {}
async record(event: ApplicationEvent) {
return this.db.transaction(async tx => {
// INSERT ... ON CONFLICT (provider, event_id) DO NOTHING
if (!await tx.inbox.insertIfAbsent(event)) return;
await tx.jobs.insert(event); // same durable transaction
});
}
async process(job: ClaimedJob) {
return this.db.transaction(async tx => {
const locked = await tx.jobs.lock(job.id);
if (locked.done) return;
const app = await tx.applications.lock(job.event.applicationId);
if (job.event.sequence <= app.sequence) {
await tx.jobs.finish(job.id, 'Deferred: Older Event');
return;
}
const next = applyValidatedEvent(app, job.event);
await tx.applications.save(next); // revision + 1
await tx.outbox.insert(outboxRecord(job.event, next));
await tx.jobs.finish(job.id, 'Processed');
}); // rollback on failure; lease/backoff makes job retryable
}
}4. GraphQL application resolver
@Resolver(() => ApplicationView)
@UseGuards(SessionGuard)
export class ApplicationResolver {
constructor(private readonly apps: Applications) {}
@Query(() => ApplicationView)
application(@Args('id') id: string, @CurrentUser() user: User) {
return this.apps.findAuthorized(id, user.id);
}
@Mutation(() => ApplicationView)
requestReview(@Args('id') id: string, @CurrentUser() user: User) {
// Check ownership and allowed transition; commit with outbox.
return this.apps.requestReviewAuthorized(id, user.id);
}
}5. Socket.io gateway
@WebSocketGateway({ namespace: '/applications' })
export class ApplicationGateway {
@WebSocketServer() server: Server;
constructor(private readonly access: ApplicationAccess) {}
@UseGuards(SocketSessionGuard)
@SubscribeMessage('application.watch')
async watch(@ConnectedSocket() client: Socket,
@MessageBody() input: WatchApplicationDto) {
await this.access.assertOwner(client.data.user.id, input.id);
await client.join(`application:${input.id}`);
return { joined: input.id }; // client then refetches via GraphQL
}
publish(event: ApplicationUpdated) {
this.server.to(`application:${event.applicationId}`)
.emit('application.updated', event);
}
} // Configure a Redis adapter for distribution across instances.6. Transactional outbox record
function outboxRecord(event: ApplicationEvent, app: Application) {
return {
id: `${event.provider}:${event.eventId}:application.updated`,
topic: 'application.updated',
applicationId: app.id,
payload: { applicationId: app.id, version: app.version },
publishedAt: null,
attempts: 0,
};
}
// Insert alongside application update and job completion.
// Publisher claims with a lease, publishes, then marks sent.
// Failed sends retry with backoff; expired leases recover crashes.
// Repeated notifications are safe: clients compare versions/refetch.