Noland / Systems & experiments

Application Event Lab

Unreliable events. Reliable application updates.

I built this simulation to explore how a backend can safely process unreliable external events while keeping a customer-facing application updated in real time. The interface is client-side, but the behavior models the validation, idempotency, ordering, persistence, queueing, and reconnection concerns I would account for in a production system.

Architectural simulation. Everything runs in memory in this tab. No real backend, Twilio connection, database, or WebSocket server is running. Refreshing clears the lab. All applicant data is fictional.

Try this: Send a normal event, then send its duplicate. Disconnect live updates, change the target status, send another event, and reconnect.

01 / Customer view

Avery Example Fictional applicant

Passport Renewal DEMO-1042

Live updates connected

Application Submitted

1 of 6
  1. Application Submitted
  2. Documents Under Review
  3. Information Required
  4. Documents Verified
  5. Submitted to Agency
  6. Completed

Your fictional application has been submitted.

Updated 09:00:00 UTC · Customer v0

02 / Webhook generator

Introduce an event

Test difficult conditions

No delayed event.

Out-of-order delivers Documents Verified, then an older Information Required snapshot. Worker failure affects the next attempt before commit.

Inspect latest webhook envelope
Send an event to inspect its simulated envelope.

Ready. Send an event to follow its journey.

03 / Processing pipeline

From delivery to notification

Waiting for the first event

  1. HTTPS webhook receivedWaiting
  2. Signature validatedWaiting
  3. Event deduplicatedWaiting
  4. Event recordedWaiting
  5. Background job queuedWaiting
  6. Application state updatedWaiting
  7. Real-time event publishedWaiting
  8. Client notifiedWaiting

The acknowledgement follows the simulated inbox + pending-job transaction. Background work commits application state and an outbox record together. Motion illustrates the stages; all times and latencies are deterministic model values, not network measurements.

04 / System health

Inside the simulated backend

Webhooks received
0
Events processed
0
Invalid signatures
0
Duplicate deliveries
0
Queue depth
0
Failed jobs
0
Connected clients
1
Application version
0

Queue depth includes failed, retryable jobs. Application version counts committed changes; source sequence orders incoming snapshots. Each is independent of webhook delivery attempts.

05 / Internal event log

Every delivery has a trace

Newest first. Attempts are delivery / worker; latency is modeled per operation.
TimeEvent IDEvent typeResultAttemptsVersionLatency

No matching events yet.

Design notes / Beyond the browser

How I Would Build the Production Version

I would keep receipt, business processing, and delivery separate. A NestJS REST controller accepts Twilio callbacks after a signature guard validates them. Injectable services normalize provider data and enforce application rules. PostgreSQL is the authority for applications, webhook events, pending jobs, and outgoing notifications.

  • Validate before trusting. The simulation uses an explicit valid/invalid flag, not cryptography. Production validates X-Twilio-Signature using the official Twilio SDK, the exact public URL including query parameters, and original request data. Form callbacks and JSON callbacks require their appropriate SDK validation paths. See Twilio’s webhook security guidance.
  • Record before acknowledging. A PostgreSQL unique constraint on (provider, event_id) prevents duplicate inbox rows, even across server instances. Insert the inbox row and pending job in the same transaction, then acknowledge. A worker claims jobs with a lease and retries failed work with bounded backoff and a dead-letter path.
  • Order deliberately. The lab assumes one trusted, application-wide source sequence. Older snapshots are recorded for audit but deferred without changing state. Twilio does not supply a universal application version: real adapters need provider-specific ordering and a server-owned application revision. Arrival timestamps alone are insufficient. Independent calls and document streams need their own ordering rules; a production state machine also validates permitted status transitions.
  • Separate state from notifications. Redis distributes messages across instances; it is not the application database. Socket.io rooms or GraphQL subscriptions deliver notifications to authorized clients. A Next.js dashboard uses authenticated GraphQL queries and mutations. On reconnect it joins its authorized channel, refetches state, and reconciles overlapping notifications by version.
  • Make failures diagnosable. Structured logs, counters, queue age, retry counts, and correlation IDs link receipt to commit to publish. Log identifiers and outcomes, never sensitive personal data, documents, tokens, or full webhook bodies. Authorization applies to every application query, mutation, and room subscription.

The transactional outbox

A database update can succeed while publishing its notification fails. The stored state remains correct, but clients may not hear about it. Writing the state change and an outgoing event in the same database transaction closes that gap. A separate publisher claims outbox records, publishes with retries, and marks them sent only after publication. A crash after publishing can cause a second notification, so consumers still deduplicate by event ID or version. This is at-least-once delivery, not a promise of exactly-once transport.

The lab models that atomic commit and a successful outbox drain. Its worker-failure switch fails before commit; it does not simulate process crashes, disk persistence, network security, or broker outages. Its database is authoritative only within this tab’s lifetime.

One responsibility per tool
TechnologyWhat I would use it for
RESTInbound Twilio webhooks with provider-defined request bodies and acknowledgements.
GraphQLFlexible dashboard queries and authorized application mutations.
Socket.ioNamed live events and application rooms; GraphQL subscriptions are an alternative.
PostgreSQLDurable state, unique event identities, and atomic transactions.
RedisCross-instance event distribution, separate from durable application state.
Background queuesAsynchronous work with retries, backoff, leases, and failure inspection.

Production Sketch

Explanatory TypeScript outlines only; none execute on this site. Imports, DTO definitions, dependency wiring, and repository implementations are omitted. The db methods below describe transaction contracts, not a particular ORM API. Normalized events share the lab’s eventId, provider, eventType, applicationId, sequence, and payload fields. signatureStatus is simulation-only and never trusted from a real request.

1. NestJS webhook controller
@Controller('webhooks/twilio')
export class WebhookController {
  constructor(private readonly events: ApplicationEvents) {}

  @Post('status')
  @UseGuards(TwilioSignatureGuard)
  @HttpCode(202)
  async receive(@Body() form: TwilioStatusCallback) {
    // Adapter validates schema, maps provider IDs to an application,
    // and derives a stable identity for this callback, not just CallSid.
    const event = await this.events.normalizeTwilio(form);
    await this.events.record(event); // inbox + pending job committed
    return { accepted: true }; // acknowledgement, not completion
  }
}
2. Signature guard outline
@Injectable()
export class TwilioSignatureGuard implements CanActivate {
  constructor(private readonly config: WebhookConfig) {}
  canActivate(context: ExecutionContext): boolean {
    const req = context.switchToHttp().getRequest();
    // This endpoint accepts form-encoded callbacks only.
    // Config contains the exact externally configured callback URL.
    const valid = twilio.validateRequest(
      this.config.authToken,
      req.get('X-Twilio-Signature') ?? '',
      this.config.exactPublicCallbackUrl,
      req.body, // original form parameters, before DTO transformation
    );
    if (!valid) throw new ForbiddenException();
    return true;
  }
}

Use the official SDK. Preserve the exact public URL and all original parameters; do not reconstruct it from untrusted forwarded headers. For JSON callbacks, preserve the raw body and use the SDK’s JSON-specific validation method. Secrets belong in server configuration only.

3. Idempotent receipt and worker service
@Injectable()
export class ApplicationEvents {
  constructor(private readonly db: Database) {}
  async record(event: ApplicationEvent) {
    return this.db.transaction(async tx => {
      // INSERT ... ON CONFLICT (provider, event_id) DO NOTHING
      if (!await tx.inbox.insertIfAbsent(event)) return;
      await tx.jobs.insert(event); // same durable transaction
    });
  }
  async process(job: ClaimedJob) {
    return this.db.transaction(async tx => {
      const locked = await tx.jobs.lock(job.id);
      if (locked.done) return;
      const app = await tx.applications.lock(job.event.applicationId);
      if (job.event.sequence <= app.sequence) {
        await tx.jobs.finish(job.id, 'Deferred: Older Event');
        return;
      }
      const next = applyValidatedEvent(app, job.event);
      await tx.applications.save(next); // revision + 1
      await tx.outbox.insert(outboxRecord(job.event, next));
      await tx.jobs.finish(job.id, 'Processed');
    }); // rollback on failure; lease/backoff makes job retryable
  }
}
4. GraphQL application resolver
@Resolver(() => ApplicationView)
@UseGuards(SessionGuard)
export class ApplicationResolver {
  constructor(private readonly apps: Applications) {}
  @Query(() => ApplicationView)
  application(@Args('id') id: string, @CurrentUser() user: User) {
    return this.apps.findAuthorized(id, user.id);
  }
  @Mutation(() => ApplicationView)
  requestReview(@Args('id') id: string, @CurrentUser() user: User) {
    // Check ownership and allowed transition; commit with outbox.
    return this.apps.requestReviewAuthorized(id, user.id);
  }
}
5. Socket.io gateway
@WebSocketGateway({ namespace: '/applications' })
export class ApplicationGateway {
  @WebSocketServer() server: Server;
  constructor(private readonly access: ApplicationAccess) {}

  @UseGuards(SocketSessionGuard)
  @SubscribeMessage('application.watch')
  async watch(@ConnectedSocket() client: Socket,
              @MessageBody() input: WatchApplicationDto) {
    await this.access.assertOwner(client.data.user.id, input.id);
    await client.join(`application:${input.id}`);
    return { joined: input.id }; // client then refetches via GraphQL
  }
  publish(event: ApplicationUpdated) {
    this.server.to(`application:${event.applicationId}`)
      .emit('application.updated', event);
  }
} // Configure a Redis adapter for distribution across instances.
6. Transactional outbox record
function outboxRecord(event: ApplicationEvent, app: Application) {
  return {
    id: `${event.provider}:${event.eventId}:application.updated`,
    topic: 'application.updated',
    applicationId: app.id,
    payload: { applicationId: app.id, version: app.version },
    publishedAt: null,
    attempts: 0,
  };
}
// Insert alongside application update and job completion.
// Publisher claims with a lease, publishes, then marks sent.
// Failed sends retry with backoff; expired leases recover crashes.
// Repeated notifications are safe: clients compare versions/refetch.

Framework references: NestJS resolvers and NestJS gateways. The proposed production architecture is a design sketch, not a deployed service.