API Architecture

Purpose

Deep-dive into apps/api — the Node.js + Express server that receives webhooks, routes intents, captures leads, escalates, notifies, and serves the admin dashboard's REST API. Start here if your task touches routes, middleware, persistence, intent routing, or realtime events.

Prerequisites

Architecture overview — system-level context
Working local env with the API running (see setup)

Entry point and layout

Text Only

apps/api/
├── src/
│   ├── index.ts                    # Express bootstrap, middleware mount
│   ├── routes/
│   │   ├── webhook.ts               # POST /webhook/whatsapp
│   │   ├── webhookHelpers.ts        # Dedup, save message, escalation, opt-out, D360 delivery statuses
│   │   ├── webhookPipeline/         # leadCaptureStep + postDispatchStep
│   │   ├── api.ts                   # Public-ish helpers (campaign redirect /c/:code)
│   │   ├── agent.ts                 # /api/v1/agent/*
│   │   ├── leads.ts                 # /api/v1/leads (legacy)
│   │   ├── leadsV2.ts               # /api/v1/leads/v2/* (filter, summary, detail, patch, activity, workload)
│   │   ├── kb.ts                    # /api/v1/kb/* (documents, eval, analytics, gaps)
│   │   ├── kbDatasets.ts            # /api/v1/kb/datasets/* (cluster KB datasets)
│   │   ├── faqLocal.ts              # /api/v1/faq (local CRUD + Dify sync)
│   │   ├── followUp.ts              # /api/v1/follow-up/*
│   │   ├── feedback.ts              # /api/v1/messages/* (thumbs, correction)
│   │   ├── intents.ts               # /api/v1/intents/*
│   │   ├── escalationRules.ts       # /api/v1/escalation-rules/*
│   │   ├── campaigns.ts             # /api/v1/campaigns/*
│   │   ├── settings.ts              # /api/v1/settings/{guard,whatsapp,faq-threshold}
│   │   ├── analytics.ts             # /api/v1/analytics/* (legacy)
│   │   ├── analyticsV2.ts           # /api/v1/analytics/v2/* (13 charts)
│   │   ├── dashboardCounselor.ts    # /api/v1/dashboard/counselor/*
│   │   ├── monitoring.ts            # /api/v1/monitoring/judge (Phase 1 Quality Gate)
│   │   ├── adminUsageStats.ts       # /api/v1/admin/usage-stats (Feature Coverage; super_admin only)
│   │   ├── eval.ts                  # /api/v1/eval/* (golden-question + run management)
│   │   ├── crawl.ts                 # /api/v1/crawl/* (KB crawler proxy + jobs/[id]/stream)
│   │   └── zitadelActions.ts        # POST /webhook/zitadel/pre-token (HMAC-signed)
│   ├── services/
│   │   ├── intentRouter/
│   │   │   ├── types.ts              # Intent, HandlerResult
│   │   │   ├── classifier.ts         # 4-tier classifier
│   │   │   ├── handlers/             # wantRegister, sharePersonalInfo, ...
│   │   │   ├── escalationRules/      # Rule engine + actions
│   │   │   └── leadCapture/
│   │   │       ├── extractor/        # regex + llm + index
│   │   │       ├── stateMachine.ts
│   │   │       ├── clusterResolver.ts
│   │   │       └── leadStore.ts       # Atomic upsert
│   │   ├── leadScoring.ts            # GPT-4o-mini scoring at milestones
│   │   ├── leadActivity.ts           # Timeline event logger (best-effort)
│   │   ├── faqKbSync.ts              # FAQ → Dify Milvus KB sync (debounced)
│   │   ├── ragClient.ts              # Dify chat dispatch + persona + multi-Q
│   │   ├── funnelEngine.ts           # Auto-transition + manual validation
│   │   ├── counselorAssignment.ts    # Round-robin auto-assign
│   │   ├── conversationContext.ts    # LLM context builder (5 msgs + lead)
│   │   ├── conversationGuard.ts      # Profanity 3-strike + repetition + cooldown
│   │   ├── followUpAI.ts             # Claude Haiku generates contextual follow-up highlight
│   │   ├── followUpEngine.ts         # Enqueue (score-aware) + process + expire follow-ups
│   │   ├── mediaDownloader.ts        # Download WhatsApp media from 360dialog → local disk
│   │   ├── notifications/
│   │   │   └── escalationNotifier.ts # Fan-out to cluster + globals (kind=escalation/notify)
│   │   └── realtime/
│   │       ├── server.ts              # Socket.io server
│   │       ├── pgBridge.ts            # Postgres LISTEN → Socket.io
│   │       ├── auth.ts                # JWE decode (uses shared verifier)
│   │       └── rooms.ts               # Room resolver
│   ├── clients/
│   │   └── langfuse.ts                # Fetches Dify chatflow traces + judge decisions
│   ├── auth/
│   │   ├── types.ts                   # AuthMode (with parseAuthMode), VerifiedSession
│   │   ├── hmacVerifier.ts            # Layer 1 — HMAC constant-time
│   │   ├── jwtVerifier.ts             # NextAuth JWE decode (canonical)
│   │   └── zitadelJwtVerifier.ts      # Zitadel JWKS-based access token verification
│   ├── middleware/
│   │   ├── dispatchAuth.ts             # Top-level mode dispatcher (legacy / both / zitadel)
│   │   ├── requireInternalSecret.ts    # Layer 1 — HMAC gate
│   │   ├── requireForwardedSession.ts  # Layer 2 — X-Forwarded-Session decode
│   │   ├── requireZitadelJwt.ts        # Bearer token verification via JWKS (audit-logged)
│   │   └── auditSensitiveAccess.ts     # Sensitive pattern + write op audit
│   ├── jobs/
│   │   ├── nightlySync.ts              # Cron: KB sync, eval, follow-up, retention, gap detection
│   │   └── alertsSweep.ts              # Every 5 min — hot_lead_wait + cron_failure alerts
│   ├── audit/
│   │   ├── logAuthEvent.ts             # Fire-and-forget audit_log writer (auth events)
│   │   └── logFeatureAccess.ts         # Generic feature-access writer (monitoring.*, etc.)
│   ├── db/
│   │   └── schema.ts                   # Drizzle ORM schema
│   └── __tests__/                      # Jest tests (~720 across 62 suites)
├── jest.config.js
├── package.json
└── tsconfig.json

Route map

Path	Handler	Auth	Notes
`GET /`	—	exempt	Landing
`GET /health`	—	exempt	Simple health check
`POST /webhook/whatsapp`	`routes/webhook.ts`	exempt	360dialog webhook (no HMAC signing — tier doesn't expose App Secret)
`GET /webhook/whatsapp`	—	exempt	`hub.verify_token` bootstrap
`POST /webhook/zitadel/pre-token`	`routes/zitadelActions.ts`	HMAC-signed (Stripe-style over rawBody)	Zitadel pre-token Action — adds `urn:huph:role` + `urn:huph:cluster_id` claims
`GET /c/:code`	`routes/api.ts`	exempt (public)	Campaign redirect — logs click, 302 to WhatsApp
`GET /api/v1/health/realtime`	—	exempt	Socket.io + pgBridge status
`GET /api/v1/health/full`	—	exempt (allow-list)	Deep health + message throughput (info disclosure — sweep monitor only)
`GET /api/v1/health/crons`	—	gated	Cron job execution history + failures 24h
`/api/v1/agent/*`	`routes/agent.ts`	gated	Agent inbox messages, reply
`/api/v1/leads`	`routes/leads.ts`	gated	Legacy leads
`/api/v1/leads/v2/*`	`routes/leadsV2.ts`	gated	Leads (filter, summary, detail, patch, activity timeline, export, workload, counselors)
`/api/v1/kb/*`	`routes/kb.ts`	gated	KB documents, eval, sources, gaps
`/api/v1/kb/datasets/*`	`routes/kbDatasets.ts`	gated	Cluster KB dataset CRUD
`/api/v1/faq`	`routes/faqLocal.ts`	gated	Local FAQ CRUD + Dify KB sync
`/api/v1/follow-up/*`	`routes/followUp.ts`	gated	Queue, rules CRUD, stats, preview (dry-run)
`/api/v1/messages/*`	`routes/feedback.ts`	gated	Thumbs up/down, correction
`/api/v1/intents/*`	`routes/intents.ts`	gated	Intent classification log + rules + export/import + test
`/api/v1/escalation-rules/*`	`routes/escalationRules.ts`	gated	Rule CRUD
`/api/v1/campaigns/*`	`routes/campaigns.ts`	gated	Campaign tracking links (CRUD, QR, stats)
`/api/v1/settings/guard`	`routes/settings.ts`	gated	Conversation guard config
`/api/v1/settings/whatsapp`	`routes/settings.ts`	gated	WhatsApp channel info + health + test message
`/api/v1/settings/faq-threshold`	`routes/settings.ts`	gated	FAQ Dify threshold knob (PUT flag-gated)
`/api/v1/analytics/*`	`routes/analytics.ts`	gated	Legacy analytics
`/api/v1/analytics/v2/*`	`routes/analyticsV2.ts`	gated	13 chart endpoints
`/api/v1/dashboard/counselor/*`	`routes/dashboardCounselor.ts`	gated	Counselor-scoped dashboard
`/api/v1/monitoring/judge`	`routes/monitoring.ts`	gated (audit-logged)	Phase 1 Quality Gate — Langfuse-backed judge decisions
`/api/v1/admin/usage-stats`	`routes/adminUsageStats.ts`	gated (super_admin only)	Feature Coverage aggregation
`/api/v1/eval/*`	`routes/eval.ts`	gated	Golden questions + run management
`/api/v1/crawl/*`	`routes/crawl.ts`	gated	KB crawler proxy + `jobs/:id/stream` SSE

Middleware chain

Text Only

Incoming request
     ↓
Helmet (security headers)
     ↓
CORS
     ↓
express.json({ limit: '50mb', verify: (...) => req.rawBody = buf })
     ↓  (verify hook captures raw bytes BEFORE parsing — used by
         webhook HMAC when enabled)
Request ID + Pino logger
     ↓
dispatchAuth — top-level mode router
   ↓ (mode = legacy | both)
Layer 1 — requireInternalSecret (HMAC gate, 3-mode)
   ↓
Layer 2 — requireForwardedSession (NextAuth JWE decode)
   ↓ (mode = zitadel | both)
requireZitadelJwt — Bearer token JWKS verify (audit-logged)
   ↓
auditSensitiveAccess (pattern + write-op audit)
     ↓
Route handler
     ↓
Zod validation (request body / params)
     ↓
Business logic
     ↓
Error handler

Auth layer details

API_AUTH_MODE=disabled — middleware code is deployed but gates are no-ops. Zero behavior change. Used during a fresh rollout window.
API_AUTH_MODE=warn — middleware logs auth failures to audit_log but allows requests. 24-72h observation phase.
API_AUTH_MODE=enforce — blocks invalid requests with 401. Steady-state production mode.
API_AUTH_MODE=legacy — dispatchAuth routes only through the HMAC + JWE path (Layer 1 + Layer 2). Used during the Zitadel rollout when not all clients carry Bearer tokens yet.
API_AUTH_MODE=both — dispatchAuth accepts EITHER the legacy path OR a Zitadel Bearer token. Steady-state during hybrid auth.

Note: parseAuthMode() (apps/api/src/auth/types.ts) maps legacy and both to enforce for the inner middlewares. A P0 bug pre-2026-04-24 made the inner gate fall through to disabled and silently bypass auth — fix landed in a1885cf.

Rollback at any phase is a sub-30s env flip:

Bash

sed -i 's|^API_AUTH_MODE=.*|API_AUTH_MODE=disabled|' /opt/huph/.env
docker compose up -d --no-deps huph-api

Writing audit events

Two helpers exist:

apps/admin/src/lib/audit.ts:logAudit — Prisma-based. Default choice from Next.js route handlers; it parses request headers for IP/UA/device/session.
apps/api/src/audit/logAuthEvent.ts:logAuthEvent — raw-SQL, used by Express middleware and by services that don't have a Request object.

Both accept optional beforeState / afterState fields for actions that mutate a resource and benefit from a diff view (threshold flips, rule edits, config updates). Example:

TypeScript

await logAudit({
  userId: session.user.id,
  action: 'escalation_rule.update',
  resourceType: 'escalation_rule',
  resourceId: rule.id,
  beforeState: { threshold: oldRule.threshold, enabled: oldRule.enabled },
  afterState:  { threshold: newRule.threshold, enabled: newRule.enabled },
  request,
});

Semantics:

Omit both fields for non-mutating actions (reads, auth events, notifications).
Pass beforeState: null when creating a resource (no prior state).
Pass afterState: null when deleting a resource.
Store only the fields that actually changed + enough identifying context — full resource snapshots waste storage and leak PII.

Columns are jsonb NULL in Postgres (added 2026-04-24 for Phase 4 prep). See docs/superpowers/plans/2026-04-24-phase4-audit-log-state-columns.md.

Key subsystem walkthroughs

Intent Router

4-tier classifier in services/intentRouter/classifier.ts:

Deterministic regex — whitelist phrases like "mau daftar", "minta bantuan manusia"
Keyword + heuristic — scoring based on language patterns
Claude Haiku — LLM classification as fallback
Default handler — generic information intent

Handlers in handlers/ — wantRegister, wantVisitCampus, sharePersonalInfo, etc. Each handler returns a HandlerResult with intent, entities, escalation?, notifyAdmin?.

Lead Capture Phase 2A

Pipeline in services/intentRouter/leadCapture/:

extractor/regex.ts — Indonesian patterns (nama saya X, panggil X, phone 62xxx)
extractor/llm.ts — Claude Haiku via Vercel AI SDK + Zod
extractor/index.ts — gate decides regex vs LLM
stateMachine.ts — awaiting_name → awaiting_email → captured, 6h TTL
leadStore.ts — atomic INSERT ... ON CONFLICT with CASE-based status recompute

See the project_lead_capture_phase2a memory for the full truth table and gotchas.

Lead Scoring

services/leadScoring.ts — milestone-based scoring via GPT-4o-mini.

Triggers at message counts 3, 5, 7, then every 5 messages
Skips if scored within last 2 minutes (scored_at check)
Uses conversationContext.ts to build structured LLM context
Returns { score: 0-100, label: hot|warm|cold, reason, program_interest, campus_interest }
Clamps score to 0-100, corrects invalid labels from LLM
Logs score_change and hot_lead_alert to lead_activity
Notifies counselor when score upgrades to HOT

Activity Timeline

services/leadActivity.ts — chronological event log for lead lifecycle.

logActivity() — best-effort INSERT (try/catch, never throws)
getTimeline() — paginated query by lead_id, newest first
8 event types: score_change, status_change, cluster_change, counselor_assigned, counselor_released, funnel_change, message_milestone, hot_lead_alert
Instrumented in: leadScoring, counselorAssignment, webhook, leadCaptureStep, postDispatchStep, leadsV2 PATCH
API: GET /api/v1/leads/v2/:id/activity?limit=20&offset=0
Admin: timeline component on lead detail page with icons + load more

FAQ KB Sync

services/faqKbSync.ts — keeps Dify Milvus KB in sync with local FAQ.

scheduleFaqKbSync() — debounced 5s after last change
Generates markdown document from all active faq_local records
Updates existing Dify KB document, or creates new one
Caches document ID after first lookup (module-level)

Realtime substrate

Single pg.Client in realtime/pgBridge.ts LISTENs on huph_events and forwards events to Socket.io rooms scoped by user:, role:, cluster:, conversation:, global:.

5 DB trigger functions: notify_message_event, notify_conversation_event, notify_lead_event, notify_notification_event, notify_followup_event. All use AT TIME ZONE 'UTC' to avoid the WIB timestamp bug.

Health: GET /api/v1/health/realtime returns pgBridge.connected, socketio.namespaces, eventCount.

Notification dedup (recent)

Recent composite index (user_id, event_key, created_at) on notifications enforces dedup for the rapid-fire fanout 30s window. See commit 8ce2c28 on main.

Gotchas (permanent)

NextAuth uses JWE, not signed JWT. Use next-auth/jwt encode/decode, never jsonwebtoken.sign/verify. Silent failure otherwise.
express.json() runs before route middleware. If you need raw bytes for a webhook HMAC, use the verify hook on the global express.json() to capture req.rawBody. Pattern from Stripe/Twilio webhook recipes.
docker-compose.yml needs explicit env passthrough. Writing to .env isn't enough — the container reads from the environment: block unless env_file: is set. Add ${VAR:-default} passthroughs for every new env var.
Don't duplicate JWT decode. jwtVerifier.ts is the single source of truth. Both HTTP middleware and Socket.io auth.ts delegate to it.
360dialog tier doesn't expose App Secret. Webhook HMAC signing was implemented and then removed in commit 419fe01. Don't try to add it back until a tier upgrade.
Notification dedup window is 30s. If 2 escalation rules fire within 30 seconds on the same (user, event_key), only one row persists. Before the composite index, this was a fanout storm.
api.auth.* audit rows require API_AUTH_MODE != disabled. In disabled mode the middleware is a no-op and writes nothing.