Production AI & Platform Systems Engineer

From a user perspective: a business configures their tenant (system prompt, business hours, agent persona, appointment types). Their phone number is pointed at a Twilio webhook. When a customer calls, APEX answers, transcribes speech in real-time, generates a context-aware reply with Groq LLaMA, synthesises audio with Azure TTS, and streams it back — in under 1.5 seconds per turn.

Outbound campaigns: a CSV of phone numbers is uploaded. APEX dials each number, delivers a message, records responses, and logs outcomes in Firestore. WhatsApp follows the same pipeline via WAHA.

FastAPI on Python 3.11 — async-first, auto-generates OpenAPI, uses Depends() for auth injection into every handler. Two API pod replicas on AKS with an HPA that scales 2–5 based on CPU. PodDisruptionBudget ensures 1 pod always available during rolling deploys.

Worker pods handle background jobs: outbound campaigns, batch TTS pre-warm, knowledge base indexing. Workers pull from a Redis ZSET priority queue. KEDA ScaledObject watches queue depth for worker autoscaling.

# Dependency injection pattern — auth verified on every handler
@router.post("/api/v1/campaign")
async def create_campaign(
    req: Request,
    user: dict = Depends(get_current_user),  # Firebase JWT → user record
    _admin = Depends(require_admin),
):
    tenant = user["tenant"]
    # All downstream ops scoped to this tenant
        

React 18 + TypeScript + Vite + TailwindCSS + React Query + Zustand + Framer Motion. Deployed on Firebase Hosting alongside the marketing site via a unified build pipeline (npm run build → Vite → sync-marketing.mjs → Firebase deploy).

useAuthStore (Zustand): single source of truth for {user, role, tenant, active}. All React Query keys include tenant to prevent cross-tenant cache leaks — this was a real production bug found and fixed. All pages are lazy-loaded via React.lazy + Suspense; main bundle reduced from 2007KB to 253KB with rollupOptions.manualChunks.

Firestore: Document-model, serverless, zero-ops. Two root collections:

users/{email}:
  active: bool, tenant: str, role: str, plan: str

tenants/{id}:
  name: str, plan: str, owner_email: str,
  system_prompt: str, waha_session: str,
  rate_limit: int, voice: str, language: str
        

Redis key patterns:

rate:{tenant_id}           # daily call counter, TTL = end of day
session:{call_sid}:{tid}   # active call state, TTL = 3600s
llm_cache:{tid}:{sha256}   # LLM response cache, TTL = 86400s
idempotency:{call_sid}     # NX key, TTL = 3600s
job:{jid}                  # ZSET member with priority score
        

File fallback: tenants/*.json and data/users.json activate automatically if Firestore is unreachable. No data loss — degraded mode only.

Jobs are scored ZSET members. Priority = score (lower score = higher priority). Workers call ZPOPMIN to claim the highest-priority job. Dead-letter: jobs that fail N times are moved to dlq:{tenant} ZSET for manual inspection.

KEDA ScaledObject watches Redis queue depth. When queue depth exceeds threshold, KEDA signals the K8s HPA to scale up worker pods. Workers scale to zero when queue is empty — no idle compute cost.

# Job dispatch
await redis.zadd("jobs", {job_id: priority_score})
# Worker poll
job_id = await redis.zpopmin("jobs", 1)
        

Deterministic Python orchestration — no LangChain, no Autogen. The orchestration is a small function that constructs the messages array: system prompt (tenant-specific) + last 8 conversation turns (sliding window) + current user transcript. This predictable structure gives consistent latency and no framework abstraction overhead.

Groq LLaMA 3.3 70B returns streaming tokens. FastAPI accumulates tokens and splits on sentence boundaries (। ? ! . for Hindi/English). Each complete sentence is passed to the TTS pipeline immediately — pipelining TTS generation while LLM continues — reducing perceived latency.

AIMD concurrency control: global semaphore caps total concurrent Groq calls to match API limits. If concurrency exceeds limit, new requests back off (multiplicative decrease). On success, limit increases additively. Per-tenant semaphore ensures no single tenant can starve others.

Redis is configured with maxmemory-policy volatile-lru: only keys with TTL set are eligible for eviction. Session state and LLM cache keys have TTL; idempotency keys have TTL; rate counters have TTL aligned to daily boundaries. Queue ZSET members (no TTL) are never evicted — they are preserved even under memory pressure.

AOF persistence (appendfsync everysec): queue state survives pod restarts. At startup, workers resume processing from where they left off. Session state (TTL 3600s) is acceptable to lose on restart — calls already in-flight will reconnect.

Firebase Auth issues JWTs on login. Backend get_current_user() verifies the JWT, extracts the email, and calls get_or_create_user(email) to fetch the user record from Firestore. The record contains {email, tenant, role, active}.

RBAC: role: "admin" can see all tenants; role: "client" is scoped to user.tenant only. RequireAdmin and require_admin guards are applied at the handler level. Admin emails are configurable via env var — auto-promoted at login time.

Machine-to-machine: X-API-Key header (APEX_API_AUTH_KEY from apex-secrets) for admin operations that don't have a Firebase token context.

Four layers of tenant isolation:

• Data: Firestore documents scoped under tenants/{tid}. Redis keys prefixed with {tenant_id}:. No query can access another tenant's data structurally.
• Auth: Every API handler receives user.tenant from get_current_user(). All downstream queries are scoped to this value. Server-side enforcement — not UI-level.
• Compute: Per-tenant asyncio.Semaphore on LLM requests. One heavy tenant can't starve others of Groq API capacity.
• Rate limits: Per-tenant daily call counters in Redis. Plan tiers (trial/spark/blaze) each have different limits checked before any Twilio call is created.

Client-side: React Query cache keys include tenant — a missing tenant scope was found and fixed in production (cross-tenant data visible to wrong user).

Secrets: 10+ secrets stored in Azure Key Vault. External Secrets Operator (ESO) syncs them into a single K8s apex-secrets Opaque secret every 5 minutes. No secrets in environment files, no secrets in code, no secrets in git history. Firebase service account mounted as a volume from a separate K8s secret.

Network: FastAPI pods listen on 127.0.0.1 — NGINX is the only external entry point. All inter-service traffic is cluster-internal. TLS terminated at the ingress layer only.

Auth depth: (1) Firebase JWT on every user request. (2) Server-side tenant scope on every data access. (3) Admin key for machine-to-machine operations. (4) Kubernetes RBAC for cluster access. (5) Azure RBAC for Key Vault access.

Error tracking: Sentry DSN injected at runtime from apex-secrets. All unhandled exceptions, circuit breaker state changes, and critical path failures are captured with context (tenant, call_sid, endpoint).

Structured logging: log_event() throughout the codebase — consistent JSON fields: event, tenant, call_sid, duration_ms, error. Aggregated via kubectl logs; future migration path to structured log sink.

Metrics: Prometheus counters at /metrics (in-process). Real-time log tail via SSE endpoint in the dashboard — operators can watch call events live without SSH.

Synthetic health: /healthz (liveness) and /readyz (readiness) endpoints — separate concerns. readyz includes Firestore and Redis connectivity checks. Pods are not marked ready until Azure TTS warmup completes (~38s at startup).

Twilio retries webhook delivery on 5xx responses. Without idempotency, a transient API error creates duplicate call sessions. Solution: SET idempotency:{call_sid} 1 NX EX 3600 — the NX flag means only the first request succeeds. Subsequent retries find the key set, skip session creation, and rehydrate the existing session from Redis.

Job queue idempotency: jobs include a content hash as part of their ZSET key. Duplicate job submissions are deduplicated at the ZADD stage. Dead-letter jobs include retry count and original error for diagnosis.

Six named circuit breakers (Groq, Azure TTS, Deepgram, Twilio, Telnyx, WAHA). Each is an independent state machine: CLOSED → OPEN (after N failures) → HALF_OPEN (after timeout) → CLOSED (if probe succeeds) or OPEN (if probe fails).

Thresholds differ by provider: Azure TTS opens after 3 failures (silent audio to caller is worse than a slow response). Groq opens after 5 failures. When OPEN, requests fail immediately (no 30s timeout wait) — fast failure is the primary value of circuit breaking, not recovery.

Barge-in: TTS send tasks check a per-call generation token at every audio chunk boundary. When a caller speaks mid-response, the token is invalidated — in-flight chunks abort themselves and a clear event is sent to Twilio to stop playback.

GitHub Actions (.github/workflows/cd-aks.yml) triggers on every push to master:

1. Build Docker image with tag YYYYMMDD-{sha7}
2. Push to ACR (apexacrprod.azurecr.io) — both tagged and :latest
3. kubectl set image deployment/apex-api api={image} -n apex
4. kubectl set image deployment/apex-worker worker={image} -n apex
5. kubectl rollout status --timeout=120s — fails fast if pods don't come up

Total: ~4 minutes from git push to production traffic on new image. Rollback: kubectl rollout undo deployment/apex-api -n apex. Probe timings: initialDelaySeconds: 60 on both liveness and readiness — Azure TTS warmup for all tenants takes ~38s at startup.

The speech-to-speech hot path per call turn:

Phone → Twilio → POST /twilio/s2s/voice (webhook)
  → FastAPI returns TwiML: <Connect><Stream url=wss://...>
  → Twilio opens WebSocket to /twilio/s2s/stream/{call_sid}
  → FastAPI opens Deepgram WebSocket (nova-2, hi-IN, 16kHz)
  → Twilio streams mulaw audio in 20ms chunks
  → FastAPI converts mulaw→PCM → Deepgram
  → Deepgram returns transcript (speech_final, ~250ms endpointing)
  → FastAPI builds messages array (last 8 turns + system prompt)
  → Groq streaming request (LLaMA 3.3 70B)
  → Split on sentence boundary → TTS cache lookup (SHA256)
  → Cache HIT: read .ulaw sidecar → stream chunks to Twilio
  → Cache MISS: Azure TTS → MP3 → decode → .ulaw → cache → stream
  → Twilio plays audio to phone caller
        

See the Engineering Decisions section for full decision log. Summary of the five highest-impact decisions:

• Redis as universal backbone: eliminates separate message broker + cache + session store. Single operational decision collapses five engineering decisions.
• Groq over OpenAI: 10× lower inference latency at 10× lower cost — only viable choice for real-time voice at current scale.
• AKS over App Service: ~$60/month for HA setup vs $200+. Kubernetes ops overhead mitigated by full CI/CD automation.
• Deterministic orchestration over LangChain: predictable latency, no framework abstraction tax, full control over context window and streaming behaviour.
• Firestore over Postgres: serverless, zero-ops, JSON document model matches tenant config structure. No connection pooling to manage.

State transitions must be atomic: The onboarding loop bug (users stuck re-onboarding forever) was caused by a gap between creating a resource and updating the user record. Two operations, not one atomic unit. Fix: always complete the state machine in a single function call.

Model tier limitations explicitly: The WAHA 422 crash happened because WAHA Core's single-session constraint was not modelled in the code. 422 was treated as a generic server error. Fix: tier limitations are a first-class concern — model them as named statuses, not exceptions.

Every cache key is a potential data boundary: The cross-tenant cache leak was a React Query key missing a tenant scope. It's not just a bug — it's a design principle. Every cache key is a data boundary decision. Apply tenant scope everywhere, not selectively.