GameDuo

Designed a multi-tenant real-time chat server to be shared across games, sizing a 6,000 CCU / 18,000 RPS target and validating it up to 20,002 CCU held for 900s in a dev load test

• No capacity model for chat infrastructure shared across games, blocking server sizing and SLO targets — Based the capacity model on commercial chat solutions' published rate-limit/PCU specs instead of internal guesswork, defining a sizing formula for a single multi-tenant server frame, Adopted 6,000 CCU / 18,000 RPS / 90,000 RPS burst targets into quarterly OKRs; a dev load test held 20,002 CCU for 900s
• Redis Streams WAL hit a single-threaded write bottleneck at 6K msg/s (EngineCPU 99.9%, ack p95 767ms) — Distributed the WAL stream across a 4-shard cluster and compacted entries into native fields (removing Lua serialization) — chose horizontal sharding over vertical scaling, since single-threaded Redis serializes writes and cannot scale them vertically, ack p95 767ms→6.6ms (~116x), WAL backlog 80,000→2,838 cleared, 99.99% accept rate at scale
• Messages were 100% rejected under concurrent multi-tenant load due to a missing scope key — Built a 5-tenant concurrent load-test environment and removed rejects via hot-path log removal and message scope keys, Achieved ack p95 42ms (target <100ms) and 0.04% error rate (target <0.1%) at 299K messages sustained over 5 minutes
• The outbox Lambda stalled at 2 concurrent executions (2% of reserved 100) with IteratorAge flat at 11 minutes despite zero throttles — Instead of batching tweaks, traced the cause to a tenant#date partition key that sent a day of writes to one partition, and spread writes across 10 partitions with a 0–9 shard suffix, In a sandbox load run, IteratorAge 658s→35s (−95%) and Lambda concurrency 2→8
• After hot-channel fanout migration, the bottleneck was not isolated between Redis server Lua, gateway event loop, a single command queue, and topology — Added attribution instrumentation before changing architecture, then validated Valkey serverless SSUBSCRIBE, per-channel subscriptions, and logicalChannelId-sticky connection pools step by step, Passed sandbox validation; a standalone measurement with connection pool size 4 reduced the command queue from 5,940 to 716 and processed 878,205 messages with zero loss
• Offline DMs, unread counts, joined-room lists, and group fanout lacked durable per-user state, risking mixed ownership between chat and game domains — Converged requirements onto a per-user room index, keeping the chat server responsible only for mechanics/characterId while game servers owned profile hydration and group-domain actions, Completed server-side DM, unread-count, joined-room, and group-fanout capabilities under one index model, with server/sandbox QA completed before Unity handoff
• Gateway tasks were repeatedly OOM-killed after performance changes, while outbound queues, observability retention, and heap caps remained competing root-cause hypotheses — Pre-registered and rejected the egress-budget hypothesis, then isolated retained-heap growth and heap-cap alignment through heap snapshots, diagnostic logs, and A/B measurements, Observed zero OOM events in post-fix validation and zero configuration drift in the targeted infrastructure rollback plan

Unified marketing platform managing Google Ads/Meta/TikTok campaigns, creatives, and metrics in a single system, built on dedicated marketing DB separation

• Large marketing-metrics archive backfill stalled at 63s Lambda runtime and 99.8% RDS CPU, ETA 15 hours — Diagnosed the root cause as N+1 queries and a non-SARGable DATE() predicate (index miss) and removed both — chose a single 30-day batch fetch with in-memory latest-version dedup over per-row reads that were saturating RDS, Improved throughput 26x (9.3→247/min), Lambda runtime 63s→1.36s
• MKT Sheet BigQuery reads — Migrated to the Storage Read API, Migration completed
• Slow marketing queries on a 100+ column denormalized table — Normalized table into main/time-series/prediction, tuned indexes + cursor pagination, Performance improved 97% (18s→0.5s)
• Meta asset sync performance bottleneck — Replaced per-row ORM saves — which incurred heavy transaction overhead — with bulk INSERT and batch deletes, Image sync reduced 72% (25.7s→7.1s), DB transactions reduced 95% (10~16s→0.3~0.5s)
• TikTok deployments created orphan campaign/adgroup/ad resources when an external API partially succeeded before internal persistence, or retries were blocked by deployment guards — Unified idempotency per deployment path: Smart+ V2 level-by-level get-or-create with immediate persistence, v1 campaign-level reuse, and regular ad-level flags, Passed 31 unit tests for partial-failure retries and applied a policy to prevent recurrence of 2 orphan campaigns and 2 orphan ad groups
• A 26.37M-row / 27GB audit-log table lacked an index for parent-based lookups, causing page queries to trigger large full scans and prolonged IO saturation — Declared the index at the entity layer to prevent drift, pre-created the online DDL during a low-load window, and hardened the migration with duplicate-index no-op handling plus real DDL integration tests, Eliminated full-scan queries that had run for up to 36 minutes and reduced EXPLAIN-estimated scans to 3 rows, resolving the audit-log lookup incident
• Lambda handler build waits and quarantined test suites delayed marketing-platform CI feedback and left gaps in regression checks — Sharded handler builds, checked for missing expected artifacts, and restored quarantined suites with validation through FULL 8-shard CI, Reduced build wall time from 1,041 to 451 seconds in a manually triggered 1-vs-6-shard workflow experiment with 58 expected handlers and 0 missing; aggregate build work increased from 1,006 to 1,801 seconds. Separately restored all 83 quarantined suites and passed FULL CI
• Missing links between Meta ads and local assets left 2,235 ads with unlinked assets under the same UI filter — Backfilled asset links across 15 Meta accounts and 292 campaigns, comparing before and after under identical UI filters, Ads with unlinked assets 2,235→430: 1,805 fewer (80.8%); the remaining 430 had no asset matching the local ad ID in the current Meta response

Development and operation of NestJS utility (10 modules) + Game Server Kit (2 packages) for multi-project code consistency

• CI TEST job averaged 15m47s, delaying feedback — Chose tsc-native ts-jest isolatedModules with explicit Entity types over @swc/jest, which had weaker decorator-metadata compatibility, Cut average CI TEST job time 61% (15m47s→6m06s); passed 81 suites/978 tests
• Game-server shared code split across 5 branches + one game's 17 game-data definition modules were 2+ years diverged from the published package — Extracted the game-data definition modules + 5 submodules into a standalone package, then migrated 610 files / 310+ imports in one game, and later audited hardcoded names, moved them to config injection, and added a regression guard, Game-data module updates now ship as a single npm version bump (manual migration of 610 files → 0); a live 500 caused by the migration was hotfixed 82 minutes after report
• Each app injected its own static-data LRU cache, so boot-time data warming on live game servers took minutes — Moved static-data storage into the shared kit, removed per-app LRU injection, and added negative caching and preloadAll, Live game-server boot warming 138.1–199.5s → 25.8–37.8s, then 0.82–1.08s after switching the app warmer to preloadAll
• Manual library upgrades taking 3 hours across 7 projects — Added workflow_dispatch+matrix and changed-package CI tests on GitHub Packages, Reduced 7-project upgrade runtime 3h→15min
• Common code duplication across services increasing maintenance cost — Designed 10-module library system: core, repository, cache, lock, slack, crypto, smb, hash, type, iac, Unified shared code across 7 projects with standardized dependency management
• Repository layer had grown beyond 2000 lines and lacked type safety — Replaced read-then-write duplicate-key races with atomic upsert and automatic audit-log branching; narrowed types with overloads/generics and split the Repository into purpose-specific modules per SRP, Refactored 2000+ lines into shared Repository abstraction layer eliminating CRUD duplication
• Game-server cache logic was duplicated per project, increasing consistency and maintenance burden — Combined local LRU and distributed cache in a decorator-based hybrid AOP library to reduce network round-trips without the cross-instance inconsistency of local-only caching; separated tag indexing and single-flight per SRP, Released cache library v0.5.0 — 262 tests across 28 suites passing

Fully isolated the customer-support (CS) chat server onto ECS from a shared deployment, and validated authentication, security, and load ceilings

• CS chat was coupled to a shared deployment, causing resource contention and deployment coupling — Fully isolated the service onto ECS Fargate and automated deployment via Terraform IaC (S3 state), GitHub Actions OIDC, and an observability sidecar — adopted a centralized game-token verification path over per-game distributed checks, Completed independent service cutover with modularized Terraform, an 11-panel Grafana dashboard, and 8 alerts
• Layer coupling and ad-hoc token lifetimes in the new CS chat domain posed security and maintenance risks — Separated responsibilities into a 4-layer Clean Architecture and introduced OAuth2 access (5min)/refresh (1h) tokens with ETag/304 polling — sending only deltas instead of full payloads per poll, Passed 39 tests across 12 suites; applied IaC plan (+10 resources) with zero incidents
• 1.0 vCPU ceiling, 40~90s cold start, and bloated images blocked spike-traffic handling — Redesigned task resources and ALB RPS autoscaling, and cut cold start via native ARM64 builds and Dockerfile slimming with migration separation, Reduced image 381→254MB (-127MB) / build 5→2min / cold start by 30%; 0% failure on a k6 500-VU spike
• No cross-project data isolation, request rate limiting, or attachment access control — Introduced cross-project isolation (triple-validation helper), Redis-backed rate limiting (auth 5/min, refresh 10/min, message 30/min), and an attachment FK integrity migration — rejected in-memory throttling for its multi-instance bypass risk, Merged the security integration with zero type errors and established cross-org/project isolation specs
• Multiple inquiries were mixed inside one CS chat room, making per-inquiry counts and handling-time metrics difficult, while video attachments for bug reports were unsupported — Designed additive APIs for immutable message-reply links, user-level chat lists, and a dedicated video-upload path with MIME, size, TTL, and daily limits, Established message-level reply/target changes, user-level lists, video inquiry playback, and inquiry-count/handling-time aggregation (20MB, 10 uploads/day, 15min TTL)

Resolved batch job limitations from marketing metrics 60-day → 360-day expansion and serverless transition for batch processing

• Full serverless migration risked operational stability — Designed hybrid architecture keeping API server on EC2 while separating batch/job processing to Lambda, Preserved API availability with zero impact, isolating only batch workloads to contain operational risk
• Batch processing limited to 60-day collection range with 2-hour runtime — Established Event-Driven flow with SQS+Lambda+EventBridge, Reduced batch time 2h→5min, expanded collection 6x (60→360 days)
• Data consistency risk during event publishing — Adopted Transactional Outbox Pattern for scheduled and delayed event publishing, Kept data consistent across distributed event processing
• After cache disk-full recovery in the CDK deploy workflow, Lambda handler webpack builds reproduced V8 OOM, combining slower deployments with no regression guard — Separated runtime-transform dependencies into generated artifacts, raised the default handler batch, and retried only failed batches through an adaptive 8→4→2→1 split sequence, Removed the Lambda handler build-OOM root cause, added CI regression guardrails, and validated the deployment path through 3 passing CI runs
• DB connection exhaustion during massive Lambda execution — Introduced RDS Proxy connection pooling, Resolved connection exhaustion and stabilized database access

Redesigned the in-game notification translation domain model — decoupling AI translation from library-based conversion into a 3-stage pipeline

• Simplified Chinese was modeled inside the AI translation settings domain despite not being AI-supported, and settings storage/retrieval depended on the Traditional Chinese AI translation state — making independent toggling impossible without a breaking change — Extracted the derived-conversion module from the AI translation module into 9 use cases with dedicated entity/repository/scheduler, and applied the global-inheritance pattern for effective settings, Clarified domain boundaries by extracting 9 use cases and enabled independent toggling of derived conversion rules
• Decoupling required a breaking API change — Simultaneously deployed FE/BE/Gateway with role permission migration for zero-downtime transition, Completed the breaking API change without incident and cut effective-settings DB queries by 50% (4→2)
• The detect cron lacked version priority, delaying translation of the latest version; 30-second interval further slowed response — Added version ID-based priority ordering to detect SQL, shortened the processing interval from 30s to 10s, and raised batch limit from 100 to 200, Eliminated latest-version translation lag; progress UI plus total-count caching improved user visibility
• In sandbox, the detect scheduler queried all 159 versions at once (full scan of a 3.27M-row table on live), saturating Read IOPS at 190–470/s and locking some users out for 60s — Ported the per-version loop from the AI detect module, added distributed-lock timeout/recovery, and replaced the NOT EXISTS full scan with an indexed staged lookup, Split into per-version queries (39ms for one version on Live EXPLAIN) and bounded each run with pagination and a per-run cap
• The AI translation scheduler stalled on Vertex AI 429 quota errors, leaving 83,796 requests unprocessed on live — Traced the retry path: in-batch and recovery retries compounded to 9 effective attempts, and 13 scheduler instances retried with no backoff or jitter, Identified both causes (a retroactive backfill of 160 versions × 4 languages and a 13-instance retry storm), deleted the 83,796 stuck requests, and split out 4 follow-up tasks

Resolved delayed balance issue response caused by inability to track data change history during game operations

• Inability to track data change history across environments and projects — Designed Git-like version control system with UUID-based cross-environment/project entity tracking, Kept data consistent across 6 games
• Manual entity change recording prone to omission — Applied Event Sourcing-based change tracking with Auditable decorator + TypeORM Subscriber pattern, Automated entity change tracking with consistent logging across all game environments
• Multi-environment version conflicts during data merge — Developed 3-Way Merge Engine with parent/child entity conflict detection and unique constraint handling, Automated version merging across 6 games' environments
• Expensive full-snapshot comparison for every version diff — Designed Version Diff Engine with dual strategy: incremental comparison and snapshot comparison based on Base Audit availability, Dual-strategy version comparison (incremental + snapshot) for performance optimization
• Entity tracking failures due to PK dependency during migration and merge — Introduced shared identifier-based entity tracking decoupled from PK dependency, Tracked entities accurately through migration, comparison, and merge

Built S3-based sync and automated DDL management system to resolve dynamic game data inconsistency across environments

• Dynamic game data inconsistency across environments — Built S3-based cross-environment data synchronization across development/staging/production, Unified game data state across all environments
• Manual DDL schema management causing sync failures — Created automated DDL management engine with dynamic PK column type resolution, column type mismatch detection with MODIFY, and automatic index creation/RENAME, Automated database schema drift detection and reconciliation
• Large-scale Cloud Data ingestion and S3 upload bottlenecks — Analyzed and optimized ingestion and upload pipeline, TRUNCATE→DELETE optimization for batch operations — 77% latency reduction (57.5s→12.9s)
• Sync job instability causing operational issues — Implemented job separation, transitioned scheduling approach, tuned timeouts, and introduced non-blocking processing, Data layer resilience through schema optimization and safe migration strategies
• Four data-sync service bugs: Redis cache invalidation, DDL-SKIP metadata, copy-key deletion, and data-protection option propagation — Diagnosed and fixed all four issues systematically, Diagnosed and fixed 4 critical bugs to stabilize data-sync operations
• Risk of unintended full data deletion when a protection option was not forwarded on some migration paths — Forwarded the data-protection option across the four POST migration paths that were missing it, Blocked unintended full data deletion risk

Built probability calculation package and CDK-based audit log analytics infrastructure to address regulatory risk from lack of game probability verification

• No reusable probability calculation module across 6 live game environments (12 branches) — Packaged NestJS DynamicModule with 5 probability functions + Kinesis logging as shared probability package, Integrated the probability package across 6 live game environments (12 branches), cherry-picking main plus 11 branches
• No audit log analytics infrastructure for probability verification — Codified CDK analytics pipeline: Kinesis→Firehose (Dynamic Partitioning+Parquet)→S3→Glue→Athena, Integrated the probability-disclosure audit-log pipeline across 6 live game environments (12 branches)
• Mock-based tests lacking regression confidence for infrastructure code — Replaced mocks with LocalStack + Testcontainers integration tests, Secured 94%+ coverage (12 suites/115 tests)

Built daily-snapshot archiving of marketing metrics (with immutability), a 4K-row custom dashboard, and alert observability

• Displaying 4K-row marketing metrics in one dashboard caused scroll lag and cell-sync bugs — Split BE (Raw SQL DISTINCT JOIN avoiding ORM hydration + in-memory cache) and FE (visible-row lazy fetch + 200-row chunks balancing network round-trips and render latency + per-pair caching) — rejected full loading due to response size and render cost, Displayed 3,341 of 4,458 rows mapped, stabilized response at 573ms cold / 221ms warm
• Sync paths used the latest snapshot instead of the reference-date cutoff, violating archive immutability (data contamination) — Redesigned the usecase/repository to honor cutoff semantics, made the stale-writer guard transactional, and introduced tombstone states for sync, Rebuilt 15,333 live archives (zero contamination), validated 287,020 archive-date records
• A transient log-query stack (Loki) outage fired 16 marketing alerts at once, flooding ops channels — After diagnosing the root cause, migrated all 16 alerts to a single analytics DB (ClickHouse) query, rewrote them via the dashboard API, and set a legacy log-stack decommission policy, Migrated 16/16 alerts (25 min), preventing alert-flood recurrence during log-stack outages

Games on AWS 2025 Customer Session

Shared a case study of a single engineer scaling data pipeline capacity 270x in 10 days, measured by hourly concurrent job throughput