Hyeonjun Kil

Backend Engineer / Platform Part Leader

kboxstar@gmail.comGitHub ↗LinkedIn ↗Web Portfolio ↗

4 Featured Case Studies

Multi-tenant Real-time Game Chat — Fixing Write Bottlenecks Through Key Design

2026.04 — Present · Platform Part lead · led server design, implementation, and load validation · Team

Multi-tenant · 6,000 CCU sustained target · 20,002 CCU validated in dev · 4-shard Valkey

A chat server designed to be shared by several games: channel-level Valkey sharding cut ACK p95 from 767.6ms to 6.6ms, and a dev load test held 20,002 CCU for 900s.

TypeScriptValkey ClusterRedis StreamsDynamoDBAWS ECS FargateAWS LambdaTerraformk6Grafana

Performance Metrics	Before	After
ACK p95 (6K msg/s)	767.6ms	6.6ms (Single node → 4-shard cluster)
Dev load held	6,000 CCU target	20,002 CCU · 900s (3.3x the target concurrency sustained)
Accept rate	99.39%	99.994% (4,787 → 5,912 msg/s)
Outbox IteratorAge	658s	35s (10-shard partition key)
Gateway command queue	5,940	716 (Split connection pool · zero loss in 878,205 messages)

Problem Solving

Under a fixed 6K msg/s load, a single Valkey node pinned EngineCPU at 99.9%, pushing ACK p95 to 767.6ms with an 80,000-entry WAL backlog. Valkey runs commands on one thread, so more cores would not help.

Picked the shard from a stable hash of the logical channel ID and gave the WAL stream and the channel's seq and dedupe keys the same hash tag, so a channel's multi-key append still completes atomically on one slot in a 4-shard cluster. A shard count of 1 keeps the legacy key shape.

→ ACK p95 767.6ms → 6.6ms, accept rate 99.39% → 99.994% (4,787 → 5,912 msg/s), WAL backlog 80,000 → 2,838, hottest shard EngineCPU 74.7%

The outbox Lambda sat at 2 concurrent executions (2% of its reserved 100) with zero throttles while IteratorAge stayed flat at 11 minutes. Tuning the batching window did not help.

Traced the cause to a tenant#date partition key that sent a whole day of writes to one partition and therefore one stream shard, then added a 0–9 shard suffix to spread writes across 10 partitions.

→ In a sandbox load run: IteratorAge 658s → 35s (−95%) and Lambda concurrency 2 → 8. When a 4K CCU run later showed 37K outbox-table write throttles, the outbox was removed in favor of stream CDC from the messages table.

Ordering accept slowed under hot-channel load, but the bottleneck could not be separated among Valkey Lua, the gateway event loop, the command queue, and topology.

Added attribution instrumentation before changing anything, cross-checked ECPU per message, command queue depth, and event-loop delay under a fixed 500 CCU channel and staged load, narrowed the cause to one connection plus a saturated 0.5 vCPU event loop, then split the connection pool.

→ With a pool of 4: command queue 5,940 → 716, zero loss across 878,205 messages, sandbox verification 19/0/0

After the performance work, gateway tasks kept getting OOM-killed, and the first hypothesis (outbound queue buildup) was rejected when the validation run showed zero triggers.

Registered hypotheses up front, then used heap snapshots, diagnostic logs, and A/B runs to separate retained-heap growth from the observability stack (statsd DNS lookups, hot-path instrumentation) and a mismatch between the V8 heap limit and the 1 GiB container memory.

→ OOM events dropped to zero after the fix

We needed evidence that connections hold beyond the 6,000 CCU target and survive a cache node scale-up.

Ran a 20K CCU load test in dev and, in a separate run, measured an online Valkey node scale-up.

→ Held 20,002 CCU for 900s. In the separate scale-up run, no connections were lost over 16m39s; 23 sends were rejected by backpressure and recovered through client retries.

Tech Decisions

•Sharding over vertical scaling: Valkey executes commands on one thread, so more cores do not relieve a write bottleneck. The trade-off is that a channel is pinned to one shard.
•Logical channel ID as the shard key: in-channel ordering is a requirement, so routing is per channel, and the WAL and channel state share one hash tag to keep multi-key operations atomic.
•Only the Valkey append happens before ACK; a separate writer persists to DynamoDB, separating client latency from database writes and letting the consumer group replay persistence.
•ARM64: a same-load comparison at 6,000 CCU showed 15% lower CPU and 54% lower memory, and multi-platform images support both x86 and ARM64.

Operating limits

•All figures come from sandbox/dev load tests, not live traffic or total live capacity.
•A channel is pinned to one shard, so writes to a single hot channel do not spread when shards are added.
•Per-VU memory limits in the k6 JS client prevented measuring the post-sharding server's throughput cliff; results only show that target loads were sustained.

Highlights

•Spread a single-node WAL bottleneck under a fixed 6K msg/s load across a 4-shard cluster: ACK p95 767.6ms → 6.6ms, accept rate 99.39% → 99.994%
•Shard key is a stable hash of the logical channel ID; the WAL stream and channel-state keys share one hash tag so the append stays atomic
•Replaced an outbox partition key that sent a whole day's writes to one partition with a 10-shard suffix: IteratorAge 658s → 35s
•Held 20,002 CCU for 900s in a dev load test; a separate run kept zero lost connections during an online cache scale-up

Lessons Learned

•Write bottlenecks in a single-threaded store are solved with key design (sharding, hash tags), not vertical scaling.
•A date-only partition key sends a whole day of writes to one partition; look at key design before tuning settings.

Multi-tenant Real-time Game Chat — Fixing Write Bottlenecks Through Key Design architecture — Multi-tenant Real-time Game Chat — Fixing Write Bottlenecks Through Key Design — System architecture

Database Saturation Root-Cause Analysis — Three Cases Fixed in Queries and Indexes

2026.04 — 2026.06 · Platform Part lead · hands-on root-cause analysis, fixes, and recurrence guards · Team

Detail ↗

MySQL (RDS) · 26.37M-row / 27GB audit log · 3.27M-row translation table · 15,333-item backfill

Traced three RDS CPU/IO saturation cases to the offending query with Performance Insights and EXPLAIN: backfill throughput up 26x (9.3 → 247 items/min), and full scans of up to 36 minutes on a 26.37M-row audit log removed (EXPLAIN rows = 3).

MySQLAWS RDSPerformance InsightsTypeORMAWS LambdaTypeScript

Performance Metrics	Before	After
Backfill throughput	9.3 items/min	247 items/min (26x · 15,333 completed, 0 pending)
Lambda run (p50)	63.2s	1.36s (p99 67.0s → 2.99s)
Audit-log query (live)	Full scans up to 36 min	EXPLAIN rows=3 (Index restored and declared on the entity)
Translation detection run (sandbox)	100–224s · overlapped the 10s interval	Per-version queries · 500 cap per run (Live EXPLAIN: 39ms for one version (bundled 10.1s))

Problem Solving

During a marketing-metrics archive backfill, Lambda runs took 63s and throughput sat at 9.3 items/min, putting the remaining work at a 15-hour ETA. The first hypothesis was a large (540MB) bulk-insert transaction, but RDS CPU was pinned at 99.8%.

Performance Insights showed 99.5% of the load coming from one query that looked up the latest result per day. It combined an N+1 loop over 30 days, a DATE()-wrapped date condition that could not use the index, and a select of the full compressed result column. I replaced it with one IN() batch for the 30 days, direct date comparison, and a minimal column list, then picked the latest version per day in memory. Bulk inserts were also split into chunks of 500.

→ Lambda p50 63.2s → 1.36s (p99 67.0s → 2.99s), throughput 9.3 → 247 items/min (26x). The backfill finished with 15,333 completed and 0 pending, and once the work was done RDS CPU sat idle at 5–7%.

Each open of the audit-log screen in the live console full-scanned a 26.37M-row, 27GB table; four concurrent queries piled up for as long as 36 minutes, saturating IO and delaying INSERT commits.

Tracing why the composite index on the parent-ID condition had disappeared showed that an earlier auto-generated migration treated it as drift, because the entity never declared it, and dropped it. The recovery migration's CREATE INDEX also had an ALGORITHM/LOCK syntax error. I declared the index on the entity so it cannot be dropped again, built the 27GB index with online DDL in a low-traffic window first, and made the deploy migration a no-op when the index already exists.

→ Confirmed the index exists on live and the migration is recorded; EXPLAIN changed to type = range, rows = 3, ending the full-scan incident. An integration test that runs the real DDL was added.

In sandbox, a translation-change detection scheduler saturated DB Read IOPS at 190–470/s (normally under 10/s); login handling exceeded 10 seconds, hit the load balancer timeout, and some users could not log in for about 60 seconds.

The scheduler bundled 159 versions into one IN condition and full-scanned the translation word table (3.27M rows on live); it ran every 10 seconds while each query took 100–224s, so runs overlapped and piled up. I ported the per-version pattern from the AI translation detector, running one query per version with pagination and a 500-request cap per run to bound each run. The fix shipped as a shared-package release, and applying it to the game server was checked for backward compatibility with a byte-level comparison of 1,073 fields.

→ On live EXPLAIN, the per-version query took 39ms for one version, while the old bundled query over 159 versions took 10.1s. Running every version does not shrink a run's total query time much, so the real fix is that pagination and a per-run cap bound the length of each run. All integration tests passed.

Tech Decisions

•Measurement over hypothesis: the backfill was suspected of a large transaction, but Performance Insights pointed at one read query, so the fix went to the read query's shape, not the write path.
•Declare indexes on the entity: an index that exists only in a migration can be dropped as drift by the next auto-generated migration, so the entity is the schema source of truth.
•Keep the 27GB online DDL out of the deploy: build it manually in a low-traffic window, and let the deploy migration skip an existing index, so no long DDL runs during a deploy.
•Cut periodic work small: per-version queries, pagination, and a per-run cap bound the length of each detection run.

Operating limits

•For the backfill, only the 99.8% pre-fix RDS CPU is used as a baseline; the 5–7% seen afterwards was an idle reading after the work finished.
•The translation-detection case was a sandbox incident. 39ms is a single-version sample on live EXPLAIN; a run that covers every version does not take much less total query time.
•The audit-log case ended with restoring the index; retention and partitioning for the large audit log were out of scope.

Highlights

•RDS CPU pinned at 99.8% during a backfill: Performance Insights put 99.5% of the load on one read query; removing its N+1 pattern and DATE() index miss raised throughput 9.3 → 247 items/min (26x) and cut Lambda p50 63.2s → 1.36s
•Live incident: an index dropped as drift by an auto-generated migration left a 26.37M-row audit log open to full scans of up to 36 minutes; after restoring it, EXPLAIN shows rows = 3
•Sandbox incident: a 10-second scheduler bundled 159 versions into one query that ran 100–224s, overlapped itself, and saturated Read IOPS at 190–470/s; per-version queries and a per-run cap now bound each run
•Recurrence guards: declared the index on the entity so it cannot drift away, and added a real-DDL integration test and a 500-request cap per run

Lessons Learned

•A first hypothesis stays a hypothesis until measured; the query behind 99.5% of the load was not where I first looked.
•An index that lives only in a migration can vanish as drift at any time; keep the schema source of truth on the entity.

Shared Package & CI Platform — Fork Realignment, 7-Project Upgrade Automation, Faster Test CI

2026.02 — 2026.04 · Backend developer, then Platform Part lead · hands-on package migration, upgrade automation, and CI work · Team

Detail ↗

7 projects, up to 12 branches · 17 data modules · 610 files · 81 suites / 978 tests

Realigned a game server's data-module fork that had drifted for 2+ years onto the shared npm package (610 files, 310+ imports migrated), automated upgrades across 7 projects with GitHub Actions (about 3 hours by hand → about 15 minutes), and cut the shared server kit's average test job time 61% (15m47s → 6m06s).

TypeScriptNestJSnpm packagesGitHub ActionsJest (ts-jest)TypeORM

Performance Metrics	Before	After
Average test job	15m47s	6m06s (61% faster · whole CI about 16 → 7 min)
Package upgrade	About 3h by hand	About 15 min automated (7-project matrix in parallel)
Module alignment	Fork drifted 2+ years	npm version bump (610 files and 47 migrations deleted)
Migration regression	Live 500 errors reported	Hotfix in 82 min (9 related tasks tracked)

Problem Solving

One game server had forked the shared data modules and gone 2+ years without realigning with upstream, so replacing its 17 internal modules with the evolving npm package meant moving 610 files and 310+ imports.

Using the alignment approach other games already followed, I split the work into two stages: first switch the internal modules to the package, then bring it to the latest version with the same pattern and package-managed migrations. Game-specific features (localization, forbidden words, dynamic environments, and so on) stayed in six thin wrapper modules.

→ Deleted the 610 internal-module files and 47 local migrations; new features can now come in with a package version bump. When the migration caused a regression that returned 500 errors in a live admin feature, a hotfix shipped 82 minutes after the report, and the work was tracked as 9 related tasks including the migration itself.

Every shared-package release meant upgrading the main/develop branches of 7 projects (up to 12 branches) by hand, taking about 3 hours plus the cost of context switching.

Built a GitHub Actions workflow triggered by workflow_dispatch that processes the 7 projects in parallel with a matrix, recording each project's outcome as skip/upgrade/downgrade in the Job Summary.

→ Upgrades went from about 3 hours by hand to about 15 minutes automated.

In the shared server kit, ts-jest compilation took about 15s per file, 53% of test time, so even pure unit tests took 15–19s.

Switched ts-jest to isolatedModules so each file is transformed on its own, leaving type checking to a separate tsc --noEmit step in CI. In this mode decorator metadata cannot see types from other files, so I declared column types explicitly on the 4 affected entities.

→ Per-file compile 15s → 1–3s, average test job across 4 shards 15m47s → 6m06s (61% faster), whole CI including build and quality checks about 16 → about 7 minutes. All 81 suites / 978 tests passing.

Tech Decisions

•Realign with upstream instead of keeping the fork: game-specific differences live in wrapper modules and the rest belongs to the package, so later updates are just a version bump.
•Two-stage migration: switch the internal modules to the package first, then align patterns and migrations with the latest version, so fewer kinds of change land at once.
•Automate upgrades with a matrix: each project's result goes into the Job Summary, so skipped and failed projects show up on one screen.
•isolatedModules for tests: per-file transforms cut compile time, and the decorator type gap it creates is filled by explicit column types.

Operating limits

•Under isolatedModules the test step does no type checking, so type errors are caught by a separate tsc --noEmit step in CI.
•Projects without the upgrade automation still needed manual rollouts; one of them was checked for backward compatibility by comparing 1,073 fields byte for byte, excluding comments.
•The migration regression was found on live after deploy; pre-release tests did not catch it.

Highlights

•Replaced 17 data modules in a game server that had drifted from upstream for 2+ years with the shared npm package: deleted 610 files and 47 local migrations, updated 310+ imports
•Data-module features can now come in with a package version bump, and the schema follows a package-managed migration CLI
•Parallelized upgrades across 7 projects (up to 12 branches) with workflow_dispatch and a matrix, reporting skip/upgrade/downgrade in the Job Summary: about 3 hours by hand → about 15 minutes
•Switched ts-jest to isolatedModules: per-file compile about 15s → 1–3s, average test job 15m47s → 6m06s (61%), whole CI about 16 → about 7 minutes

Lessons Learned

•A fork gets more expensive to realign the longer it lives; keep game-specific differences in wrappers and leave the rest to the shared package.
•Automate work that repeats once per project, and keep the results on one screen so gaps are visible.

In-house AI Coding Agent Platform — Device Policy Guard and Usage Collection Server

2026.08 — Present · Platform Part lead · design, operations, and merge approval; built with AI coding agents · Team

Detail ↗

87 macOS/Windows devices (92 people on the roster) · one shared collection host · ClickHouse and SQLite

Owned design, operations, and merge approval for the company-wide device policy guard and usage collection server; AI coding agents handled implementation. The guard's policy path switched usage reporting on for 92 people, and all 87 reporting devices applied it. After two collection-server outages (2 days 18 hours, then 17 hours 40 minutes), memory limits were reordered as app < container < host and parsing moved to streaming, bringing the largest file's parse peak to 67% of a 640MiB limit.

GoNestJSTypeScriptClickHouseDocker ComposeAWS EC2OpenTelemetryGrafana

Performance Metrics	Before	After
Devices applying the reporting switch	-	All 87 (92 people assigned · reporting online devices)
Collection server remediation	-	Budget and limits explicit (Applied after a 2-day 18-hour 31-minute outage; a reparse loop remains the likely trigger)
Largest-file parse peak	OOM kill at 770,900kB	427.8MiB (67% of the 640MiB limit · streaming parse)
Alert notification changes	-	Mentions on 5 alerts (Added after 47 firings with no human messages over 3 days; shared routing unchanged)

Problem Solving

As AI service use grew, requests that would publish or share company material outside had to be blocked and recorded at the moment they ran, on both macOS and Windows and across a mix of subscription plans.

A resident daemon blocks policy-breaking publish and share requests, notifies the user, and records each attempt. A guard server registers devices through company-account OAuth (PKCE), collects heartbeats and block events, and sends only where a rule lives and its version; the daemon applies only signature-checked rules and keeps blocking when registration or reporting fails. The same policy path turned the usage-reporting switch on globally after two canary devices.

→ All 92 people on the active roster got the reporting switch, and all 87 reporting online devices applied it. macOS and Windows protection reports were confirmed; install, operations, and recovery guides were handed over, and the company-wide rollout belongs to another team.

The collection server, newly moved from an office NAS to AWS, was completely unresponsive for 2 days 18 hours 31 minutes and collected 0 records. The instance was running, with no error logs and no OOM kill.

The last sar samples before the gap showed memory pressure (memory use 62.90%→81.66%, page scan 153.77→332.58/s); disk, hypervisor contention, and CPU credits (exhausted 38 hours in, a result) were ruled out with evidence. ClickHouse claimed 90% of RAM by default on a host shared with other containers and without swap, so it could thrash instead of OOM-killing. The budget is now explicit, and the two large containers got mem_limit.

→ That closed one path, a default budget taking over a shared host; analysis of the next outage with the same signature points to a reparse loop as this one's likely trigger too. Upsizing waited: before observing the fix, it would only have paid to cover a wrong setting.

Once memory limits were in place, the collection app hit 19 OOM restarts under a 512m limit; after the limit went up to 768m, the host stopped responding for 17 hours 40 minutes.

The first fix for the restarts, an explicit Node heap cap, was dropped before it shipped: measurement showed Node 22 already sized its heap at about 51% of the cgroup limit. The 241MiB baseline the limit was based on had also been measured at idle. The restarts and the 17h 40m stall shared one cause, an endless reparse where any file with a pending chunk was reread whole; at 512m the loop killed only the container and the host held. Once a 176MB file had 9 pending chunks, every cycle pulled 45 chunks, 254.3MB including that file, into a string array. The 512MB file cap never applied to a 176MB file, the 2,048MB worker heap exceeded host RAM, and the 768m limit, the only one that fired, pushed out page cache each time and thrashed the host; my own raise from 512m to 768m turned restarts into a host stall. A 64MB file cap, 256MB heap, and 640m container broke the loop; with chunk-by-chunk streaming in place, the file cap went back to 256MB.

→ Parsing the largest file now peaks at 427.8MiB, 67% of the 640MiB limit. The O(n²) pattern, where one pending chunk rereads the whole file, is split out as follow-up work.

Throughout the 17h 40m outage the alerts fired and reached their channel, but nobody saw them, and the same had happened in the outage of 2 days 18 hours before it.

Of 104 channel messages over 3 days, 98 came from the bot and 0 from people, and none of the 47 firings carried a mention, so no mobile push went out. Shared routing and contact points stayed untouched; mentions went into the summary text of the 5 team-managed alert rules.

→ Without changing shared settings, mentions were added to the summary text of the 5 alerts.

Tech Decisions

•Synced records as the source of truth: device usage records backfill after an outage, and reparsing recomputes metrics.
•Order the limits: app heap cap < container limit < host RAM; otherwise the app's cap never acts, or a tripped limit stalls the host.
•Local blocking does not depend on the server: failed registration or heartbeats never stop it, and the daemon checks rule signatures.
•Separate who can read what: the IT role that follows up on installs cannot see block events, which narrows who can read those records.

Operating limits

•87 counts online devices applying the reporting switch, not people, and does not mean all 92 people on the roster are collecting normally.
•The company-wide staged rollout belongs to another team, and some checks remain, such as visual QA of screens and hands-on testing under other Windows accounts.
•The O(n²) reparse pattern, where one pending chunk rereads the whole file, remains a separate follow-up.

Highlights

•Usage-reporting switch sent through the guard policy path after two canaries: 92 people assigned, all 87 reporting devices apply it
•Guard server sends only rule location and version; daemon applies only signed rules, keeps blocking if registration or reporting fails
•2 days 18 hours unresponsive: thrashing with no OOM kill. ClickHouse's default budget (90% of RAM) on a shared host was found and the budget and container limits set explicitly; analysis of the next outage points to a reparse loop as the likely trigger
•17h 40m stall: the file cap missed a 176MB file, the 2,048MB worker heap exceeded the host, and the 768m limit thrashed the host each time it tripped. Lowered to 64MB, 256MB, and 640m, plus streaming parsing: largest-file peak 427.8MiB (67% of the limit)

Lessons Learned

•Judge a limit by what it allows and by what the host loses when it trips; my own raise turned container restarts into a host stall.
•Defaults carry the assumptions of the machine they were set for; when a server moves, recompute its limits.

5 Additional Public Projects

A Public Launcher for AI CLI Isolation and Durable FIFO Delivery

Personal · 2026.04 — Present

Built project-scoped AI CLI isolation, then replaced a long-held shared delivery lock with durable submission and a destination-ordered FIFO queue. Integrated OMP through an upstream extension, configuration overlay, and thin adapter rather than a local fork, with verified runtime, print, and signal boundaries.

Zsh, Bash, Python, Homebrew, GitHub Actions, MCP, Runtime Isolation, Fail-closed Validation

GitHub ↗

Daesin Logistics Dispatch Operations Ledger

Personal · 2026.01 — Present

Separated collection into a singleton and deployed the dispatch ledger’s web and collector on ARM64. Recovered the previous collector after a failed first attempt, then verified backup integrity and real changed-row API parity on redeployment. Published the owner-recorded single-deployment observation separately from ten synthetic scenarios.

TypeScript, Express, Next.js, React, Prisma, SQLite, Cheerio, TSyringe, TanStack Query, Docker, Traefik, Tailwind CSS, Vitest

Raw synthetic receipt · two invocations · not production performance ↗

Concert Reservation Service

Personal · 2024.10 — 2024.11

Concert reservation project that verifies same-seat booking and wallet payment with optimistic and Redisson locks, then reduces cache-stampede p95 from 5.74s to 676ms through cache warming and TTL changes in the public load-test report

Java, Spring Boot, JPA, Redis, MySQL, Kafka, Docker, K6

GitHub ↗

Work Schedule Planner — Scheduling That Surfaces Its Constraints

Personal · 2026.07 — Present

A local-first work planner that combines recognized load, actual work, leave, and daily/weekly caps into one deterministic allocation contract, exposing infeasible targets as explicit gaps instead of forced over-allocation. CI runs 62 pytest and 9 Playwright end-to-end tests. The public implementation is a synthetic preview and does not modify an official attendance system.

Python, React, TypeScript, FastAPI, Playwright, Property Testing, GitHub Actions

GitHub ↗

Family Budget — A Shared Household Budget for Daily Entry and Month-End Close

Personal · 2026.07 — Present

A household budget connecting daily entries, joint and individual ownership, purpose accounts, and recurring plans to month-end review. The public demo exercises existing-record linking, realization, reopen, and month-end warnings with synthetic browser-local data.

TypeScript, Next.js, React, Vitest, Vercel

Manifest ↗