Concert Reservation Service
2024-10-05 — 2024-11-30
Concert reservation project that verifies same-seat booking and wallet payment with optimistic and Redisson locks, then reduces cache-stampede p95 from 5.74s to 676ms through cache warming and TTL changes in the public load-test report
System Architecture
Problem Solving
Duplicate-reservation risk when 100 threads target the same seat
Exercised the conflict path with ConcertSeat @Version optimistic locking and a 100-thread Spring integration test
Assertions confirm one reservation row and the reserved state after the test
Balance integrity corruption under concurrent point charging and payments
Serialized per-user charge and payment paths with a Redisson distributed lock (userWalletLock:{userId})
A 100-thread charge test compares the final balance with the expected sum, while the payment test checks a single payment state
Observed a cache stampede and p95 5.74s latency during the reservation load test
Applied cache warming three minutes before reservations and changed TTL from two to five minutes
Public report records reservation p95 5.74s → 676ms, within its configured 1s SLO
Payment-event persistence, publication, and failed-event retry states needed separate boundaries
Kafka Transactional Outbox stores a PENDING outbox_event within the payment transaction, then a scheduler publishes it asynchronously
Separated PENDING outbox creation from publication-failure state and retry paths in code and tests
Project Description
Public-repository 100-thread integration tests verify that same-seat requests leave one reservation row and that per-user Redisson locking produces the expected charge and payment state. The payment transaction creates a PENDING outbox row, while a scheduler handles Kafka publication and failed-event retries. The k6 reservation scenario configures 3,000 requests/s with a 2,400 max-VU ceiling, so it is not presented as achieved throughput or concurrent users. Only the public report's observed 0.64% reservation error rate, p95 5.74s, and post-cache-warming/TTL p95 676ms are reported as outcomes.
Highlights
- One reservation row after a 100-thread same-seat integration test
- Per-user Redisson serialization for wallet and payment paths
- Public-report p95 5.74s → 676ms after cache warming and TTL changes
- PENDING outbox in the payment transaction plus scheduler publication and retry paths
- Configured 3,000 requests/s and 2,400 max VUs kept separate from observed outcomes
Performance Metrics
| Performance Metrics | Before | After |
|---|---|---|
| Reservation p95 | 5.74s | 676ms (cache warming + TTL 2m→5m) |
| Reservation error rate | - | 0.64% (394 failures in the reservation scenario) |
| Queue p95 | - | 62.5ms (0% error rate in the queue scenario) |
Tech Decisions
- ▶@Version optimistic locking for seats: a 100-thread same-seat integration test asserts one reservation row
- ▶Per-user Redisson locking for payment and charging: 100-thread tests compare payment state and final balance with expected values
- ▶Redis Sorted Set queue: scores track entry order and expiry state
- ▶Create a PENDING outbox record in the payment transaction, then let the scheduler manage publication-failure and retry paths
Lessons Learned
- •Separated optimistic locking for same-seat contention from per-user distributed locking for wallet contention
- •Kept configured k6 rate and max VUs distinct from observations reported by the load test
- •Scoped cache warming and TTL changes to the reservation p95 improvement recorded in the public report
- •Made the asynchronous boundary explicit through separate PENDING, publication-failure, and retry states in the payment outbox