Wróć do bloga

Burngate — Surviving a Spam Flood and What We Learned

Four days after launching tempy.email, we experienced our first major incident. At 2:47 AM UTC on February 17th, our monitoring alerted us to abnormal traffic patterns.

TL;DR: A spam campaign targeted us with thousands of emails. We implemented rate limiting in under 2 hours and hardened our infrastructure. No user data was compromised (we don't store any anyway).

The Incident Timeline

02:47 UTC — First Signs

Redis memory usage spiked from 180 MB to 2.1 GB in 8 minutes. Email throughput jumped from ~12/min to 847/min.

02:51 UTC — Pattern Recognition

Logs revealed a single IP sending emails to hundreds of randomly-generated tempy.email addresses. Classic spam flood.

02:53 UTC — Immediate Response

Temporarily blocked the offending IP at the nginx level. Memory stabilized but continued monitoring.

03:12 UTC — Root Cause Analysis

The attacker was exploiting our open email acceptance — we accepted mail to any @tempy.email address, even ones that didn't exist yet.

The flaw: Our SMTP server checked Redis for active mailboxes, but our SES Lambda accepted everything and created mailboxes on-demand.

03:45 UTC — Permanent Fix Deployed

Implemented rate limiting using AspNetCoreRateLimit:

// Incoming email endpoint rate limit
new RateLimitRule {
    Endpoint = "POST:/api/emailwebhook/incoming",
    Period = "1m",
    Limit = 60
},

// Address check endpoint (used by SES Lambda)
new RateLimitRule {
    Endpoint = "GET:/api/emailwebhook/check",
    Period = "1m",
    Limit = 120
}

Added IP-based throttling: 200 requests/minute per IP for general endpoints.

04:12 UTC — SES Lambda Hardening

Updated the Lambda to verify mailbox existence before accepting email:

# Before: Accept all, create mailbox
# After: Check existence, reject if not found
response = requests.get(f"{API_URL}/api/emailwebhook/check?address={recipient}")
if response.status_code != 200:
    return reject_email("Mailbox not found")

04:23 UTC — All Clear

Traffic normalized. Memory usage back to baseline. Rate limiting blocked 3 subsequent flood attempts automatically.

What We Learned

1. Rate Limiting is Table Stakes

We should have launched with rate limits on day one. Our "move fast" approach created an obvious vulnerability.

2. Defense in Depth

Having dual email paths (SMTP + SES) helped — we could patch one while the other handled legitimate traffic.

3. Memory-Only Storage is Risky

Redis memory floods are real. We implemented:

  • Aggressive TTLs (default 10 min)
  • LRU eviction policies
  • Memory usage alerts at 1.5 GB

4. Logs Are Gold

Structured logging with Serilog saved us. grep-friendly log tags like [MAIL-RECEIVED] made pattern recognition trivial.

5. Cloud Costs Can Spike

During the 90-minute incident, AWS SES charges hit $4.37 (vs. typical $0.12/day). Not catastrophic, but a reminder to set billing alerts.

The Fix in Detail

Rate Limiting Configuration

We added granular limits for every public endpoint:

options.GeneralRules = new[]
{
    // Email ingestion (highest risk)
    new RateLimitRule { Endpoint = "POST:/api/emailwebhook/incoming", Period = "1m", Limit = 60 },

    // Mailbox checks (SES Lambda)
    new RateLimitRule { Endpoint = "GET:/api/emailwebhook/check", Period = "1m", Limit = 120 },

    // Public API (developer usage)
    new RateLimitRule { Endpoint = "POST:/api/v1/mailbox", Period = "1m", Limit = 30 },
    new RateLimitRule { Endpoint = "GET:/api/v1/mailbox/*", Period = "1m", Limit = 100 },

    // Global fallback
    new RateLimitRule { Endpoint = "*", Period = "1m", Limit = 200 }
};

Whitelisted WebSocket and Blazor endpoints to avoid breaking real-time functionality.

Lambda Hardening

The SES Lambda now performs 3 checks before accepting email:

  1. Mailbox exists — GET /api/emailwebhook/check?address=
  2. Not expired — Check TTL via /api/emailwebhook/remaining?address=
  3. Rate limit compliance — Honors HTTP 429 from API

If any check fails, the email is rejected at the SES level (before we even download it from S3).

Performance Impact

Before rate limiting:

  • Legitimate traffic: ~12-15 emails/min
  • Spam incident: 847 emails/min (98.3% spam)

After rate limiting:

  • Legitimate traffic: Unchanged (12-15 emails/min)
  • Spam attempts: 0 successful, 847+ blocked/min during tests

Latency impact: None. AspNetCoreRateLimit adds ~2ms overhead.

What's Still Vulnerable

We're not claiming invincibility. Remaining attack vectors:

  1. Distributed spam — If an attacker uses 1000 IPs, our per-IP rate limits are less effective. We'd need to add per-mailbox limits.
  2. Slowloris attacks — Long-lived connections could exhaust resources. Nginx timeout configs help but aren't foolproof.
  3. DDoS — We're behind Cloudflare but haven't tested their limits.

Recommendations for Builders

If you're building a public email service (or any open API):

Day 1 Essentials

  • Rate limiting on all public endpoints
  • IP-based throttling (even if generous)
  • Memory limits and eviction policies
  • Billing alerts (set at 150% of normal)

Week 1 Additions

  • Structured logging with searchable tags
  • Metrics dashboards (we use Grafana + Redis metrics)
  • Automated alerts (Serilog → Slack)

Month 1 Hardening

  • Load testing with realistic attack scenarios
  • DDoS simulation (Cloudflare has tools for this)
  • Chaos engineering (kill processes, saturate memory)

The Name

Why "Burngate"? At 3 AM, delirious from debugging, we joked that our Redis was "burning" through memory. The name stuck.

Definitely not as glamorous as Facebook's "The Social Network" origin story, but at least we didn't get sued.

Conclusion

Every service gets attacked eventually. What matters is:

  1. Detecting it quickly
  2. Responding decisively
  3. Learning from it
  4. Hardening for next time

Four days post-launch, we got our first real battle scar. tempy.email is better for it.


Incident Summary

  • Duration: 90 minutes
  • Emails blocked: 847+ spam attempts
  • User impact: None (no data to lose)
  • Cost: $4.37 in AWS charges
  • Lessons: Priceless

Published February 17, 2026

Opublikowano lutego 17, 2026