bugger/docs/SSE_TIMEOUT_ANALYSIS.md

SSE Long-Lived Connection — Timeout & Reconnection Analysis

Date: 2026-07-10 Status: Analysis complete — recommendations pending implementation Scope: Bugger (macOS SSE client) → Ingress → bugger-feishu (k8s pod, SSE server)

---

TL;DR

Bugger subscribes to bugger-feishu via a long-lived SSE connection that traverses a Kubernetes ingress. The connection's survival depends on server-side heartbeats and the ingress idle-timeout configuration — neither of which Bugger can control from the client side.

Concern	Finding	Risk
Heartbeat	Server emits : heartbeat\n\n every ~30s; client relies on it	✅ if ingress idle-timeout ≥ 60s; ❌ if tighter
Reconnection	Present, fixed 30s delay, well-guarded	Works, but no backoff/jitter → retry storms
Exponential backoff	Not implemented	Retry storms under sustained outage

Top recommendations:

1. Set ingress proxy_read_timeout / LB idle-timeout to ≥ 60s with proxy_buffering off.
2. Verify the 30s heartbeat is actually deployed in bugger-feishu (it is only specified in this repo's docs, not in shipped code).
3. Add exponential backoff with jitter and a cap to BitableEventService reconnect logic.

---

1. Architecture Under Test

┌──────────────┐    SSE (outbound, long-lived)    ┌─────────────────────────────┐
│   Bugger      │ ──────────────────────────────▶ │  Kubernetes Ingress          │
│  (macOS)      │   GET /api/v1/bitable/events    │  (idle-timeout = T_ingress)  │
│  SSE client  │   Accept: text/event-stream     │      │                       │
└──────────────┘                                 │      ▼                       │
                                                 │  ┌────────────────────────┐ │
                                                 │  │  bugger-feishu pod      │ │
                                                 │  │  (FastAPI SSE server)   │ │
                                                 │  │  heartbeat every 30s    │ │
                                                 │  └────────────────────────┘ │
                                                 └─────────────────────────────┘

• Bugger is the SSE client (Sources/Services/BitableEventService.swift, 227 LOC).
• bugger-feishu is the SSE server — a separate FastAPI service deployed as a k8s pod.
• The ingress sits between them and will close any connection that appears idle for longer than its configured idle-timeout (proxy_read_timeout for nginx ingress, default 60s; cloud LBs vary).

---

2. Heartbeat Analysis

2.1 Client side — relies on server, sends nothing

SSE is a one-way protocol (server → client). Bugger's client cannot keep the connection alive from its end — it can only receive. It depends entirely on the server periodically sending bytes to reset the ingress's idle timer.

Client-side idle-timeout configuration (BitableEventService.swift:61-66):

let config = URLSessionConfiguration.ephemeral
// The server sends a heartbeat every ~30s, so 5 min of silence is a
// safe "something went wrong" threshold.
config.timeoutIntervalForRequest = 300
config.timeoutIntervalForResource = .infinity
config.waitsForConnectivity = true

• timeoutIntervalForRequest = 300 (5 min): the client's own dead-connection detector. Set generously above the 30s server heartbeat so it never fires under normal operation.
• timeoutIntervalForResource = .infinity: unbounded resource lifetime (correct for a stream).
• waitsForConnectivity = true: OS waits through transient network gaps rather than failing.

The client's 300s timeout is never the bottleneck — it is the ingress that will drop the connection first.

2.2 Server side — specified, must be verified deployed

The heartbeat is specified in docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:686-693:

# Wait for events with heartbeat timeout
event = await asyncio.wait_for(conn.queue.get(), timeout=30.0)
yield f"event: {event['event']}\ndata: {event['data']}\n\n"
except asyncio.TimeoutError:
    # Heartbeat
    yield ": heartbeat\n\n"

• Interval: 30 seconds. Emits an SSE comment frame (: heartbeat\n\n) when no real event is queued within 30s.
• Response headers include Connection: keep-alive and X-Accel-Buffering: no (lines 707-711) to prevent buffering proxies from holding the heartbeat.

⚠️ This code lives in the separate bugger-feishu repo and is only specified in this repo's docs. Whether it is actually deployed must be verified in the bugger-feishu repo before relying on it.

2.3 Will the heartbeat prevent ingress disconnect?

The ingress closes an upstream connection when no bytes flow for its idle-timeout period. The heartbeat resets that timer each time it is sent.

Ingress idle-timeout	Heartbeat (30s) prevents disconnect?
≥ 60s (nginx ingress default)	✅ Yes — 30s < 60s, timer resets each cycle
30s	⚠️ Marginal — race conditions; not safe
< 30s (some cloud LBs)	❌ No — connection dropped every cycle

Requirements for the heartbeat to work:

1. Ingress proxy_read_timeout / LB idle-timeout ≥ 60s (30s heartbeat + safety margin).
2. proxy_buffering off on the ingress location, or the server's X-Accel-Buffering: no header must be honored — otherwise heartbeats are buffered and never reach the client until the response completes (which never happens for a stream).
3. The 30s heartbeat must actually be running in the deployed bugger-feishu pod.

2.4 What happens on a too-tight ingress timeout?

If the ingress idle-timeout is, say, 15s:

1. Bugger connects.
2. After 15s of silence (before the 30s heartbeat fires), the ingress closes the upstream.
3. The client sees the stream end → schedules a reconnect in 30s.
4. Repeat every ~45s indefinitely — a steady-state reconnect loop.

This wastes bandwidth, defeats the push model, and — with many Bugger clients — creates a synchronized reconnect storm (no jitter, see §3.3).

---

3. Reconnection Analysis

3.1 Mechanism — present and well-guarded

Reconnection is implemented in BitableEventService.swift. On any stream end (error or clean), if the user still intends to be connected, a reconnect is scheduled.

State guard (BitableEventService.swift:21-23):

/// The user's intent to be connected. Distinguishes an unexpected stream
/// drop (should auto-reconnect) from an explicit `disconnect()` (should not).
private var isEnabled = false

Reconnect scheduling (BitableEventService.swift:102-124):

private func handleStreamEnd(error: Error?) {
    task = nil
    session = nil
    delegate = nil

    if let error {
        BuggerLog.error("BitableEventService: stream ended (\(error.localizedDescription))")
    } else {
        BuggerLog.info("BitableEventService: stream ended")
    }

    // Only auto-reconnect if the user still wants to be connected.
    guard isEnabled else { return }

    let work = DispatchWorkItem { [weak self] in
        guard let self, self.isEnabled else { return }
        guard let url = self.buildURL() else { return }
        BuggerLog.info("BitableEventService: reconnecting...")
        self.openStream(at: url)
    }
    reconnectWorkItem = work
    DispatchQueue.main.asyncAfter(deadline: .now() + reconnectDelay, execute: work)
}

Fixed delay constant (BitableEventService.swift:25):

private let reconnectDelay: TimeInterval = 30

Guard quality — good:

• Double-guarded against intentional disconnect: isEnabled check at line 114, plus re-checked inside the DispatchWorkItem at line 117.
• Pending reconnect is cancellable: closeStream() calls reconnectWorkItem?.cancel() (line 93), so disconnect() cannot race with a scheduled reconnect.
• All state mutation happens on the main queue (callbacks routed via DispatchQueue.main.async in SSESessionDelegate at lines 202, 223).

Gap — no catch-up fetch: reconnection does not trigger an immediate fetchNow() to recover events missed during the downtime window. It relies on the next server push or the periodic PollerService to eventually resync. This is acceptable given polling is the fallback, but means event freshness is bounded by the poll interval after any reconnect.

3.2 Exponential backoff — NOT implemented

The reconnect delay is a constant 30s, every time, forever, regardless of how many consecutive failures occur.

Searching the repo, "exponential backoff" appears only in design docs for the rejected direct-WebSocket design — never in shipped SSE code:

• docs/TECH_INVESTIGATION.md:56-58 — Feishu raw WebSocket option (not chosen)
• docs/TECH_INVESTIGATION.md:424 — planned task for the abandoned WSClient approach
• docs/TECH_INVESTIGATION.md:544 — same

3.3 Risks of the fixed-delay strategy

Scenario	Behavior	Problem
bugger-feishu pod down 1 hour	120 reconnect attempts at fixed 30s	No de-escalation; wastes resources
bugger-feishu pod restarts	All clients reconnect simultaneously after 30s	Thundering herd — no jitter
Ingress drops every cycle (timeout < 30s)	Reconnect every ~45s per client	Steady-state storm across all clients
Many clients, transient outage	All retry on identical 30s cadence	Synchronized load spikes on ingress

The lack of jitter is the most operationally significant gap: with N Bugger clients all using the same fixed 30s delay, a pod restart causes all N to hit the ingress at the same instant, every 30s, until one succeeds.

---

4. Findings Summary

Q1 — Heartbeat & ingress timeout

• Yes, a heartbeat exists — but it is server-side only (bugger-feishu emits : heartbeat\n\n every ~30s). Bugger's client sends nothing.
• Will it prevent ingress disconnect? Only if the ingress idle-timeout (proxy_read_timeout / LB idle-timeout) is ≥ the heartbeat interval (30s), with margin — recommend ≥ 60s. It also requires proxy_buffering off (or honored X-Accel-Buffering: no) so heartbeats are not buffered.
• Caveat: the heartbeat code lives in the separate bugger-feishu repo. It is only specified in this repo's docs — verify it is actually deployed.

Q2 — Reconnection & exponential backoff

• Yes, reconnection exists. It is triggered on stream end when isEnabled == true, uses a fixed 30s delay, and is properly guarded against intentional disconnects (double guard + cancellable work item).
• No, it does not use exponential backoff. The delay is constant 30s, indefinitely, with no cap, no jitter, and no escalation/de-escalation. This creates retry-storm and thundering-herd risks under sustained or synchronized outages.

---

5. Recommendations

R1 — Ingress / LB configuration (ops, no code change)

Configure the ingress in front of bugger-feishu:

1. proxy_read_timeout 60s; (or higher) — must exceed the 30s heartbeat with margin.
2. proxy_buffering off; on the SSE location — or ensure X-Accel-Buffering: no is honored.
3. proxy_send_timeout 60s; — symmetric.
4. Verify the cloud LB idle-timeout (if any, in front of the ingress) is also ≥ 60s — some LBs default to 30s or less and override the ingress setting.

R2 — Verify heartbeat is deployed (ops, no code change)

Confirm in the bugger-feishu repo that the 30s : heartbeat\n\n emission is actually running in the deployed image, not just specified in this repo's docs. If absent, the ingress will drop the connection on every idle cycle regardless of R1.

R3 — Add exponential backoff with jitter and cap (code change)

Modify BitableEventService.swift to replace the fixed reconnectDelay with a backoff strategy. Suggested parameters:

attempt 1:  delay = 1s  (±20% jitter)
attempt 2:  delay = 2s  (±20% jitter)
attempt 3:  delay = 4s  (±20% jitter)
attempt 4:  delay = 8s  (±20% jitter)
attempt 5:  delay = 16s (±20% jitter)
attempt 6+: delay = 30s (±20% jitter)  ← cap

• Base: delay = min(cap, base * 2^(attempt-1))
• Cap: 30s (preserves current steady-state behavior)
• Jitter: ±20% of computed delay (prevents thundering herd)
• Reset attempt counter on a successful connected event from the server.

R4 — Catch-up fetch on reconnect (optional code change)

On a successful reconnect, trigger PollerService.shared.fetchNow() once to recover any events missed during the downtime window, rather than waiting for the next poll cycle. This bounds event freshness loss to one fetch rather than the full poll interval.

---

6. Key Code Locations

All paths in /Users/tigeren/Dev/xorbitlab/bugger/:

Location	What
Sources/Services/BitableEventService.swift:25	reconnectDelay = 30 (fixed, no backoff)
Sources/Services/BitableEventService.swift:21-23	isEnabled guard
Sources/Services/BitableEventService.swift:61-66	Client idle-timeout config (300s request, infinite resource)
Sources/Services/BitableEventService.swift:83	Request timeout (300s)
Sources/Services/BitableEventService.swift:92-100	closeStream — cancels pending reconnect
Sources/Services/BitableEventService.swift:102-124	handleStreamEnd — auto-reconnect scheduling
Sources/Services/BitableEventService.swift:126-138	handleEvent — change→fetchNow, heartbeat no-op
Sources/Services/BitableEventService.swift:170-226	SSESessionDelegate — frame parser, comment/heartbeat ignored
Sources/AppDelegate.swift:33	connect() on app launch
Sources/Views/Settings/SettingsView.swift:293	reconnect() on settings save
Sources/Views/Settings/SettingsView.swift:181	disconnect() on field clear
docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:686-693	Server heartbeat spec (30s, must verify deployed)
docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:707-711	Server SSE response headers (keep-alive, no buffering)
docs/TECH_INVESTIGATION.md:56-58,424,544	Backoff references (rejected WSClient design only)