# SSE Long-Lived Connection — Timeout & Reconnection Analysis

**Date:** 2026-07-10
**Status:** Analysis complete — recommendations pending implementation
**Scope:** Bugger (macOS SSE client) → Ingress → bugger-feishu (k8s pod, SSE server)

---

## TL;DR

Bugger subscribes to bugger-feishu via a long-lived SSE connection that traverses a
Kubernetes ingress. The connection's survival depends on **server-side heartbeats** and
the **ingress idle-timeout configuration** — neither of which Bugger can control from the
client side.

| Concern | Finding | Risk |
|---|---|---|
| Heartbeat | Server emits `: heartbeat\n\n` every ~30s; client relies on it | ✅ if ingress idle-timeout ≥ 60s; ❌ if tighter |
| Reconnection | Present, fixed 30s delay, well-guarded | Works, but no backoff/jitter → retry storms |
| Exponential backoff | **Not implemented** | Retry storms under sustained outage |

**Top recommendations:**
1. Set ingress `proxy_read_timeout` / LB idle-timeout to **≥ 60s** with `proxy_buffering off`.
2. Verify the 30s heartbeat is actually deployed in bugger-feishu (it is only *specified* in
   this repo's docs, not in shipped code).
3. Add exponential backoff with jitter and a cap to `BitableEventService` reconnect logic.

---

## 1. Architecture Under Test

```
┌──────────────┐    SSE (outbound, long-lived)    ┌─────────────────────────────┐
│   Bugger      │ ──────────────────────────────▶ │  Kubernetes Ingress          │
│  (macOS)      │   GET /api/v1/bitable/events    │  (idle-timeout = T_ingress)  │
│  SSE client  │   Accept: text/event-stream     │      │                       │
└──────────────┘                                 │      ▼                       │
                                                 │  ┌────────────────────────┐ │
                                                 │  │  bugger-feishu pod      │ │
                                                 │  │  (FastAPI SSE server)   │ │
                                                 │  │  heartbeat every 30s    │ │
                                                 │  └────────────────────────┘ │
                                                 └─────────────────────────────┘
```

- **Bugger** is the SSE *client* (`Sources/Services/BitableEventService.swift`, 227 LOC).
- **bugger-feishu** is the SSE *server* — a separate FastAPI service deployed as a k8s pod.
- The ingress sits between them and **will close any connection that appears idle** for
  longer than its configured idle-timeout (`proxy_read_timeout` for nginx ingress, default
  60s; cloud LBs vary).

---

## 2. Heartbeat Analysis

### 2.1 Client side — relies on server, sends nothing

SSE is a one-way protocol (server → client). Bugger's client **cannot keep the connection
alive from its end** — it can only receive. It depends entirely on the server periodically
sending bytes to reset the ingress's idle timer.

Client-side idle-timeout configuration (`BitableEventService.swift:61-66`):
```swift
let config = URLSessionConfiguration.ephemeral
// The server sends a heartbeat every ~30s, so 5 min of silence is a
// safe "something went wrong" threshold.
config.timeoutIntervalForRequest = 300
config.timeoutIntervalForResource = .infinity
config.waitsForConnectivity = true
```
- `timeoutIntervalForRequest = 300` (5 min): the client's own dead-connection detector.
  Set generously above the 30s server heartbeat so it never fires under normal operation.
- `timeoutIntervalForResource = .infinity`: unbounded resource lifetime (correct for a stream).
- `waitsForConnectivity = true`: OS waits through transient network gaps rather than failing.

The client's 300s timeout is **never the bottleneck** — it is the ingress that will drop
the connection first.

### 2.2 Server side — specified, must be verified deployed

The heartbeat is specified in `docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:686-693`:
```python
# Wait for events with heartbeat timeout
event = await asyncio.wait_for(conn.queue.get(), timeout=30.0)
yield f"event: {event['event']}\ndata: {event['data']}\n\n"
except asyncio.TimeoutError:
    # Heartbeat
    yield ": heartbeat\n\n"
```
- **Interval: 30 seconds.** Emits an SSE comment frame (`: heartbeat\n\n`) when no real
  event is queued within 30s.
- Response headers include `Connection: keep-alive` and `X-Accel-Buffering: no`
  (lines 707-711) to prevent buffering proxies from holding the heartbeat.

⚠️ **This code lives in the separate bugger-feishu repo and is only *specified* in this
repo's docs.** Whether it is actually deployed must be verified in the bugger-feishu repo
before relying on it.

### 2.3 Will the heartbeat prevent ingress disconnect?

The ingress closes an upstream connection when **no bytes flow** for its idle-timeout
period. The heartbeat resets that timer each time it is sent.

| Ingress idle-timeout | Heartbeat (30s) prevents disconnect? |
|---|---|
| ≥ 60s (nginx ingress default) | ✅ Yes — 30s < 60s, timer resets each cycle |
| 30s | ⚠️ Marginal — race conditions; not safe |
| < 30s (some cloud LBs) | ❌ No — connection dropped every cycle |

**Requirements for the heartbeat to work:**
1. Ingress `proxy_read_timeout` / LB idle-timeout **≥ 60s** (30s heartbeat + safety margin).
2. `proxy_buffering off` on the ingress location, **or** the server's `X-Accel-Buffering: no`
   header must be honored — otherwise heartbeats are buffered and never reach the client
   until the response completes (which never happens for a stream).
3. The 30s heartbeat must actually be running in the deployed bugger-feishu pod.

### 2.4 What happens on a too-tight ingress timeout?

If the ingress idle-timeout is, say, 15s:
1. Bugger connects.
2. After 15s of silence (before the 30s heartbeat fires), the ingress closes the upstream.
3. The client sees the stream end → schedules a reconnect in 30s.
4. Repeat every ~45s indefinitely — a steady-state reconnect loop.

This wastes bandwidth, defeats the push model, and — with many Bugger clients — creates a
synchronized reconnect storm (no jitter, see §3.3).

---

## 3. Reconnection Analysis

### 3.1 Mechanism — present and well-guarded

Reconnection is implemented in `BitableEventService.swift`. On any stream end (error or
clean), if the user still intends to be connected, a reconnect is scheduled.

State guard (`BitableEventService.swift:21-23`):
```swift
/// The user's intent to be connected. Distinguishes an unexpected stream
/// drop (should auto-reconnect) from an explicit `disconnect()` (should not).
private var isEnabled = false
```

Reconnect scheduling (`BitableEventService.swift:102-124`):
```swift
private func handleStreamEnd(error: Error?) {
    task = nil
    session = nil
    delegate = nil

    if let error {
        BuggerLog.error("BitableEventService: stream ended (\(error.localizedDescription))")
    } else {
        BuggerLog.info("BitableEventService: stream ended")
    }

    // Only auto-reconnect if the user still wants to be connected.
    guard isEnabled else { return }

    let work = DispatchWorkItem { [weak self] in
        guard let self, self.isEnabled else { return }
        guard let url = self.buildURL() else { return }
        BuggerLog.info("BitableEventService: reconnecting...")
        self.openStream(at: url)
    }
    reconnectWorkItem = work
    DispatchQueue.main.asyncAfter(deadline: .now() + reconnectDelay, execute: work)
}
```

Fixed delay constant (`BitableEventService.swift:25`):
```swift
private let reconnectDelay: TimeInterval = 30
```

**Guard quality — good:**
- Double-guarded against intentional disconnect: `isEnabled` check at line 114, plus
  re-checked inside the `DispatchWorkItem` at line 117.
- Pending reconnect is cancellable: `closeStream()` calls `reconnectWorkItem?.cancel()`
  (line 93), so `disconnect()` cannot race with a scheduled reconnect.
- All state mutation happens on the main queue (callbacks routed via `DispatchQueue.main.async`
  in `SSESessionDelegate` at lines 202, 223).

**Gap — no catch-up fetch:** reconnection does **not** trigger an immediate `fetchNow()`
to recover events missed during the downtime window. It relies on the next server push or
the periodic `PollerService` to eventually resync. This is acceptable given polling is the
fallback, but means event freshness is bounded by the poll interval after any reconnect.

### 3.2 Exponential backoff — NOT implemented

The reconnect delay is a **constant 30s, every time, forever**, regardless of how many
consecutive failures occur.

Searching the repo, "exponential backoff" appears only in design docs for the *rejected*
direct-WebSocket design — never in shipped SSE code:
- `docs/TECH_INVESTIGATION.md:56-58` — Feishu raw WebSocket option (not chosen)
- `docs/TECH_INVESTIGATION.md:424` — planned task for the abandoned WSClient approach
- `docs/TECH_INVESTIGATION.md:544` — same

### 3.3 Risks of the fixed-delay strategy

| Scenario | Behavior | Problem |
|---|---|---|
| bugger-feishu pod down 1 hour | 120 reconnect attempts at fixed 30s | No de-escalation; wastes resources |
| bugger-feishu pod restarts | All clients reconnect simultaneously after 30s | **Thundering herd** — no jitter |
| Ingress drops every cycle (timeout < 30s) | Reconnect every ~45s per client | Steady-state storm across all clients |
| Many clients, transient outage | All retry on identical 30s cadence | Synchronized load spikes on ingress |

The lack of jitter is the most operationally significant gap: with N Bugger clients all
using the same fixed 30s delay, a pod restart causes all N to hit the ingress at the same
instant, every 30s, until one succeeds.

---

## 4. Findings Summary

### Q1 — Heartbeat & ingress timeout

- **Yes**, a heartbeat exists — but it is **server-side only** (bugger-feishu emits
  `: heartbeat\n\n` every ~30s). Bugger's client sends nothing.
- **Will it prevent ingress disconnect?** Only if the ingress idle-timeout
  (`proxy_read_timeout` / LB idle-timeout) is **≥ the heartbeat interval (30s)**, with
  margin — recommend **≥ 60s**. It also requires `proxy_buffering off` (or honored
  `X-Accel-Buffering: no`) so heartbeats are not buffered.
- **Caveat:** the heartbeat code lives in the separate bugger-feishu repo. It is only
  *specified* in this repo's docs — verify it is actually deployed.

### Q2 — Reconnection & exponential backoff

- **Yes**, reconnection exists. It is triggered on stream end when `isEnabled == true`,
  uses a fixed **30s delay**, and is properly guarded against intentional disconnects
  (double guard + cancellable work item).
- **No**, it does **not** use exponential backoff. The delay is constant 30s, indefinitely,
  with no cap, no jitter, and no escalation/de-escalation. This creates retry-storm and
  thundering-herd risks under sustained or synchronized outages.

---

## 5. Recommendations

### R1 — Ingress / LB configuration (ops, no code change)

Configure the ingress in front of bugger-feishu:
1. `proxy_read_timeout 60s;` (or higher) — must exceed the 30s heartbeat with margin.
2. `proxy_buffering off;` on the SSE location — or ensure `X-Accel-Buffering: no` is honored.
3. `proxy_send_timeout 60s;` — symmetric.
4. Verify the cloud LB idle-timeout (if any, in front of the ingress) is also ≥ 60s — some
   LBs default to 30s or less and override the ingress setting.

### R2 — Verify heartbeat is deployed (ops, no code change)

Confirm in the bugger-feishu repo that the 30s `: heartbeat\n\n` emission is actually
running in the deployed image, not just specified in this repo's docs. If absent, the
ingress will drop the connection on every idle cycle regardless of R1.

### R3 — Add exponential backoff with jitter and cap (code change)

Modify `BitableEventService.swift` to replace the fixed `reconnectDelay` with a backoff
strategy. Suggested parameters:

```
attempt 1:  delay = 1s  (±20% jitter)
attempt 2:  delay = 2s  (±20% jitter)
attempt 3:  delay = 4s  (±20% jitter)
attempt 4:  delay = 8s  (±20% jitter)
attempt 5:  delay = 16s (±20% jitter)
attempt 6+: delay = 30s (±20% jitter)  ← cap
```

- Base: `delay = min(cap, base * 2^(attempt-1))`
- Cap: 30s (preserves current steady-state behavior)
- Jitter: ±20% of computed delay (prevents thundering herd)
- Reset attempt counter on a successful `connected` event from the server.

### R4 — Catch-up fetch on reconnect (optional code change)

On a successful reconnect, trigger `PollerService.shared.fetchNow()` once to recover any
events missed during the downtime window, rather than waiting for the next poll cycle.
This bounds event freshness loss to one fetch rather than the full poll interval.

---

## 6. Key Code Locations

All paths in `/Users/tigeren/Dev/xorbitlab/bugger/`:

| Location | What |
|---|---|
| `Sources/Services/BitableEventService.swift:25` | `reconnectDelay = 30` (fixed, no backoff) |
| `Sources/Services/BitableEventService.swift:21-23` | `isEnabled` guard |
| `Sources/Services/BitableEventService.swift:61-66` | Client idle-timeout config (300s request, infinite resource) |
| `Sources/Services/BitableEventService.swift:83` | Request timeout (300s) |
| `Sources/Services/BitableEventService.swift:92-100` | `closeStream` — cancels pending reconnect |
| `Sources/Services/BitableEventService.swift:102-124` | `handleStreamEnd` — auto-reconnect scheduling |
| `Sources/Services/BitableEventService.swift:126-138` | `handleEvent` — `change`→`fetchNow`, heartbeat no-op |
| `Sources/Services/BitableEventService.swift:170-226` | `SSESessionDelegate` — frame parser, comment/heartbeat ignored |
| `Sources/AppDelegate.swift:33` | `connect()` on app launch |
| `Sources/Views/Settings/SettingsView.swift:293` | `reconnect()` on settings save |
| `Sources/Views/Settings/SettingsView.swift:181` | `disconnect()` on field clear |
| `docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:686-693` | Server heartbeat spec (30s, must verify deployed) |
| `docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:707-711` | Server SSE response headers (keep-alive, no buffering) |
| `docs/TECH_INVESTIGATION.md:56-58,424,544` | Backoff references (rejected WSClient design only) |