303 lines
14 KiB
Markdown
303 lines
14 KiB
Markdown
# SSE Long-Lived Connection — Timeout & Reconnection Analysis
|
||
|
||
**Date:** 2026-07-10
|
||
**Status:** Analysis complete — recommendations pending implementation
|
||
**Scope:** Bugger (macOS SSE client) → Ingress → bugger-feishu (k8s pod, SSE server)
|
||
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
Bugger subscribes to bugger-feishu via a long-lived SSE connection that traverses a
|
||
Kubernetes ingress. The connection's survival depends on **server-side heartbeats** and
|
||
the **ingress idle-timeout configuration** — neither of which Bugger can control from the
|
||
client side.
|
||
|
||
| Concern | Finding | Risk |
|
||
|---|---|---|
|
||
| Heartbeat | Server emits `: heartbeat\n\n` every ~30s; client relies on it | ✅ if ingress idle-timeout ≥ 60s; ❌ if tighter |
|
||
| Reconnection | Present, fixed 30s delay, well-guarded | Works, but no backoff/jitter → retry storms |
|
||
| Exponential backoff | **Not implemented** | Retry storms under sustained outage |
|
||
|
||
**Top recommendations:**
|
||
1. Set ingress `proxy_read_timeout` / LB idle-timeout to **≥ 60s** with `proxy_buffering off`.
|
||
2. Verify the 30s heartbeat is actually deployed in bugger-feishu (it is only *specified* in
|
||
this repo's docs, not in shipped code).
|
||
3. Add exponential backoff with jitter and a cap to `BitableEventService` reconnect logic.
|
||
|
||
---
|
||
|
||
## 1. Architecture Under Test
|
||
|
||
```
|
||
┌──────────────┐ SSE (outbound, long-lived) ┌─────────────────────────────┐
|
||
│ Bugger │ ──────────────────────────────▶ │ Kubernetes Ingress │
|
||
│ (macOS) │ GET /api/v1/bitable/events │ (idle-timeout = T_ingress) │
|
||
│ SSE client │ Accept: text/event-stream │ │ │
|
||
└──────────────┘ │ ▼ │
|
||
│ ┌────────────────────────┐ │
|
||
│ │ bugger-feishu pod │ │
|
||
│ │ (FastAPI SSE server) │ │
|
||
│ │ heartbeat every 30s │ │
|
||
│ └────────────────────────┘ │
|
||
└─────────────────────────────┘
|
||
```
|
||
|
||
- **Bugger** is the SSE *client* (`Sources/Services/BitableEventService.swift`, 227 LOC).
|
||
- **bugger-feishu** is the SSE *server* — a separate FastAPI service deployed as a k8s pod.
|
||
- The ingress sits between them and **will close any connection that appears idle** for
|
||
longer than its configured idle-timeout (`proxy_read_timeout` for nginx ingress, default
|
||
60s; cloud LBs vary).
|
||
|
||
---
|
||
|
||
## 2. Heartbeat Analysis
|
||
|
||
### 2.1 Client side — relies on server, sends nothing
|
||
|
||
SSE is a one-way protocol (server → client). Bugger's client **cannot keep the connection
|
||
alive from its end** — it can only receive. It depends entirely on the server periodically
|
||
sending bytes to reset the ingress's idle timer.
|
||
|
||
Client-side idle-timeout configuration (`BitableEventService.swift:61-66`):
|
||
```swift
|
||
let config = URLSessionConfiguration.ephemeral
|
||
// The server sends a heartbeat every ~30s, so 5 min of silence is a
|
||
// safe "something went wrong" threshold.
|
||
config.timeoutIntervalForRequest = 300
|
||
config.timeoutIntervalForResource = .infinity
|
||
config.waitsForConnectivity = true
|
||
```
|
||
- `timeoutIntervalForRequest = 300` (5 min): the client's own dead-connection detector.
|
||
Set generously above the 30s server heartbeat so it never fires under normal operation.
|
||
- `timeoutIntervalForResource = .infinity`: unbounded resource lifetime (correct for a stream).
|
||
- `waitsForConnectivity = true`: OS waits through transient network gaps rather than failing.
|
||
|
||
The client's 300s timeout is **never the bottleneck** — it is the ingress that will drop
|
||
the connection first.
|
||
|
||
### 2.2 Server side — specified, must be verified deployed
|
||
|
||
The heartbeat is specified in `docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:686-693`:
|
||
```python
|
||
# Wait for events with heartbeat timeout
|
||
event = await asyncio.wait_for(conn.queue.get(), timeout=30.0)
|
||
yield f"event: {event['event']}\ndata: {event['data']}\n\n"
|
||
except asyncio.TimeoutError:
|
||
# Heartbeat
|
||
yield ": heartbeat\n\n"
|
||
```
|
||
- **Interval: 30 seconds.** Emits an SSE comment frame (`: heartbeat\n\n`) when no real
|
||
event is queued within 30s.
|
||
- Response headers include `Connection: keep-alive` and `X-Accel-Buffering: no`
|
||
(lines 707-711) to prevent buffering proxies from holding the heartbeat.
|
||
|
||
⚠️ **This code lives in the separate bugger-feishu repo and is only *specified* in this
|
||
repo's docs.** Whether it is actually deployed must be verified in the bugger-feishu repo
|
||
before relying on it.
|
||
|
||
### 2.3 Will the heartbeat prevent ingress disconnect?
|
||
|
||
The ingress closes an upstream connection when **no bytes flow** for its idle-timeout
|
||
period. The heartbeat resets that timer each time it is sent.
|
||
|
||
| Ingress idle-timeout | Heartbeat (30s) prevents disconnect? |
|
||
|---|---|
|
||
| ≥ 60s (nginx ingress default) | ✅ Yes — 30s < 60s, timer resets each cycle |
|
||
| 30s | ⚠️ Marginal — race conditions; not safe |
|
||
| < 30s (some cloud LBs) | ❌ No — connection dropped every cycle |
|
||
|
||
**Requirements for the heartbeat to work:**
|
||
1. Ingress `proxy_read_timeout` / LB idle-timeout **≥ 60s** (30s heartbeat + safety margin).
|
||
2. `proxy_buffering off` on the ingress location, **or** the server's `X-Accel-Buffering: no`
|
||
header must be honored — otherwise heartbeats are buffered and never reach the client
|
||
until the response completes (which never happens for a stream).
|
||
3. The 30s heartbeat must actually be running in the deployed bugger-feishu pod.
|
||
|
||
### 2.4 What happens on a too-tight ingress timeout?
|
||
|
||
If the ingress idle-timeout is, say, 15s:
|
||
1. Bugger connects.
|
||
2. After 15s of silence (before the 30s heartbeat fires), the ingress closes the upstream.
|
||
3. The client sees the stream end → schedules a reconnect in 30s.
|
||
4. Repeat every ~45s indefinitely — a steady-state reconnect loop.
|
||
|
||
This wastes bandwidth, defeats the push model, and — with many Bugger clients — creates a
|
||
synchronized reconnect storm (no jitter, see §3.3).
|
||
|
||
---
|
||
|
||
## 3. Reconnection Analysis
|
||
|
||
### 3.1 Mechanism — present and well-guarded
|
||
|
||
Reconnection is implemented in `BitableEventService.swift`. On any stream end (error or
|
||
clean), if the user still intends to be connected, a reconnect is scheduled.
|
||
|
||
State guard (`BitableEventService.swift:21-23`):
|
||
```swift
|
||
/// The user's intent to be connected. Distinguishes an unexpected stream
|
||
/// drop (should auto-reconnect) from an explicit `disconnect()` (should not).
|
||
private var isEnabled = false
|
||
```
|
||
|
||
Reconnect scheduling (`BitableEventService.swift:102-124`):
|
||
```swift
|
||
private func handleStreamEnd(error: Error?) {
|
||
task = nil
|
||
session = nil
|
||
delegate = nil
|
||
|
||
if let error {
|
||
BuggerLog.error("BitableEventService: stream ended (\(error.localizedDescription))")
|
||
} else {
|
||
BuggerLog.info("BitableEventService: stream ended")
|
||
}
|
||
|
||
// Only auto-reconnect if the user still wants to be connected.
|
||
guard isEnabled else { return }
|
||
|
||
let work = DispatchWorkItem { [weak self] in
|
||
guard let self, self.isEnabled else { return }
|
||
guard let url = self.buildURL() else { return }
|
||
BuggerLog.info("BitableEventService: reconnecting...")
|
||
self.openStream(at: url)
|
||
}
|
||
reconnectWorkItem = work
|
||
DispatchQueue.main.asyncAfter(deadline: .now() + reconnectDelay, execute: work)
|
||
}
|
||
```
|
||
|
||
Fixed delay constant (`BitableEventService.swift:25`):
|
||
```swift
|
||
private let reconnectDelay: TimeInterval = 30
|
||
```
|
||
|
||
**Guard quality — good:**
|
||
- Double-guarded against intentional disconnect: `isEnabled` check at line 114, plus
|
||
re-checked inside the `DispatchWorkItem` at line 117.
|
||
- Pending reconnect is cancellable: `closeStream()` calls `reconnectWorkItem?.cancel()`
|
||
(line 93), so `disconnect()` cannot race with a scheduled reconnect.
|
||
- All state mutation happens on the main queue (callbacks routed via `DispatchQueue.main.async`
|
||
in `SSESessionDelegate` at lines 202, 223).
|
||
|
||
**Gap — no catch-up fetch:** reconnection does **not** trigger an immediate `fetchNow()`
|
||
to recover events missed during the downtime window. It relies on the next server push or
|
||
the periodic `PollerService` to eventually resync. This is acceptable given polling is the
|
||
fallback, but means event freshness is bounded by the poll interval after any reconnect.
|
||
|
||
### 3.2 Exponential backoff — NOT implemented
|
||
|
||
The reconnect delay is a **constant 30s, every time, forever**, regardless of how many
|
||
consecutive failures occur.
|
||
|
||
Searching the repo, "exponential backoff" appears only in design docs for the *rejected*
|
||
direct-WebSocket design — never in shipped SSE code:
|
||
- `docs/TECH_INVESTIGATION.md:56-58` — Feishu raw WebSocket option (not chosen)
|
||
- `docs/TECH_INVESTIGATION.md:424` — planned task for the abandoned WSClient approach
|
||
- `docs/TECH_INVESTIGATION.md:544` — same
|
||
|
||
### 3.3 Risks of the fixed-delay strategy
|
||
|
||
| Scenario | Behavior | Problem |
|
||
|---|---|---|
|
||
| bugger-feishu pod down 1 hour | 120 reconnect attempts at fixed 30s | No de-escalation; wastes resources |
|
||
| bugger-feishu pod restarts | All clients reconnect simultaneously after 30s | **Thundering herd** — no jitter |
|
||
| Ingress drops every cycle (timeout < 30s) | Reconnect every ~45s per client | Steady-state storm across all clients |
|
||
| Many clients, transient outage | All retry on identical 30s cadence | Synchronized load spikes on ingress |
|
||
|
||
The lack of jitter is the most operationally significant gap: with N Bugger clients all
|
||
using the same fixed 30s delay, a pod restart causes all N to hit the ingress at the same
|
||
instant, every 30s, until one succeeds.
|
||
|
||
---
|
||
|
||
## 4. Findings Summary
|
||
|
||
### Q1 — Heartbeat & ingress timeout
|
||
|
||
- **Yes**, a heartbeat exists — but it is **server-side only** (bugger-feishu emits
|
||
`: heartbeat\n\n` every ~30s). Bugger's client sends nothing.
|
||
- **Will it prevent ingress disconnect?** Only if the ingress idle-timeout
|
||
(`proxy_read_timeout` / LB idle-timeout) is **≥ the heartbeat interval (30s)**, with
|
||
margin — recommend **≥ 60s**. It also requires `proxy_buffering off` (or honored
|
||
`X-Accel-Buffering: no`) so heartbeats are not buffered.
|
||
- **Caveat:** the heartbeat code lives in the separate bugger-feishu repo. It is only
|
||
*specified* in this repo's docs — verify it is actually deployed.
|
||
|
||
### Q2 — Reconnection & exponential backoff
|
||
|
||
- **Yes**, reconnection exists. It is triggered on stream end when `isEnabled == true`,
|
||
uses a fixed **30s delay**, and is properly guarded against intentional disconnects
|
||
(double guard + cancellable work item).
|
||
- **No**, it does **not** use exponential backoff. The delay is constant 30s, indefinitely,
|
||
with no cap, no jitter, and no escalation/de-escalation. This creates retry-storm and
|
||
thundering-herd risks under sustained or synchronized outages.
|
||
|
||
---
|
||
|
||
## 5. Recommendations
|
||
|
||
### R1 — Ingress / LB configuration (ops, no code change)
|
||
|
||
Configure the ingress in front of bugger-feishu:
|
||
1. `proxy_read_timeout 60s;` (or higher) — must exceed the 30s heartbeat with margin.
|
||
2. `proxy_buffering off;` on the SSE location — or ensure `X-Accel-Buffering: no` is honored.
|
||
3. `proxy_send_timeout 60s;` — symmetric.
|
||
4. Verify the cloud LB idle-timeout (if any, in front of the ingress) is also ≥ 60s — some
|
||
LBs default to 30s or less and override the ingress setting.
|
||
|
||
### R2 — Verify heartbeat is deployed (ops, no code change)
|
||
|
||
Confirm in the bugger-feishu repo that the 30s `: heartbeat\n\n` emission is actually
|
||
running in the deployed image, not just specified in this repo's docs. If absent, the
|
||
ingress will drop the connection on every idle cycle regardless of R1.
|
||
|
||
### R3 — Add exponential backoff with jitter and cap (code change)
|
||
|
||
Modify `BitableEventService.swift` to replace the fixed `reconnectDelay` with a backoff
|
||
strategy. Suggested parameters:
|
||
|
||
```
|
||
attempt 1: delay = 1s (±20% jitter)
|
||
attempt 2: delay = 2s (±20% jitter)
|
||
attempt 3: delay = 4s (±20% jitter)
|
||
attempt 4: delay = 8s (±20% jitter)
|
||
attempt 5: delay = 16s (±20% jitter)
|
||
attempt 6+: delay = 30s (±20% jitter) ← cap
|
||
```
|
||
|
||
- Base: `delay = min(cap, base * 2^(attempt-1))`
|
||
- Cap: 30s (preserves current steady-state behavior)
|
||
- Jitter: ±20% of computed delay (prevents thundering herd)
|
||
- Reset attempt counter on a successful `connected` event from the server.
|
||
|
||
### R4 — Catch-up fetch on reconnect (optional code change)
|
||
|
||
On a successful reconnect, trigger `PollerService.shared.fetchNow()` once to recover any
|
||
events missed during the downtime window, rather than waiting for the next poll cycle.
|
||
This bounds event freshness loss to one fetch rather than the full poll interval.
|
||
|
||
---
|
||
|
||
## 6. Key Code Locations
|
||
|
||
All paths in `/Users/tigeren/Dev/xorbitlab/bugger/`:
|
||
|
||
| Location | What |
|
||
|---|---|
|
||
| `Sources/Services/BitableEventService.swift:25` | `reconnectDelay = 30` (fixed, no backoff) |
|
||
| `Sources/Services/BitableEventService.swift:21-23` | `isEnabled` guard |
|
||
| `Sources/Services/BitableEventService.swift:61-66` | Client idle-timeout config (300s request, infinite resource) |
|
||
| `Sources/Services/BitableEventService.swift:83` | Request timeout (300s) |
|
||
| `Sources/Services/BitableEventService.swift:92-100` | `closeStream` — cancels pending reconnect |
|
||
| `Sources/Services/BitableEventService.swift:102-124` | `handleStreamEnd` — auto-reconnect scheduling |
|
||
| `Sources/Services/BitableEventService.swift:126-138` | `handleEvent` — `change`→`fetchNow`, heartbeat no-op |
|
||
| `Sources/Services/BitableEventService.swift:170-226` | `SSESessionDelegate` — frame parser, comment/heartbeat ignored |
|
||
| `Sources/AppDelegate.swift:33` | `connect()` on app launch |
|
||
| `Sources/Views/Settings/SettingsView.swift:293` | `reconnect()` on settings save |
|
||
| `Sources/Views/Settings/SettingsView.swift:181` | `disconnect()` on field clear |
|
||
| `docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:686-693` | Server heartbeat spec (30s, must verify deployed) |
|
||
| `docs/BITABLE_CHANGE_NOTIFICATION_IMPL.md:707-711` | Server SSE response headers (keep-alive, no buffering) |
|
||
| `docs/TECH_INVESTIGATION.md:56-58,424,544` | Backoff references (rejected WSClient design only) |
|