725 lines
31 KiB
Markdown
725 lines
31 KiB
Markdown
# Push Reliability — Design & Implementation Plan
|
||
|
||
**Date:** 2026-07-10
|
||
**Status:** Draft — pending review
|
||
**Scope:** Bugger (macOS SSE client) + bugger-feishu (k8s SSE server)
|
||
|
||
---
|
||
|
||
## 1. Problem Statement
|
||
|
||
Push notifications (SSE `change` events from bugger-feishu → Bugger) can be
|
||
**delayed or missed entirely**. The root cause is unknown — it could be:
|
||
|
||
- **Feishu side** failing to deliver WebSocket events to bugger-feishu.
|
||
- **bugger-feishu** failing to push the SSE event to the client (e.g. connection
|
||
dropped, queue overflow, server crash).
|
||
- **Ingress** (k8s nginx / cloud LB) silently closing the long-lived SSE
|
||
connection before the heartbeat resets the idle timer (see
|
||
`SSE_TIMEOUT_ANALYSIS.md`).
|
||
|
||
The current reconnection strategy (fixed 30s delay, no jitter) compounds the
|
||
problem: when the ingress drops many clients simultaneously, they all retry at
|
||
the same instant — a thundering herd that can overwhelm the ingress and prevent
|
||
any client from reconnecting.
|
||
|
||
### Goals
|
||
|
||
| # | Goal | Owner |
|
||
|---|------|-------|
|
||
| 1 | Reconnect with **exponential backoff + jitter** to avoid retry storms | Bugger |
|
||
| 2 | bugger-feishu exposes an **owned-records endpoint** so Bugger can calibrate | bugger-feishu |
|
||
| 3 | Bugger has an **advanced settings toggle** (disabled by default) to enable periodic calibration | Bugger |
|
||
| 4 | Bugger **optimizes the Feishu query** — fetch only the owner's bugs, not all records | Bugger |
|
||
|
||
---
|
||
|
||
## 2. Architecture Overview
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────┐
|
||
│ Bugger (macOS) │
|
||
│ │
|
||
│ ┌──────────────────┐ SSE push ┌──────────────────────────────┐ │
|
||
│ │ BitableEventSvc │ ◀─────────────│ Ingress → bugger-feishu │ │
|
||
│ │ (exponential │ change │ (SSE server, 30s heartbeat) │ │
|
||
│ │ backoff+jitter)│ └──────────────────────────────┘ │
|
||
│ └────────┬─────────┘ │ │
|
||
│ │ fetchNow() │ GET /owned_records │
|
||
│ ▼ ▼ │
|
||
│ ┌──────────────────┐ ┌──────────────────────┐ │
|
||
│ │ PollerService │ │ CalibrationService │ │
|
||
│ │ (periodic poll │ │ (optional, every │ │
|
||
│ │ + fetchNow) │ │ 5 min, compares │ │
|
||
│ └────────┬─────────┘ │ record IDs) │ │
|
||
│ │ └──────────┬───────────┘ │
|
||
│ ▼ │ mismatch? │
|
||
│ ┌──────────────────┐ │ │
|
||
│ │ FeishuService │ ◀─────────────────────┘ fetchNow() │
|
||
│ │ (search records │ │
|
||
│ │ with filter) │ │
|
||
│ └──────────────────┘ │
|
||
└──────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
**Data flow for calibration:**
|
||
1. `CalibrationService` calls `GET {feishuAppBaseURL}/api/v1/bitable/owned_records`
|
||
2. bugger-feishu returns the record IDs assigned to this user (from its in-memory cache)
|
||
3. Bugger compares the returned IDs with `BugStore.shared.bugs.map(\.id)`
|
||
4. On mismatch → triggers `PollerService.shared.fetchNow()` → `FeishuService` fetches
|
||
only the owner's bugs (server-side filtered) → `BugStore` updates
|
||
|
||
---
|
||
|
||
## 3. Design Details
|
||
|
||
### 3.1 Item 1 — Exponential Backoff with Jitter
|
||
|
||
**Problem:** `BitableEventService.swift:25` uses a fixed `reconnectDelay = 30`.
|
||
All clients retry on the same cadence, creating synchronized load spikes.
|
||
|
||
**Design:**
|
||
|
||
Replace the fixed delay with exponential backoff + decorrelated jitter:
|
||
|
||
```
|
||
attempt 1: delay ≈ 1s (±25% jitter)
|
||
attempt 2: delay ≈ 2s (±25% jitter)
|
||
attempt 3: delay ≈ 4s (±25% jitter)
|
||
attempt 4: delay ≈ 8s (±25% jitter)
|
||
attempt 5: delay ≈ 16s (±25% jitter)
|
||
attempt 6+: delay ≈ 30s (±25% jitter) ← cap
|
||
```
|
||
|
||
**Formula:**
|
||
```
|
||
base = 1.0 (seconds)
|
||
factor = 2.0
|
||
cap = 30.0 (seconds) — preserves current steady-state behavior
|
||
jitter = ±25% (random uniform in [0.75, 1.25])
|
||
|
||
raw_delay = min(cap, base * factor^(attempt - 1))
|
||
delay = raw_delay * Double.random(in: 0.75...1.25)
|
||
```
|
||
|
||
**State management:**
|
||
- `reconnectAttempt: Int = 0` — incremented on each `handleStreamEnd`, reset to 0
|
||
when a `connected` event is received from the server.
|
||
- The attempt counter is an instance variable on `BitableEventService`, mutated
|
||
only on the main queue (all callbacks are already routed to main).
|
||
|
||
**Reset logic:**
|
||
In `handleEvent`, when event == `"connected"`:
|
||
```swift
|
||
reconnectAttempt = 0
|
||
```
|
||
This ensures the backoff de-escalates as soon as the connection is healthy.
|
||
|
||
**No catch-up fetch on reconnect (unchanged):**
|
||
The existing `handleEvent` already calls `PollerService.shared.fetchNow()` on
|
||
`change` events. The calibration service (item 3) provides the safety net for
|
||
missed events, so we do not add an explicit catch-up fetch on reconnect.
|
||
|
||
**Changes:**
|
||
| File | Change |
|
||
|------|--------|
|
||
| `bugger/Sources/Services/BitableEventService.swift` | Replace `reconnectDelay` constant with backoff logic; add `reconnectAttempt` counter; reset on `connected` event |
|
||
|
||
---
|
||
|
||
### 3.2 Item 2 — bugger-feishu Owned-Records Endpoint
|
||
|
||
**Problem:** Bugger has no way to verify it has received all change pushes.
|
||
If a push is missed (SSE connection down, Feishu WS event lost), the bug list
|
||
is stale until the next periodic poll.
|
||
|
||
**Design:**
|
||
|
||
New endpoint on bugger-feishu:
|
||
|
||
```
|
||
GET /api/v1/bitable/owned_records
|
||
?file_token={app_token}
|
||
&assignee_field={field_name}
|
||
&assignee_name={user_name}
|
||
```
|
||
|
||
**Response (200):**
|
||
```json
|
||
{
|
||
"file_token": "bascnXXX",
|
||
"assignee_name": "张三",
|
||
"assignee_field": "负责人",
|
||
"record_ids": ["recAAA", "recBBB", "recCCC"],
|
||
"count": 3,
|
||
"cache_status": "warm",
|
||
"server_time": 1720000000
|
||
}
|
||
```
|
||
|
||
**`cache_status` values:**
|
||
| Value | Meaning | Bugger action |
|
||
|-------|---------|---------------|
|
||
| `"warm"` | Cache has been populated (WS events received or full pull done) | Compare IDs, calibrate on mismatch |
|
||
| `"cold"` | Cache is empty — no WS events received yet, no full pull done | **Skip calibration** — would cause false positives |
|
||
| `"stale"` | Cache exists but hasn't been refreshed recently (> 1h since last update) | Compare IDs, but log a warning |
|
||
|
||
**Implementation on bugger-feishu:**
|
||
|
||
The server already maintains `self._cache: Dict[str, Dict[str, Dict[str, str]]]`
|
||
(file_token → assignee_field → record_id → assignee_value) in
|
||
`BitableEventService`. The endpoint reads from this cache — **no Feishu API call**
|
||
is needed, making it fast and cheap.
|
||
|
||
```python
|
||
# bitable_event_service.py — new method
|
||
|
||
def get_owned_record_ids(
|
||
self, file_token: str, assignee_field: str, assignee_name: str
|
||
) -> dict:
|
||
"""Return the record IDs assigned to a user, from the in-memory cache."""
|
||
file_cache = self._cache.get(file_token, {})
|
||
field_cache = file_cache.get(assignee_field, {})
|
||
|
||
if not field_cache:
|
||
# Check if we've ever seen this file_token at all
|
||
if file_token not in self._cache:
|
||
return {"record_ids": [], "cache_status": "cold", "count": 0}
|
||
return {"record_ids": [], "cache_status": "cold", "count": 0}
|
||
|
||
# Check staleness: if no table_id learned yet, cache may be incomplete
|
||
has_table_id = file_token in self._table_ids
|
||
|
||
owned = [
|
||
rid for rid, assignee in field_cache.items()
|
||
if assignee == assignee_name
|
||
]
|
||
status = "warm" if has_table_id else "stale"
|
||
return {
|
||
"record_ids": owned,
|
||
"cache_status": status,
|
||
"count": len(owned),
|
||
}
|
||
```
|
||
|
||
```python
|
||
# bitable_subscription.py — new route
|
||
|
||
@router.get("/owned_records")
|
||
async def owned_records(
|
||
file_token: str = Query(...),
|
||
assignee_field: str = Query(...),
|
||
assignee_name: str = Query(...),
|
||
):
|
||
service = getattr(request.app.state, "bitable_event_service", None)
|
||
if service is None:
|
||
return {"record_ids": [], "cache_status": "cold", "count": 0}
|
||
|
||
result = service.get_owned_record_ids(file_token, assignee_field, assignee_name)
|
||
result["file_token"] = file_token
|
||
result["assignee_name"] = assignee_name
|
||
result["assignee_field"] = assignee_field
|
||
result["server_time"] = int(time.time())
|
||
return result
|
||
```
|
||
|
||
**Why read from cache, not live Feishu API?**
|
||
- The cache is already maintained and reconciled every 24h by the server.
|
||
- A live API call per calibration request (every 5 min × N clients) would be
|
||
expensive and rate-limited.
|
||
- The cache is "good enough" for calibration — it catches missed pushes, which
|
||
is the goal. The periodic reconciliation task (every 24h) ensures the cache
|
||
stays eventually consistent.
|
||
|
||
**Changes:**
|
||
| File | Change |
|
||
|------|--------|
|
||
| `bugger-feishu/app/services/bitable_event_service.py` | Add `get_owned_record_ids()` method |
|
||
| `bugger-feishu/app/api/bitable_subscription.py` | Add `GET /owned_records` route |
|
||
| `bugger-feishu/tests/test_bitable_change_notification.py` | Add unit tests for the endpoint |
|
||
|
||
---
|
||
|
||
### 3.3 Item 3 — Bugger Calibration Service + Advanced Settings Toggle
|
||
|
||
**Problem:** Even with backoff, the SSE connection may be unreliable (ingress
|
||
issues, server restarts). A periodic calibration check provides a safety net.
|
||
|
||
**Design:**
|
||
|
||
#### 3.3.1 AppConfig — new field
|
||
|
||
```swift
|
||
// AppConfig.swift
|
||
/// When true, Bugger periodically queries bugger-feishu for the list of
|
||
/// record IDs assigned to the current user and triggers a fetchNow() if
|
||
/// the list doesn't match the bugs in BugStore. Disabled by default —
|
||
/// only useful when feishuAppBaseURL is set and push reliability is a
|
||
/// concern.
|
||
var calibrationEnabled: Bool = false
|
||
```
|
||
|
||
- Default: `false` (disabled)
|
||
- Only meaningful when `feishuAppBaseURL` is set (the calibration endpoint
|
||
lives on the same server)
|
||
- Tolerant decoding (like all other AppConfig fields) so existing persisted
|
||
configs don't break
|
||
|
||
#### 3.3.2 CalibrationService — new service
|
||
|
||
```swift
|
||
// Sources/Services/CalibrationService.swift
|
||
|
||
final class CalibrationService {
|
||
static let shared = CalibrationService()
|
||
|
||
private var timer: Timer?
|
||
private let interval: TimeInterval = 300 // 5 minutes
|
||
private var isRunning = false
|
||
private let session = URLSession.shared
|
||
|
||
private init() {}
|
||
|
||
func start() {
|
||
guard !isRunning else { return }
|
||
guard let config = AppStateService.shared.config,
|
||
config.calibrationEnabled,
|
||
!config.feishuAppBaseURL.isEmpty else { return }
|
||
isRunning = true
|
||
|
||
// First check after 60s (let the initial fetch + SSE connect settle)
|
||
timer = Timer.scheduledTimer(withTimeInterval: interval, repeats: true) { [weak self] _ in
|
||
Task { await self?.calibrate() }
|
||
}
|
||
timer?.tolerance = 30
|
||
// Delay the first run
|
||
DispatchQueue.main.asyncAfter(deadline: .now() + 60) { [weak self] in
|
||
Task { await self?.calibrate() }
|
||
}
|
||
}
|
||
|
||
func stop() {
|
||
timer?.invalidate()
|
||
timer = nil
|
||
isRunning = false
|
||
}
|
||
|
||
func restart() {
|
||
stop()
|
||
start()
|
||
}
|
||
|
||
private func calibrate() async {
|
||
guard let config = AppStateService.shared.config,
|
||
!config.feishuAppBaseURL.isEmpty,
|
||
!config.assigneeName.isEmpty,
|
||
!config.fieldMappings.assigneeField.isEmpty else { return }
|
||
|
||
// Build URL: {feishuAppBaseURL}/api/v1/bitable/owned_records?...
|
||
guard let url = buildURL(config: config) else { return }
|
||
|
||
do {
|
||
let (data, response) = try await session.data(from: url)
|
||
guard let http = response as? HTTPURLResponse, http.statusCode == 200 else { return }
|
||
|
||
let result = try JSONDecoder().decode(OwnedRecordsResponse.self, from: data)
|
||
|
||
// Skip if cache is cold — would cause false positives
|
||
guard result.cacheStatus == "warm" || result.cacheStatus == "stale" else { return }
|
||
|
||
let serverIDs = Set(result.recordIds)
|
||
let localIDs = Set(BugStore.shared.bugs.map(\.id))
|
||
|
||
// Only calibrate if we have local data (skip on fresh launch
|
||
// before the first fetch completes)
|
||
guard !localIDs.isEmpty else { return }
|
||
|
||
let mismatch = serverIDs != localIDs
|
||
if mismatch {
|
||
BuggerLog.info("CalibrationService: mismatch detected (server=\(serverIDs.count), local=\(localIDs.count)), triggering fetchNow")
|
||
await PollerService.shared.fetchNow()
|
||
}
|
||
} catch {
|
||
BuggerLog.error("CalibrationService: error — \(error.localizedDescription)")
|
||
}
|
||
}
|
||
|
||
private func buildURL(config: AppConfig) -> URL? {
|
||
let base = config.feishuAppBaseURL.trimmingCharacters(in: CharacterSet(charactersIn: "/"))
|
||
guard var comps = URLComponents(string: "\(base)/api/v1/bitable/owned_records") else { return nil }
|
||
comps.queryItems = [
|
||
URLQueryItem(name: "file_token", value: config.appToken),
|
||
URLQueryItem(name: "assignee_field", value: config.fieldMappings.assigneeField),
|
||
URLQueryItem(name: "assignee_name", value: config.assigneeName),
|
||
]
|
||
return comps.url
|
||
}
|
||
}
|
||
|
||
// Response model
|
||
struct OwnedRecordsResponse: Decodable {
|
||
let recordIds: [String]
|
||
let cacheStatus: String
|
||
let count: Int
|
||
|
||
enum CodingKeys: String, CodingKey {
|
||
case recordIds = "record_ids"
|
||
case cacheStatus = "cache_status"
|
||
case count
|
||
}
|
||
}
|
||
```
|
||
|
||
#### 3.3.3 Settings UI — advanced section
|
||
|
||
In `SettingsView.swift`, inside the existing `DisclosureGroup` for Advanced
|
||
options, add a new section before Field Mappings:
|
||
|
||
```swift
|
||
// Advanced section — calibration toggle
|
||
VStack(alignment: .leading, spacing: 4) {
|
||
Toggle("Enable calibration check", isOn: $config.calibrationEnabled)
|
||
Text("Periodically verifies with the notification server that all your assigned bugs are in sync. Helps recover missed push notifications. Only works when a Subscribe base URL is set.")
|
||
.font(.caption)
|
||
.foregroundStyle(.secondary)
|
||
}
|
||
```
|
||
|
||
#### 3.3.4 Lifecycle wiring
|
||
|
||
| Location | Action |
|
||
|----------|--------|
|
||
| `AppDelegate.applicationDidFinishLaunching` | After `BitableEventService.shared.connect()`, add `CalibrationService.shared.start()` |
|
||
| `SettingsView.save()` | After `BitableEventService.shared.reconnect()`, add `CalibrationService.shared.restart()` |
|
||
| `SettingsView` Disconnect button | Add `CalibrationService.shared.stop()` |
|
||
|
||
**Changes:**
|
||
| File | Change |
|
||
|------|--------|
|
||
| `bugger/Sources/Models/AppConfig.swift` | Add `calibrationEnabled` field + CodingKey + tolerant decode |
|
||
| `bugger/Sources/Services/CalibrationService.swift` | **New file** — calibration service |
|
||
| `bugger/Sources/Views/Settings/SettingsView.swift` | Add toggle in Advanced section; wire lifecycle in `save()` and Disconnect |
|
||
| `bugger/Sources/AppDelegate.swift` | Start calibration on launch |
|
||
|
||
---
|
||
|
||
### 3.4 Item 4 — Optimize Feishu Query (Filter by Assignee)
|
||
|
||
**Problem:** `FeishuService.fetchBugs()` fetches **ALL** records from the Bitable
|
||
table (up to 25,000, paginated 500/page = 50 API calls), then filters
|
||
client-side by assignee name. For large tables, this is slow and wasteful.
|
||
|
||
**Feishu API capabilities (verified from official docs):**
|
||
|
||
Two approaches are available:
|
||
|
||
| Approach | Endpoint | Filter format | Person field support |
|
||
|----------|----------|---------------|---------------------|
|
||
| **A. List records** | `GET .../records` | `filter` query param: `CurrentValue.[FieldName]="value"` | Unreliable for Person fields — test code already notes it may not work |
|
||
| **B. Search records** | `POST .../records/search` | JSON body with structured `filter` object | **Supported** — Person fields require `open_id` (user ID), not display name |
|
||
|
||
**Key constraint (from Feishu filter guide):**
|
||
> Person field value: `["ou_9a971ded01b4ca66f4798549878abcef"]` — Fill in the
|
||
> corresponding user **ID**. The user ID type must match the `user_id_type`
|
||
> parameter (default: `open_id`).
|
||
|
||
This means filtering by Person fields requires the user's `open_id`, which is
|
||
only available if the user authenticated via OAuth. If the user entered their
|
||
name manually in Settings (no OAuth), we cannot filter Person fields server-side.
|
||
|
||
**Chosen strategy — multi-tier with fallback:**
|
||
|
||
```
|
||
1. If user's open_id is available (from OAuth):
|
||
→ Use search endpoint with filter: {field_name, operator: "is", value: [open_id]}
|
||
→ Works for Person-type assignee fields
|
||
|
||
2. If open_id not available OR search returns error/0 results:
|
||
→ Use list endpoint with filter: CurrentValue.[assigneeField]="assigneeName"
|
||
→ Works for text-type assignee fields
|
||
|
||
3. If both fail:
|
||
→ Fall back to current unfiltered list endpoint (fetch all, filter client-side)
|
||
```
|
||
|
||
**Additionally — `field_names` optimization (applies to all tiers):**
|
||
|
||
Both list and search endpoints support requesting only specific fields. By
|
||
passing only the fields Bugger needs (title, priority, status, assignee, reporter,
|
||
customer, created_at, updated_at), we reduce the per-record payload significantly
|
||
— especially in tables with many columns or large text/attachment fields.
|
||
|
||
#### 3.4.1 Capture user's open_id during OAuth
|
||
|
||
Extend `UserInfoData` to include `openId`:
|
||
|
||
```swift
|
||
// FeishuModels.swift
|
||
struct UserInfoData: Decodable {
|
||
let name: String?
|
||
let enName: String?
|
||
let openId: String? // NEW — needed for server-side Person field filtering
|
||
}
|
||
```
|
||
|
||
Extend `TokenManager` to cache the open_id:
|
||
|
||
```swift
|
||
// TokenManager.swift
|
||
private let openIdKey = "feishu.open_id"
|
||
|
||
var cachedOpenId: String? {
|
||
keychain.read(openIdKey)
|
||
}
|
||
|
||
// In handleCallback() and refreshAccessToken() — after getting user info:
|
||
// Store open_id alongside tokens
|
||
```
|
||
|
||
#### 3.4.2 New FeishuService method — search records with filter
|
||
|
||
```swift
|
||
// FeishuService.swift
|
||
|
||
func fetchBugs(
|
||
config: AppConfig,
|
||
assigneeName: String,
|
||
accessToken: String,
|
||
userOpenId: String? = nil // NEW parameter
|
||
) async throws -> [Bug] {
|
||
// Try filtered search first, fall back to unfiltered list
|
||
let records = try await fetchRecordsFiltered(
|
||
config: config, assigneeName: assigneeName,
|
||
accessToken: accessToken, userOpenId: userOpenId
|
||
)
|
||
return records.map { BugMapper.map($0, config: config) }
|
||
.filter { bug in bug.assignee.localizedCaseInsensitiveContains(assigneeName) }
|
||
}
|
||
|
||
private func fetchRecordsFiltered(
|
||
config: AppConfig, assigneeName: String,
|
||
accessToken: String, userOpenId: String?
|
||
) async throws -> [RecordItem] {
|
||
// Tier 1: search endpoint with open_id filter (Person fields)
|
||
if let openId = userOpenId {
|
||
if let records = try? await searchRecords(
|
||
config: config, accessToken: accessToken,
|
||
filterConditions: [
|
||
["field_name": config.fieldMappings.assigneeField,
|
||
"operator": "is",
|
||
"value": [openId]]
|
||
]
|
||
), !records.isEmpty {
|
||
return records
|
||
}
|
||
}
|
||
|
||
// Tier 2: list endpoint with CurrentValue filter (text fields)
|
||
let filterStr = "CurrentValue.[\(config.fieldMappings.assigneeField)]=\"\(assigneeName)\""
|
||
if let records = try? await listRecords(
|
||
config: config, accessToken: accessToken, filter: filterStr
|
||
), !records.isEmpty {
|
||
return records
|
||
}
|
||
|
||
// Tier 3: fallback — unfiltered list (current behavior)
|
||
return try await listRecords(config: config, accessToken: accessToken, filter: nil)
|
||
}
|
||
```
|
||
|
||
**Note:** The client-side `.filter` on `assigneeName` is retained as a safety
|
||
net even after server-side filtering. It's a no-op if the filter worked
|
||
correctly, but prevents incorrect records from appearing if the filter is
|
||
imperfect.
|
||
|
||
#### 3.4.3 PollerService — pass open_id to FeishuService
|
||
|
||
```swift
|
||
// PollerService.swift — in performFetch()
|
||
let openId = TokenManager.shared.cachedOpenId
|
||
let bugs = try await feishuService.fetchBugs(
|
||
config: config,
|
||
assigneeName: assignee,
|
||
accessToken: token,
|
||
userOpenId: openId // NEW
|
||
)
|
||
```
|
||
|
||
**Changes:**
|
||
| File | Change |
|
||
|------|--------|
|
||
| `bugger/Sources/Services/Feishu/FeishuModels.swift` | Add `openId` to `UserInfoData` |
|
||
| `bugger/Sources/Services/TokenManager.swift` | Cache `open_id` from OAuth; expose `cachedOpenId` |
|
||
| `bugger/Sources/Services/Feishu/FeishuService.swift` | Add `searchRecords` method; add `fetchRecordsFiltered` with 3-tier fallback; add `field_names` to list calls; accept `userOpenId` parameter |
|
||
| `bugger/Sources/Services/PollerService.swift` | Pass `userOpenId` to `fetchBugs` |
|
||
|
||
---
|
||
|
||
## 4. Ingress Considerations
|
||
|
||
The user suspects the ingress plays a role in missed pushes. The code changes
|
||
above make Bugger **resilient** to ingress issues, but the ingress itself should
|
||
also be configured correctly (per `SSE_TIMEOUT_ANALYSIS.md` R1):
|
||
|
||
| Setting | Required value | Why |
|
||
|---------|---------------|-----|
|
||
| `proxy_read_timeout` | ≥ 60s | Must exceed the 30s heartbeat with margin |
|
||
| `proxy_buffering` | `off` | Prevents heartbeats from being buffered and never sent |
|
||
| `proxy_send_timeout` | ≥ 60s | Symmetric to read timeout |
|
||
| Cloud LB idle-timeout | ≥ 60s | Some LBs override ingress settings |
|
||
|
||
**How the code changes mitigate ingress issues:**
|
||
- **Backoff + jitter** → When the ingress drops many SSE connections (e.g. on
|
||
pod restart), clients reconnect at randomized intervals instead of
|
||
simultaneously, reducing the load spike that could cause the ingress to reject
|
||
new connections.
|
||
- **Calibration service** → Even if the SSE connection is chronically
|
||
unreliable (ingress kills it every cycle), the 5-minute calibration check
|
||
ensures the bug list stays eventually consistent.
|
||
- **Optimized Feishu query** → When calibration triggers a `fetchNow()`, the
|
||
filtered query is much faster and lighter, reducing the load on both the
|
||
Feishu API and the client.
|
||
|
||
---
|
||
|
||
## 5. Implementation Plan
|
||
|
||
### Phase 1 — Bugger: Exponential Backoff (Item 1)
|
||
|
||
**Scope:** `BitableEventService.swift` only. No server changes. No new files.
|
||
|
||
| Step | Task | File |
|
||
|------|------|------|
|
||
| 1.1 | Remove `reconnectDelay` constant; add `reconnectAttempt`, `reconnectBaseDelay`, `reconnectMaxDelay`, `reconnectJitterRange` properties | `BitableEventService.swift` |
|
||
| 1.2 | Add `computeReconnectDelay() -> TimeInterval` method implementing the backoff + jitter formula | `BitableEventService.swift` |
|
||
| 1.3 | Update `handleStreamEnd` to call `computeReconnectDelay()` and increment `reconnectAttempt` | `BitableEventService.swift` |
|
||
| 1.4 | Update `handleEvent` to reset `reconnectAttempt = 0` on `"connected"` event | `BitableEventService.swift` |
|
||
| 1.5 | Reset `reconnectAttempt = 0` in `closeStream()` (clean disconnect resets the counter) | `BitableEventService.swift` |
|
||
|
||
**Testing:** Manual — disconnect Wi-Fi, observe reconnect delays in logs.
|
||
Verify: 1st retry ~1s, 2nd ~2s, 3rd ~4s... capping at ~30s. Reconnect after
|
||
recovery resets to 1s.
|
||
|
||
### Phase 2 — bugger-feishu: Owned-Records Endpoint (Item 2)
|
||
|
||
**Scope:** bugger-feishu server only.
|
||
|
||
| Step | Task | File |
|
||
|------|------|------|
|
||
| 2.1 | Add `get_owned_record_ids()` method to `BitableEventService` | `bitable_event_service.py` |
|
||
| 2.2 | Add `GET /owned_records` route to `bitable_subscription.py` | `bitable_subscription.py` |
|
||
| 2.3 | Add unit tests (mock cache warm/cold/stale states) | `test_bitable_change_notification.py` |
|
||
|
||
**Testing:** `pytest bugger-feishu/tests/test_bitable_change_notification.py`
|
||
|
||
### Phase 3 — Bugger: Calibration Service + Settings (Item 3)
|
||
|
||
**Scope:** Bugger client only. Depends on Phase 2 (endpoint must exist).
|
||
|
||
| Step | Task | File |
|
||
|------|------|------|
|
||
| 3.1 | Add `calibrationEnabled: Bool = false` to `AppConfig` (field + CodingKey + tolerant decode) | `AppConfig.swift` |
|
||
| 3.2 | Create `CalibrationService.swift` — timer, `calibrate()`, `OwnedRecordsResponse` model | **New file** |
|
||
| 3.3 | Add calibration toggle in Advanced section of `SettingsView` | `SettingsView.swift` |
|
||
| 3.4 | Wire `CalibrationService.shared.start()` in `AppDelegate` | `AppDelegate.swift` |
|
||
| 3.5 | Wire `CalibrationService.shared.restart()` in `SettingsView.save()` | `SettingsView.swift` |
|
||
| 3.6 | Wire `CalibrationService.shared.stop()` in Disconnect button | `SettingsView.swift` |
|
||
|
||
**Testing:** Manual — enable calibration in Advanced settings, verify the timer
|
||
fires every 5 min (check logs), verify `fetchNow()` triggers on ID mismatch.
|
||
|
||
### Phase 4 — Bugger: Optimized Feishu Query (Item 4)
|
||
|
||
**Scope:** Bugger client only. Independent of Phases 2-3.
|
||
|
||
| Step | Task | File |
|
||
|------|------|------|
|
||
| 4.1 | Add `openId` to `UserInfoData` | `FeishuModels.swift` |
|
||
| 4.2 | Cache `open_id` in `TokenManager` (store on OAuth callback + token refresh) | `TokenManager.swift` |
|
||
| 4.3 | Add `searchRecords()` method using `POST .../records/search` with structured filter | `FeishuService.swift` |
|
||
| 4.4 | Add `fetchRecordsFiltered()` with 3-tier fallback (open_id → name → unfiltered) | `FeishuService.swift` |
|
||
| 4.5 | Add `field_names` parameter to `listRecords`/`searchRecords` to reduce payload | `FeishuService.swift` |
|
||
| 4.6 | Update `PollerService.performFetch()` to pass `userOpenId` | `PollerService.swift` |
|
||
|
||
**Testing:** Manual — verify filtered query returns correct results (check
|
||
`BuggerLog.debug` output for which tier was used). Test with both Person-type
|
||
and text-type assignee fields. Verify fallback works when open_id is unavailable.
|
||
|
||
### Phase 5 — Verification & Documentation
|
||
|
||
| Step | Task |
|
||
|------|------|
|
||
| 5.1 | Build Bugger in Xcode — verify no compiler errors |
|
||
| 5.2 | Run bugger-feishu tests — `pytest` |
|
||
| 5.3 | End-to-end test: start bugger-feishu → connect Bugger → kill SSE → verify backoff → verify calibration catches missed push |
|
||
| 5.4 | Update `SSE_TIMEOUT_ANALYSIS.md` — mark R3 (backoff) as implemented, link to this plan |
|
||
|
||
---
|
||
|
||
## 6. Key Code Locations (current state)
|
||
|
||
All paths relative to `/Users/tigeren/Dev/aptsell/bugger-boundle/`:
|
||
|
||
### Bugger (macOS client)
|
||
|
||
| Location | What |
|
||
|----------|------|
|
||
| `bugger/Sources/Services/BitableEventService.swift:25` | `reconnectDelay = 30` (fixed — to be replaced with backoff) |
|
||
| `bugger/Sources/Services/BitableEventService.swift:102-124` | `handleStreamEnd` — reconnect scheduling |
|
||
| `bugger/Sources/Services/BitableEventService.swift:126-138` | `handleEvent` — `connected` event (no-op — to reset attempt counter) |
|
||
| `bugger/Sources/Services/BitableEventService.swift:140-165` | `buildURL` — SSE endpoint URL construction |
|
||
| `bugger/Sources/Services/PollerService.swift:88-126` | `performFetch` — calls `FeishuService.fetchBugs` |
|
||
| `bugger/Sources/Services/Feishu/FeishuService.swift:15-47` | `fetchBugs` — fetches ALL records, filters client-side |
|
||
| `bugger/Sources/Services/Feishu/FeishuService.swift:64-119` | `fetchPage` — list records API call |
|
||
| `bugger/Sources/Services/Feishu/FeishuModels.swift:54-58` | `UserInfoData` — has `name`, `enName` (needs `openId`) |
|
||
| `bugger/Sources/Services/TokenManager.swift:112-117` | `resolveAssigneeName` — returns name, not open_id |
|
||
| `bugger/Sources/Models/AppConfig.swift:3-87` | `AppConfig` — needs `calibrationEnabled` field |
|
||
| `bugger/Sources/Views/Settings/SettingsView.swift:140-170` | Advanced section — needs calibration toggle |
|
||
| `bugger/Sources/AppDelegate.swift:29-34` | App launch — needs `CalibrationService.start()` |
|
||
|
||
### bugger-feishu (SSE server)
|
||
|
||
| Location | What |
|
||
|----------|------|
|
||
| `bugger-feishu/app/api/bitable_subscription.py:24-94` | SSE `/events` endpoint — model for new `/owned_records` route |
|
||
| `bugger-feishu/app/services/bitable_event_service.py:49-66` | `BitableEventService` — maintains `_cache` (record_id → assignee) |
|
||
| `bugger-feishu/app/services/bitable_event_service.py:124-182` | `warm_cache` — full pull that populates the cache |
|
||
| `bugger-feishu/app/services/bitable_event_service.py:215-263` | `add_connection` / `remove_connection` — SSE lifecycle |
|
||
| `bugger-feishu/app/services/bitable_event_service.py:460-525` | Reconciliation task — periodic full pull (24h) |
|
||
| `bugger-feishu/app/main.py:88-91` | Router registration — add `/owned_records` route here |
|
||
|
||
---
|
||
|
||
## 7. Risks & Mitigations
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|------|--------|------------|
|
||
| Feishu search endpoint filter doesn't work for Person fields with open_id | Filtered query returns 0 results → fallback to unfiltered (current behavior) | 3-tier fallback strategy ensures no regression |
|
||
| Calibration false positives (server cache stale, triggers unnecessary fetchNow) | Extra Feishu API calls every 5 min | `cache_status` check — skip when cold; `localIDs.isEmpty` check — skip on fresh launch; 5-min interval limits impact |
|
||
| Server cache doesn't include newly assigned records (reconciliation runs every 24h) | Calibration misses recent assignments | The SSE push should catch these; calibration is a safety net, not the primary mechanism. Server reconciliation interval can be reduced if needed. |
|
||
| Backoff delays push recovery (30s cap vs. current 30s fixed) | Slightly slower first reconnect (1s vs 30s — actually faster!) | Backoff starts at 1s (faster than current 30s), caps at 30s. Net improvement. |
|
||
| `open_id` not available (user entered name manually, no OAuth) | Can't filter Person fields server-side | Fall back to text filter or unfiltered list. No regression. |
|
||
|
||
---
|
||
|
||
## 8. Open Questions
|
||
|
||
1. **Calibration interval** — 5 min is proposed. Should it be configurable in
|
||
settings, or fixed? (Recommend: fixed for simplicity; can add a setting
|
||
later if needed.)
|
||
|
||
2. **Server cache freshness** — The server's reconciliation runs every 24h.
|
||
Should the calibration endpoint trigger a cache refresh if the cache is
|
||
stale? (Recommend: no — keep the endpoint read-only and cheap. The
|
||
reconciliation task handles freshness.)
|
||
|
||
3. **Person field filter with `contains` operator** — Could we use
|
||
`operator: "contains"` with the user's name instead of `is` with open_id?
|
||
The Feishu docs say Person field values must be user IDs, but `contains`
|
||
might behave differently. (Recommend: test during implementation; if
|
||
`contains` with name works, it simplifies the flow by not requiring open_id.)
|
||
|
||
4. **`field_names` URL encoding** — The list endpoint expects `field_names` as
|
||
a JSON array string (e.g. `["Title","Status"]`). Need to verify that
|
||
`URLQueryItem` encodes this correctly for the Feishu API.
|