231 lines
12 KiB
Markdown
231 lines
12 KiB
Markdown
# Backend Review — `htracker` retranslator + monitor
|
||
|
||
**Date:** 2026-06-19 · **Reviewer:** Claude (for Leo) · branch `feature/test500unidades`
|
||
**Scope:** `retranslator.py`, `retranslator_async.py`, `gps_logger.py`,
|
||
`web_monitor.py`, `notifier.py`, `load_test.py`, service/unit files.
|
||
**Why:** the RCA flagged a **backend common-mode** (H2) as the cheapest, most
|
||
likely explanation for *all units disconnecting at once* — and for a *single unit
|
||
"dying" and never coming back*. **This review found concrete mechanisms for both,
|
||
in software, independent of the field hardware.**
|
||
|
||
Severity: 🔴 can take units off Wialon · 🟠 reliability/scale · 🟡 hygiene.
|
||
|
||
---
|
||
|
||
## ✅ RESOLUTION (2026-06-20) — fixed & deployed to production
|
||
|
||
All three 🔴 mechanisms were fixed in `retranslator.py` and **deployed** via the new
|
||
`deploy.sh` (backup + health-check + auto-rollback; gateway/monitor/notifier all active).
|
||
Approach: **harden in place**, not the async rewrite.
|
||
|
||
| # | Issue | Fix shipped |
|
||
|---|-------|-------------|
|
||
| P1 | Synchronous Wialon I/O in the receive loop (HOL blocking) | `WialonForwarder` worker pool decouples TCP I/O from UDP ingest; the receive loop never blocks. Verified: a healthy unit kept forwarding while another unit's upstream was dark. |
|
||
| P1 | Anti-jump filter permanently locks units out | Time-aware plausibility envelope (`MAX_SPEED·Δt`) + self-heal (re-anchor after N rejects) + clear anchor on disconnect. Verified: a teleporting unit recovers instead of being black-holed. |
|
||
| P2 | Identity keyed by source IP (CGNAT) | Transport = full `(ip, port)`; all per-device state + GGA/RMC pairing keyed by `cell_id`. Verified: two SIMs on one IP don't cross-mix. |
|
||
| 🟠 | `SQL_THROTTLE_S = 0`; thread-safety | Throttle set to 300 s; stats threads snapshot shared dicts with `list(...)`. |
|
||
| 🟠 | Load test measured the wrong thing | Superseded by an end-to-end smoke test (mock Wialon + slow/dark upstream + CGNAT + jump cases). |
|
||
| 🟡 | Secrets in repo | Parameterized out of README/compose into `.env` (gitignored) + `.env.example`. **Open:** rotate the password + purge git history. |
|
||
|
||
Also shipped: the **backend-victim signal** in the monitor (ONLINE-on-UDP but not forwarding),
|
||
an **offline-unit detail panel**, a ~100× **log-volume trim** + off-box shipping to **Grafana
|
||
Loki** (via Alloy → journald, capped), and the unit-aware `deploy.sh`. Outstanding: secret
|
||
rotation + history purge; optional convergence of the dead retranslator variants; code review
|
||
(Leo to run). The original analysis is preserved below for the record.
|
||
|
||
---
|
||
|
||
## TL;DR (read this if nothing else)
|
||
|
||
1. **Production runs the *blocking* `retranslator.py`** (per `htracker.service`),
|
||
which sends each position to Wialon and **waits for the TCP ack — and even the
|
||
20 s connect — inside the single UDP receive loop**. One slow/unreachable
|
||
upstream stalls the loop and **every device drops off Wialon at once**. This is
|
||
a textbook common-mode and will get dramatically worse at 500 units. 🔴
|
||
2. **The anti-jump filter can permanently black-hole a healthy unit.** On a
|
||
rejected "jump" it returns **without updating the reference position**, and the
|
||
reference **survives disconnects**. A unit that moves >500 m while offline (or
|
||
whose first fix after reconnect is an outlier) has **100 % of its packets
|
||
dropped forever** — it transmits fine but never appears in Wialon. 🔴
|
||
3. **Device identity is keyed by source IP**, which collides under carrier CGNAT
|
||
(500 SIMs, shared/rotating IPs) → **cross-device data mixing** → false jumps →
|
||
disconnects. 🔴
|
||
4. There's a **correct async rewrite** (`retranslator_async.py`) that fixes the
|
||
blocking model — but it's **not deployed**, still **keys by IP**, and **dropped
|
||
the anti-jump filter** entirely.
|
||
|
||
**Powerful triage signal (use today):** Helgen's own monitor marks a unit
|
||
**ONLINE on UDP receipt** (`_mark_seen`), *before* the anti-jump filter and the
|
||
Wialon send. So a unit hit by #2 or #1 shows **ONLINE in htracker but OFFLINE in
|
||
Wialon**. **Any unit with that split is a backend victim, not a dead radio** —
|
||
check this against ZPM 020 / RS-SAN-001 before blaming the hardware.
|
||
|
||
---
|
||
|
||
## What is actually running (confirm this first)
|
||
|
||
- `htracker.service` → `ExecStart=/usr/bin/python3 retranslator.py` ← **the
|
||
blocking gateway**, "Up to 500 Wialon TCP sockets," `LimitNOFILE=8192`.
|
||
- `htracker_ctl.sh` → also launches `retranslator.py`.
|
||
- `run_retranslator.sh` → launches **`rtk2mqtt_retranslator.py`** (a third
|
||
variant). ⚠️ **Ambiguity:** confirm which process is live in prod
|
||
(`systemctl status htracker`), because the review's #1/#2 are about
|
||
`retranslator.py`.
|
||
|
||
---
|
||
|
||
## 🔴 P1 — Synchronous Wialon I/O in the single receive loop (HOL blocking)
|
||
|
||
`main()` is one thread doing `recvfrom` → parse → `_process_device_telemetry` →
|
||
`send_to_wialon`, which **blocks**:
|
||
|
||
- on connect: `s.connect((DEST_HOST, DEST_PORT))` with `TCP_TIMEOUT_S = 20`;
|
||
- on every packet: `current_sock.sendall(...)` then **`current_sock.recv(1024)`**
|
||
waiting for the Wialon ack.
|
||
|
||
While that one device's socket is slow, **no UDP from any other device is read**;
|
||
the kernel UDP buffer fills and **datagrams from all 500 units are silently
|
||
dropped**. A brief upstream hiccup at `us.gpsgsm.org` therefore looks exactly like
|
||
"the whole fleet disconnected." Reconnect storms make it worse: cellular IP churn
|
||
closes/reopens sockets (each a 20 s-timeout `connect` in the loop) with
|
||
exponential backoff — all serialized in the same loop.
|
||
|
||
**Symptom match:** simultaneous death (A) + recurrent disconnects (B).
|
||
|
||
**Fix:** deploy the async model (decouple UDP ingest from Wialon I/O via a queue +
|
||
worker tasks; per-connection async with timeouts) — see `retranslator_async.py`,
|
||
which already does this. Until then, at minimum make `send_to_wialon`
|
||
non-blocking / move it off the receive loop, and bump `SO_RCVBUF`.
|
||
|
||
---
|
||
|
||
## 🔴 P1 — Anti-jump filter permanently locks units out
|
||
|
||
In `_process_device_telemetry`:
|
||
|
||
```python
|
||
_prev = device_last_pos.get(cell_id)
|
||
if _prev:
|
||
_dist = haversine(_prev, (_lat,_lon))
|
||
if _dist >= 0.5: # flat 500 m, ignores Δt
|
||
log.warning("Anti-Salto… descartado")
|
||
return # ← returns WITHOUT updating reference
|
||
device_last_pos[cell_id] = (_lat, _lon) # only runs if accepted
|
||
```
|
||
|
||
Three compounding defects:
|
||
|
||
1. **Reference not updated on reject** → once "stuck," every later fix is measured
|
||
against a stale anchor. If the device keeps moving, every packet is >500 m from
|
||
the anchor → **permanent 100 % drop**.
|
||
2. **Anchor persists across disconnects** — `device_last_pos` is **not** cleared by
|
||
the TTL cleanup (unlike every other per-device dict). A unit that moves >500 m
|
||
while powered-off and reconnects is **black-holed on return**.
|
||
3. **Flat 500 m threshold ignores time.** After any packet-loss gap (see P1
|
||
above), a normally-moving unit legitimately advances >500 m between *received*
|
||
fixes → rejected → then locked out by defect #1. **P1 packet loss directly
|
||
triggers P2 lockouts.**
|
||
|
||
Also: a device's **first** fix after (re)appearing has no `_prev`, so it's accepted
|
||
unconditionally — if that first fix is an RTK outlier, it becomes the anchor and
|
||
locks out all subsequent *good* fixes.
|
||
|
||
**Symptom match:** a unit that "died" and never came back, while its hardware and
|
||
UDP stream are fine (→ shows ONLINE in htracker, OFFLINE in Wialon).
|
||
|
||
**Minimal safe hotfix** (time-aware + self-healing + clear on disconnect):
|
||
|
||
```python
|
||
# store (lat, lon, ts); clear device_last_pos[cell_id] in the TTL cleanup too
|
||
_prev = device_last_pos.get(cell_id)
|
||
now = time.time()
|
||
if _prev:
|
||
dt = max(1.0, now - _prev[2])
|
||
max_km = (MAX_SPEED_KMH / 3600.0) * dt * 1.5 # plausibility envelope
|
||
if _dist > max(0.5, max_km):
|
||
n = device_jump_rejects.get(cell_id, 0) + 1
|
||
device_jump_rejects[cell_id] = n
|
||
if n < 3: # tolerate a few outliers…
|
||
return # …then re-anchor to self-heal
|
||
device_jump_rejects[cell_id] = 0
|
||
device_last_pos[cell_id] = (_lat, _lon, now)
|
||
# …send to Wialon…
|
||
```
|
||
|
||
This keeps odometer protection (rejects true teleports) but **cannot lock a unit
|
||
out forever**.
|
||
|
||
---
|
||
|
||
## 🔴 P2 — Identity keyed by source IP breaks under CGNAT
|
||
|
||
`ip_to_device_cell_id[source_ip]`, `ip_to_last_gga[source_ip]`,
|
||
`ip_to_last_rmc[source_ip]` assume **one device ⇄ one IP**. On cellular at fleet
|
||
scale that's false: carrier **CGNAT** puts many SIMs behind shared public IPs and
|
||
rotates them. Consequences:
|
||
|
||
- Two devices sharing a public IP **overwrite each other's** cell-id mapping and
|
||
**mix GGA/RMC** → the anti-jump filter sees a "jump" between two trucks'
|
||
positions → drops + can trigger the lockout above.
|
||
- The GGA↔RMC pairing (by IP, 2 s window) pairs the **wrong** device's sentences.
|
||
|
||
**Fix:** key everything by **`cell_id`** (the `$GPTXT` identity), not IP. Buffer
|
||
last GGA/RMC per `cell_id`; treat `source_ip` only as transport. The async version
|
||
inherits this bug too and should be fixed the same way.
|
||
|
||
---
|
||
|
||
## 🟠 Reliability / scale
|
||
|
||
- **`load_test.py` measures the wrong thing.** It only counts **UDP sends from the
|
||
simulator** ("errores UDP"), never whether the retranslator kept up or whether
|
||
Wialon received anything. A green load test gives **false confidence** — it
|
||
can't see the P1 HOL stall (the actual bottleneck is the upstream TCP path).
|
||
Add end-to-end metrics: packets forwarded to Wialon, ack latency, UDP-buffer
|
||
drops (`netstat -su`), queue depth.
|
||
- **`SQL_THROTTLE_S = 0`** while the comment/log say "every 5 min." Every packet is
|
||
enqueued to the SQL writer (bounded queue 1000). The writer is correctly a
|
||
background thread (good — not blocking the loop), but at 500 pkt/s it can
|
||
saturate/drop. Set the throttle to the intended 300 s, or document the change.
|
||
- **GGA/RMC pairing is fragile**: requires *both* within 2 s, every cycle, or
|
||
nothing is sent. Under packet loss this drops position updates even with a valid
|
||
GGA fix. Consider sending on GGA alone with last-known speed/course.
|
||
- **Thread-safety:** worker threads mutate shared dicts the main loop also mutates
|
||
(`device_last_seen`, `wialon_sockets`, `device_first_connected`…). The GIL makes
|
||
single ops atomic, but `_save_device_state` iterating a dict the main loop grows
|
||
can raise `RuntimeError: dictionary changed size during iteration` (caught, so a
|
||
stats cycle is skipped). Snapshot with `list(...)` or add a lock. The async
|
||
model (single event loop) avoids this entirely.
|
||
|
||
## 🟡 Hygiene / security
|
||
|
||
- **Secrets in the repo.** `README.md` ships a **plaintext MSSQL `sa` password**;
|
||
`.env` and `.htracker_env` are committed. Rotate that password now, purge from
|
||
git history, and move to env/secret store. (`gps_logger.py` already does the
|
||
right thing — env var, "NUNCA hardcodear" — so just align the README/compose.)
|
||
- File literally named `import socket.py` in the repo root — will shadow the stdlib
|
||
`socket` if ever imported by name; delete/rename.
|
||
- Three live retranslator variants (`retranslator.py`, `_async`, `rtk2mqtt_…`,
|
||
plus `-original`) — pick one source of truth to avoid "which one is prod?"
|
||
|
||
---
|
||
|
||
## Recommended sequence
|
||
|
||
1. **Now (hours):** confirm the live process; pull the gateway log around the
|
||
death window; check **UDP buffer drops** (`netstat -su`) and whether all units
|
||
dropped within seconds of each other (→ P1) or one-by-one (→ P2 lockouts).
|
||
Cross-check **htracker-online-vs-Wialon-offline** for ZPM 020 / RS-SAN-001.
|
||
2. **This week (small, safe):** ship the **anti-jump hotfix** above to the running
|
||
`retranslator.py` and **clear `device_last_pos` on TTL cleanup**. This stops the
|
||
permanent black-holing with minimal risk.
|
||
3. **Migrate to `retranslator_async.py`** as the real concurrency fix, but first:
|
||
re-add the corrected anti-jump filter, **switch identity to `cell_id`**, and
|
||
re-run a **realistic** load test (slow-upstream + measured forwarding).
|
||
4. **Hygiene:** rotate/purge the DB secret; converge on one retranslator.
|
||
|
||
> Backend fixes and the hardware/firmware fixes are complementary: the hardware
|
||
> can be perfect and units will still vanish from Wialon under P1/P2 — and vice
|
||
> versa. The RCA differential is updated to reflect that H2 now has **confirmed
|
||
> code-level mechanisms.**
|