# Backend Review — `htracker` retranslator + monitor

**Date:** 2026-06-19 · **Reviewer:** Claude (for Leo) · branch `feature/test500unidades`
**Scope:** `retranslator.py`, `retranslator_async.py`, `gps_logger.py`,
`web_monitor.py`, `notifier.py`, `load_test.py`, service/unit files.
**Why:** the RCA flagged a **backend common-mode** (H2) as the cheapest, most
likely explanation for *all units disconnecting at once* — and for a *single unit
"dying" and never coming back*. **This review found concrete mechanisms for both,
in software, independent of the field hardware.**

Severity: 🔴 can take units off Wialon · 🟠 reliability/scale · 🟡 hygiene.

---

## ✅ RESOLUTION (2026-06-20) — fixed & deployed to production

All three 🔴 mechanisms were fixed in `retranslator.py` and **deployed** via the new
`deploy.sh` (backup + health-check + auto-rollback; gateway/monitor/notifier all active).
Approach: **harden in place**, not the async rewrite.

| # | Issue | Fix shipped |
|---|-------|-------------|
| P1 | Synchronous Wialon I/O in the receive loop (HOL blocking) | `WialonForwarder` worker pool decouples TCP I/O from UDP ingest; the receive loop never blocks. Verified: a healthy unit kept forwarding while another unit's upstream was dark. |
| P1 | Anti-jump filter permanently locks units out | Time-aware plausibility envelope (`MAX_SPEED·Δt`) + self-heal (re-anchor after N rejects) + clear anchor on disconnect. Verified: a teleporting unit recovers instead of being black-holed. |
| P2 | Identity keyed by source IP (CGNAT) | Transport = full `(ip, port)`; all per-device state + GGA/RMC pairing keyed by `cell_id`. Verified: two SIMs on one IP don't cross-mix. |
| 🟠 | `SQL_THROTTLE_S = 0`; thread-safety | Throttle set to 300 s; stats threads snapshot shared dicts with `list(...)`. |
| 🟠 | Load test measured the wrong thing | Superseded by an end-to-end smoke test (mock Wialon + slow/dark upstream + CGNAT + jump cases). |
| 🟡 | Secrets in repo | Parameterized out of README/compose into `.env` (gitignored) + `.env.example`. **Open:** rotate the password + purge git history. |

Also shipped: the **backend-victim signal** in the monitor (ONLINE-on-UDP but not forwarding),
an **offline-unit detail panel**, a ~100× **log-volume trim** + off-box shipping to **Grafana
Loki** (via Alloy → journald, capped), and the unit-aware `deploy.sh`. Outstanding: secret
rotation + history purge; optional convergence of the dead retranslator variants; code review
(Leo to run). The original analysis is preserved below for the record.

---

## TL;DR (read this if nothing else)

1. **Production runs the *blocking* `retranslator.py`** (per `htracker.service`),
   which sends each position to Wialon and **waits for the TCP ack — and even the
   20 s connect — inside the single UDP receive loop**. One slow/unreachable
   upstream stalls the loop and **every device drops off Wialon at once**. This is
   a textbook common-mode and will get dramatically worse at 500 units. 🔴
2. **The anti-jump filter can permanently black-hole a healthy unit.** On a
   rejected "jump" it returns **without updating the reference position**, and the
   reference **survives disconnects**. A unit that moves >500 m while offline (or
   whose first fix after reconnect is an outlier) has **100 % of its packets
   dropped forever** — it transmits fine but never appears in Wialon. 🔴
3. **Device identity is keyed by source IP**, which collides under carrier CGNAT
   (500 SIMs, shared/rotating IPs) → **cross-device data mixing** → false jumps →
   disconnects. 🔴
4. There's a **correct async rewrite** (`retranslator_async.py`) that fixes the
   blocking model — but it's **not deployed**, still **keys by IP**, and **dropped
   the anti-jump filter** entirely.

**Powerful triage signal (use today):** Helgen's own monitor marks a unit
**ONLINE on UDP receipt** (`_mark_seen`), *before* the anti-jump filter and the
Wialon send. So a unit hit by #2 or #1 shows **ONLINE in htracker but OFFLINE in
Wialon**. **Any unit with that split is a backend victim, not a dead radio** —
check this against ZPM 020 / RS-SAN-001 before blaming the hardware.

---

## What is actually running (confirm this first)

- `htracker.service` → `ExecStart=/usr/bin/python3 retranslator.py` ← **the
  blocking gateway**, "Up to 500 Wialon TCP sockets," `LimitNOFILE=8192`.
- `htracker_ctl.sh` → also launches `retranslator.py`.
- `run_retranslator.sh` → launches **`rtk2mqtt_retranslator.py`** (a third
  variant). ⚠️ **Ambiguity:** confirm which process is live in prod
  (`systemctl status htracker`), because the review's #1/#2 are about
  `retranslator.py`.

---

## 🔴 P1 — Synchronous Wialon I/O in the single receive loop (HOL blocking)

`main()` is one thread doing `recvfrom` → parse → `_process_device_telemetry` →
`send_to_wialon`, which **blocks**:

- on connect: `s.connect((DEST_HOST, DEST_PORT))` with `TCP_TIMEOUT_S = 20`;
- on every packet: `current_sock.sendall(...)` then **`current_sock.recv(1024)`**
  waiting for the Wialon ack.

While that one device's socket is slow, **no UDP from any other device is read**;
the kernel UDP buffer fills and **datagrams from all 500 units are silently
dropped**. A brief upstream hiccup at `us.gpsgsm.org` therefore looks exactly like
"the whole fleet disconnected." Reconnect storms make it worse: cellular IP churn
closes/reopens sockets (each a 20 s-timeout `connect` in the loop) with
exponential backoff — all serialized in the same loop.

**Symptom match:** simultaneous death (A) + recurrent disconnects (B).

**Fix:** deploy the async model (decouple UDP ingest from Wialon I/O via a queue +
worker tasks; per-connection async with timeouts) — see `retranslator_async.py`,
which already does this. Until then, at minimum make `send_to_wialon`
non-blocking / move it off the receive loop, and bump `SO_RCVBUF`.

---

## 🔴 P1 — Anti-jump filter permanently locks units out

In `_process_device_telemetry`:

```python
_prev = device_last_pos.get(cell_id)
if _prev:
    _dist = haversine(_prev, (_lat,_lon))
    if _dist >= 0.5:                      # flat 500 m, ignores Δt
        log.warning("Anti-Salto… descartado")
        return                            # ← returns WITHOUT updating reference
device_last_pos[cell_id] = (_lat, _lon)   # only runs if accepted
```

Three compounding defects:

1. **Reference not updated on reject** → once "stuck," every later fix is measured
   against a stale anchor. If the device keeps moving, every packet is >500 m from
   the anchor → **permanent 100 % drop**.
2. **Anchor persists across disconnects** — `device_last_pos` is **not** cleared by
   the TTL cleanup (unlike every other per-device dict). A unit that moves >500 m
   while powered-off and reconnects is **black-holed on return**.
3. **Flat 500 m threshold ignores time.** After any packet-loss gap (see P1
   above), a normally-moving unit legitimately advances >500 m between *received*
   fixes → rejected → then locked out by defect #1. **P1 packet loss directly
   triggers P2 lockouts.**

Also: a device's **first** fix after (re)appearing has no `_prev`, so it's accepted
unconditionally — if that first fix is an RTK outlier, it becomes the anchor and
locks out all subsequent *good* fixes.

**Symptom match:** a unit that "died" and never came back, while its hardware and
UDP stream are fine (→ shows ONLINE in htracker, OFFLINE in Wialon).

**Minimal safe hotfix** (time-aware + self-healing + clear on disconnect):

```python
# store (lat, lon, ts); clear device_last_pos[cell_id] in the TTL cleanup too
_prev = device_last_pos.get(cell_id)
now = time.time()
if _prev:
    dt = max(1.0, now - _prev[2])
    max_km = (MAX_SPEED_KMH / 3600.0) * dt * 1.5      # plausibility envelope
    if _dist > max(0.5, max_km):
        n = device_jump_rejects.get(cell_id, 0) + 1
        device_jump_rejects[cell_id] = n
        if n < 3:                                     # tolerate a few outliers…
            return                                    # …then re-anchor to self-heal
device_jump_rejects[cell_id] = 0
device_last_pos[cell_id] = (_lat, _lon, now)
# …send to Wialon…
```

This keeps odometer protection (rejects true teleports) but **cannot lock a unit
out forever**.

---

## 🔴 P2 — Identity keyed by source IP breaks under CGNAT

`ip_to_device_cell_id[source_ip]`, `ip_to_last_gga[source_ip]`,
`ip_to_last_rmc[source_ip]` assume **one device ⇄ one IP**. On cellular at fleet
scale that's false: carrier **CGNAT** puts many SIMs behind shared public IPs and
rotates them. Consequences:

- Two devices sharing a public IP **overwrite each other's** cell-id mapping and
  **mix GGA/RMC** → the anti-jump filter sees a "jump" between two trucks'
  positions → drops + can trigger the lockout above.
- The GGA↔RMC pairing (by IP, 2 s window) pairs the **wrong** device's sentences.

**Fix:** key everything by **`cell_id`** (the `$GPTXT` identity), not IP. Buffer
last GGA/RMC per `cell_id`; treat `source_ip` only as transport. The async version
inherits this bug too and should be fixed the same way.

---

## 🟠 Reliability / scale

- **`load_test.py` measures the wrong thing.** It only counts **UDP sends from the
  simulator** ("errores UDP"), never whether the retranslator kept up or whether
  Wialon received anything. A green load test gives **false confidence** — it
  can't see the P1 HOL stall (the actual bottleneck is the upstream TCP path).
  Add end-to-end metrics: packets forwarded to Wialon, ack latency, UDP-buffer
  drops (`netstat -su`), queue depth.
- **`SQL_THROTTLE_S = 0`** while the comment/log say "every 5 min." Every packet is
  enqueued to the SQL writer (bounded queue 1000). The writer is correctly a
  background thread (good — not blocking the loop), but at 500 pkt/s it can
  saturate/drop. Set the throttle to the intended 300 s, or document the change.
- **GGA/RMC pairing is fragile**: requires *both* within 2 s, every cycle, or
  nothing is sent. Under packet loss this drops position updates even with a valid
  GGA fix. Consider sending on GGA alone with last-known speed/course.
- **Thread-safety:** worker threads mutate shared dicts the main loop also mutates
  (`device_last_seen`, `wialon_sockets`, `device_first_connected`…). The GIL makes
  single ops atomic, but `_save_device_state` iterating a dict the main loop grows
  can raise `RuntimeError: dictionary changed size during iteration` (caught, so a
  stats cycle is skipped). Snapshot with `list(...)` or add a lock. The async
  model (single event loop) avoids this entirely.

## 🟡 Hygiene / security

- **Secrets in the repo.** `README.md` ships a **plaintext MSSQL `sa` password**;
  `.env` and `.htracker_env` are committed. Rotate that password now, purge from
  git history, and move to env/secret store. (`gps_logger.py` already does the
  right thing — env var, "NUNCA hardcodear" — so just align the README/compose.)
- File literally named `import socket.py` in the repo root — will shadow the stdlib
  `socket` if ever imported by name; delete/rename.
- Three live retranslator variants (`retranslator.py`, `_async`, `rtk2mqtt_…`,
  plus `-original`) — pick one source of truth to avoid "which one is prod?"

---

## Recommended sequence

1. **Now (hours):** confirm the live process; pull the gateway log around the
   death window; check **UDP buffer drops** (`netstat -su`) and whether all units
   dropped within seconds of each other (→ P1) or one-by-one (→ P2 lockouts).
   Cross-check **htracker-online-vs-Wialon-offline** for ZPM 020 / RS-SAN-001.
2. **This week (small, safe):** ship the **anti-jump hotfix** above to the running
   `retranslator.py` and **clear `device_last_pos` on TTL cleanup**. This stops the
   permanent black-holing with minimal risk.
3. **Migrate to `retranslator_async.py`** as the real concurrency fix, but first:
   re-add the corrected anti-jump filter, **switch identity to `cell_id`**, and
   re-run a **realistic** load test (slow-upstream + measured forwarding).
4. **Hygiene:** rotate/purge the DB secret; converge on one retranslator.

> Backend fixes and the hardware/firmware fixes are complementary: the hardware
> can be perfect and units will still vanish from Wialon under P1/P2 — and vice
> versa. The RCA differential is updated to reflect that H2 now has **confirmed
> code-level mechanisms.**