Backend Review — `htracker` retranslator + monitor

Date: 2026-06-19 · Reviewer: Claude (for Leo) · branch feature/test500unidades Scope: retranslator.py, retranslator_async.py, gps_logger.py, web_monitor.py, notifier.py, load_test.py, service/unit files. Why: the RCA flagged a backend common-mode (H2) as the cheapest, most likely explanation for all units disconnecting at once — and for a single unit "dying" and never coming back. This review found concrete mechanisms for both, in software, independent of the field hardware.

Severity: 🔴 can take units off Wialon · 🟠 reliability/scale · 🟡 hygiene.

✅ RESOLUTION (2026-06-20) — fixed & deployed to production

All three 🔴 mechanisms were fixed in retranslator.py and deployed via the new deploy.sh (backup + health-check + auto-rollback; gateway/monitor/notifier all active). Approach: harden in place, not the async rewrite.

#	Issue	Fix shipped
P1	Synchronous Wialon I/O in the receive loop (HOL blocking)	`WialonForwarder` worker pool decouples TCP I/O from UDP ingest; the receive loop never blocks. Verified: a healthy unit kept forwarding while another unit's upstream was dark.
P1	Anti-jump filter permanently locks units out	Time-aware plausibility envelope (`MAX_SPEED·Δt`) + self-heal (re-anchor after N rejects) + clear anchor on disconnect. Verified: a teleporting unit recovers instead of being black-holed.
P2	Identity keyed by source IP (CGNAT)	Transport = full `(ip, port)`; all per-device state + GGA/RMC pairing keyed by `cell_id`. Verified: two SIMs on one IP don't cross-mix.
🟠	`SQL_THROTTLE_S = 0`; thread-safety	Throttle set to 300 s; stats threads snapshot shared dicts with `list(...)`.
🟠	Load test measured the wrong thing	Superseded by an end-to-end smoke test (mock Wialon + slow/dark upstream + CGNAT + jump cases).
🟡	Secrets in repo	Parameterized out of README/compose into `.env` (gitignored) + `.env.example`. Open: rotate the password + purge git history.

Also shipped: the backend-victim signal in the monitor (ONLINE-on-UDP but not forwarding), an offline-unit detail panel, a ~100× log-volume trim + off-box shipping to Grafana Loki (via Alloy → journald, capped), and the unit-aware deploy.sh. Outstanding: secret rotation + history purge; optional convergence of the dead retranslator variants; code review (Leo to run). The original analysis is preserved below for the record.

TL;DR (read this if nothing else)

Production runs the blocking retranslator.py (per htracker.service), which sends each position to Wialon and waits for the TCP ack — and even the 20 s connect — inside the single UDP receive loop. One slow/unreachable upstream stalls the loop and every device drops off Wialon at once. This is a textbook common-mode and will get dramatically worse at 500 units. 🔴
The anti-jump filter can permanently black-hole a healthy unit. On a rejected "jump" it returns without updating the reference position, and the reference survives disconnects. A unit that moves >500 m while offline (or whose first fix after reconnect is an outlier) has 100 % of its packets dropped forever — it transmits fine but never appears in Wialon. 🔴
Device identity is keyed by source IP, which collides under carrier CGNAT (500 SIMs, shared/rotating IPs) → cross-device data mixing → false jumps → disconnects. 🔴
There's a correct async rewrite (retranslator_async.py) that fixes the blocking model — but it's not deployed, still keys by IP, and dropped the anti-jump filter entirely.

Powerful triage signal (use today): Helgen's own monitor marks a unit ONLINE on UDP receipt (_mark_seen), before the anti-jump filter and the Wialon send. So a unit hit by #2 or #1 shows ONLINE in htracker but OFFLINE in Wialon. Any unit with that split is a backend victim, not a dead radio — check this against ZPM 020 / RS-SAN-001 before blaming the hardware.

What is actually running (confirm this first)

htracker.service → ExecStart=/usr/bin/python3 retranslator.py ← the blocking gateway, "Up to 500 Wialon TCP sockets," LimitNOFILE=8192.
htracker_ctl.sh → also launches retranslator.py.
run_retranslator.sh → launches rtk2mqtt_retranslator.py (a third variant). ⚠️ Ambiguity: confirm which process is live in prod (systemctl status htracker), because the review's #1/#2 are about retranslator.py.

🔴 P1 — Synchronous Wialon I/O in the single receive loop (HOL blocking)

main() is one thread doing recvfrom → parse → _process_device_telemetry → send_to_wialon, which blocks:

on connect: s.connect((DEST_HOST, DEST_PORT)) with TCP_TIMEOUT_S = 20;
on every packet: current_sock.sendall(...) then current_sock.recv(1024) waiting for the Wialon ack.

While that one device's socket is slow, no UDP from any other device is read; the kernel UDP buffer fills and datagrams from all 500 units are silently dropped. A brief upstream hiccup at us.gpsgsm.org therefore looks exactly like "the whole fleet disconnected." Reconnect storms make it worse: cellular IP churn closes/reopens sockets (each a 20 s-timeout connect in the loop) with exponential backoff — all serialized in the same loop.

Symptom match: simultaneous death (A) + recurrent disconnects (B).

Fix: deploy the async model (decouple UDP ingest from Wialon I/O via a queue + worker tasks; per-connection async with timeouts) — see retranslator_async.py, which already does this. Until then, at minimum make send_to_wialon non-blocking / move it off the receive loop, and bump SO_RCVBUF.

🔴 P1 — Anti-jump filter permanently locks units out

In _process_device_telemetry:

_prev = device_last_pos.get(cell_id)
if _prev:
    _dist = haversine(_prev, (_lat,_lon))
    if _dist >= 0.5:                      # flat 500 m, ignores Δt
        log.warning("Anti-Salto… descartado")
        return                            # ← returns WITHOUT updating reference
device_last_pos[cell_id] = (_lat, _lon)   # only runs if accepted

Three compounding defects:

Reference not updated on reject → once "stuck," every later fix is measured against a stale anchor. If the device keeps moving, every packet is >500 m from the anchor → permanent 100 % drop.
Anchor persists across disconnects — device_last_pos is not cleared by the TTL cleanup (unlike every other per-device dict). A unit that moves >500 m while powered-off and reconnects is black-holed on return.
Flat 500 m threshold ignores time. After any packet-loss gap (see P1 above), a normally-moving unit legitimately advances >500 m between received fixes → rejected → then locked out by defect #1. P1 packet loss directly triggers P2 lockouts.

Also: a device's first fix after (re)appearing has no _prev, so it's accepted unconditionally — if that first fix is an RTK outlier, it becomes the anchor and locks out all subsequent good fixes.

Symptom match: a unit that "died" and never came back, while its hardware and UDP stream are fine (→ shows ONLINE in htracker, OFFLINE in Wialon).

Minimal safe hotfix (time-aware + self-healing + clear on disconnect):

# store (lat, lon, ts); clear device_last_pos[cell_id] in the TTL cleanup too
_prev = device_last_pos.get(cell_id)
now = time.time()
if _prev:
    dt = max(1.0, now - _prev[2])
    max_km = (MAX_SPEED_KMH / 3600.0) * dt * 1.5      # plausibility envelope
    if _dist > max(0.5, max_km):
        n = device_jump_rejects.get(cell_id, 0) + 1
        device_jump_rejects[cell_id] = n
        if n < 3:                                     # tolerate a few outliers…
            return                                    # …then re-anchor to self-heal
device_jump_rejects[cell_id] = 0
device_last_pos[cell_id] = (_lat, _lon, now)
# …send to Wialon…

This keeps odometer protection (rejects true teleports) but cannot lock a unit out forever.

🔴 P2 — Identity keyed by source IP breaks under CGNAT

ip_to_device_cell_id[source_ip], ip_to_last_gga[source_ip], ip_to_last_rmc[source_ip] assume one device ⇄ one IP. On cellular at fleet scale that's false: carrier CGNAT puts many SIMs behind shared public IPs and rotates them. Consequences:

Two devices sharing a public IP overwrite each other's cell-id mapping and mix GGA/RMC → the anti-jump filter sees a "jump" between two trucks' positions → drops + can trigger the lockout above.
The GGA↔RMC pairing (by IP, 2 s window) pairs the wrong device's sentences.

Fix: key everything by cell_id (the $GPTXT identity), not IP. Buffer last GGA/RMC per cell_id; treat source_ip only as transport. The async version inherits this bug too and should be fixed the same way.

🟠 Reliability / scale

load_test.py measures the wrong thing. It only counts UDP sends from the simulator ("errores UDP"), never whether the retranslator kept up or whether Wialon received anything. A green load test gives false confidence — it can't see the P1 HOL stall (the actual bottleneck is the upstream TCP path). Add end-to-end metrics: packets forwarded to Wialon, ack latency, UDP-buffer drops (netstat -su), queue depth.
SQL_THROTTLE_S = 0 while the comment/log say "every 5 min." Every packet is enqueued to the SQL writer (bounded queue 1000). The writer is correctly a background thread (good — not blocking the loop), but at 500 pkt/s it can saturate/drop. Set the throttle to the intended 300 s, or document the change.
GGA/RMC pairing is fragile: requires both within 2 s, every cycle, or nothing is sent. Under packet loss this drops position updates even with a valid GGA fix. Consider sending on GGA alone with last-known speed/course.
Thread-safety: worker threads mutate shared dicts the main loop also mutates (device_last_seen, wialon_sockets, device_first_connected…). The GIL makes single ops atomic, but _save_device_state iterating a dict the main loop grows can raise RuntimeError: dictionary changed size during iteration (caught, so a stats cycle is skipped). Snapshot with list(...) or add a lock. The async model (single event loop) avoids this entirely.

🟡 Hygiene / security

Secrets in the repo. README.md ships a plaintext MSSQL sa password; .env and .htracker_env are committed. Rotate that password now, purge from git history, and move to env/secret store. (gps_logger.py already does the right thing — env var, "NUNCA hardcodear" — so just align the README/compose.)
File literally named import socket.py in the repo root — will shadow the stdlib socket if ever imported by name; delete/rename.
Three live retranslator variants (retranslator.py, _async, rtk2mqtt_…, plus -original) — pick one source of truth to avoid "which one is prod?"

Recommended sequence

Now (hours): confirm the live process; pull the gateway log around the death window; check UDP buffer drops (netstat -su) and whether all units dropped within seconds of each other (→ P1) or one-by-one (→ P2 lockouts). Cross-check htracker-online-vs-Wialon-offline for ZPM 020 / RS-SAN-001.
This week (small, safe): ship the anti-jump hotfix above to the running retranslator.py and clear device_last_pos on TTL cleanup. This stops the permanent black-holing with minimal risk.
Migrate to retranslator_async.py as the real concurrency fix, but first: re-add the corrected anti-jump filter, switch identity to cell_id, and re-run a realistic load test (slow-upstream + measured forwarding).
Hygiene: rotate/purge the DB secret; converge on one retranslator.

Backend fixes and the hardware/firmware fixes are complementary: the hardware can be perfect and units will still vanish from Wialon under P1/P2 — and vice versa. The RCA differential is updated to reflect that H2 now has confirmed code-level mechanisms.

12 KiB Raw Blame History Unescape Escape

Backend Review — htracker retranslator + monitor