RCA — HBOX RTK Field Failures (Veracruz / ICAVE)

Status: DRAFT — open hypotheses, awaiting field logs Date: 2026-06-19 · Owner: Leo Rodriguez (Helgen) Scope: 5 HBOX units at Hutchison Ports ICAVE (RTG crane, forklift, electric truck, 2 reachstackers). "Working-ish," then a near-simultaneous death; chronic recurrent disconnects in Wialon. Client escalation 2026-06-17.

⚠️ This RCA is hypothesis-ranked, not concluded. The single most useful next input is the field log set (battery voltage, MCU temp, fix quality, reconnect timestamps) plus the backend/SIM/caster logs. Update §6 when they arrive.

1. The two failures are probably different failures

Do not collapse these into one cause:

(A) Sudden, near-simultaneous death of the fleet. Multiple independent units failing at ~the same time points to a common-mode stressor, not five coincidental hardware faults. Candidates: a heat spell, a backend/caster/SIM outage, or a server/retranslator problem.
(B) Chronic "recurrent disconnections" in Wialon over time. This is the drip-drip the client has been logging for weeks. Candidates: vibration-induced intermittents (Dupont/SMA/solder), 4G NTRIP card hangs, and a watchdog that recovers slowly and noisily.

Treating (A) as "heat" and stopping there risks missing a cheap backend fix — and treating (B) as "heat" risks shipping a thermal redesign that doesn't stop the vibration intermittents. Both need addressing; the permanent fix does.

2. Differential diagnosis (check cheap/common before exotic)

#	Hypothesis	Explains A?	Explains B?	How to confirm (fast)	Prior
H1	Thermal (component-level). A sealed ASA HBOX sits inside a sealed ABS IP67 cabinet that also holds the Mean Well supply (internal heat source), in full sun → cooks the 18650 / modem / boost. The ASA shell itself is fine (UV-stable, softens ~95–100 °C).	✓✓	✓	Inspect pulled units for swollen 18650, discoloured boost; thermocouple inside a representative cabinet over a sunny shift	High
H2	Backend common-mode — CONFIRMED mechanism. The production `retranslator.py` sends to Wialon and waits for the TCP ack (and 20 s connect) inside the single UDP loop → one slow upstream stalls the loop and all units drop at once. Plus caster/SIM-plan exhaustion.	✓✓	✓	Gateway log + `netstat -su` UDP drops in the death window; did all units drop within seconds (→ this) or one-by-one (→ H2b)? See `02-Architecture-Software/backend-review.md`	High (and cheapest)
H2b	Anti-jump filter lock-out — CONFIRMED mechanism. On a rejected "jump" the filter never updates its reference and the reference survives reconnects → a healthy unit's packets are dropped forever; it transmits but never reaches Wialon.	✗	✓✓	The unit shows ONLINE in htracker but OFFLINE in Wialon (monitor marks online on UDP receipt, before the filter). Check this per unit.	High
H3	Vibration intermittents — Dupont/SMA/terminal back-out, cracked solder	✗	✓✓	Pulled-unit inspection: wiggle test, continuity under vibration, visual on joints/SMA torque	High
H4	4G NTRIP card hang/failure (client's #1 point)	~	✓✓	Pulled-unit bench: does modem re-enumerate after power cycle? thermal-soak test	High
H5	Under-rated fuse (client's one remaining power suspicion)	✓ (if blown by inrush/load)	~	Check fuse rating vs measured worst-case + inrush current; look for blown/derated fuses	Medium
H6	Watchdog logic turning brief RTK loss into long outages / reboot loops	✗	✓✓	Code review (done — §4) + correlate Wialon gaps with ~5/10-min cadence	High for B
H7	LiPo brownout — hot/aged cell sags under load, watchdog cuts power at 3.4 V	✓	✓	Battery-voltage log; capacity test on pulled cells	Medium-High
H8	Power-rail/EMI on machinery — alternator/inverter transients, ground bounce on a crane	~	✓	Scope the 12–24 V input under operation; check for resets correlated with motion	Medium

The two cheapest checks (H2 backend, H5 fuse) should be done first — they can be confirmed or killed in an hour without pulling a single unit, and H2 in particular could explain the simultaneous death entirely.

Backend code review is done (02-Architecture-Software/backend-review.md) and it materially raised the priors on H2/H2b. Two software defects can take healthy units off Wialon: (1) synchronous Wialon send-and-ack in the single receive loop → head-of-line stall → simultaneous fleet dropout; (2) an anti-jump filter that can permanently black-hole one unit. The single best triage step is now free: compare Helgen's htracker monitor (online on UDP receipt) against the client's Wialon (online on positions received). Units that are online in one and offline in the other are backend victims, not dead hardware — check ZPM 020 / RS-SAN-001 this way before the teardown.

3. Leading physical hypothesis: thermal (H1/H7)

Why heat is the prime suspect for the sudden death — and the new component data makes it worse than first assumed:

It's a nested thermal trap, not one box. The HBOX is a 3D-printed ASA shell sealed inside a second sealed enclosure — the ABS IP67 cabinet — which also contains the Mean Well 24 V supply (60 W class, several watts of continuous self-heat). So the HBOX bakes in an enclosure that has both solar gain and an internal heat source, with no airflow (IP67). See 03-Architecture-Hardware/cabinet-electrical-bom.md.
The ASA enclosure is adequate — not the liability. ASA is UV-stable and softens at ~95–100 °C, with margin over the cabinet's interior temperature, so the shell will not sag/warp at these temperatures. (This corrects an earlier assumption that the inner shell was PLA; both the ABS outer cabinet, Tg ~105 °C, and the inner ASA are fine — the thermal risk is to the components.)
Environment: Veracruz in June — coastal, hot, humid, full sun. Sealed enclosure interiors commonly run 20–35 °C above ambient; with an internal supply added, 60–80 °C is realistic.
The weak components sit below the GNSS:
- LiPo — safe charge only 0–45 °C; above ~45 °C you get lithium plating, swelling, capacity loss, and (worst case) thermal runaway. Optimal is 20–25 °C. A hot LiPo also sags under load → the watchdog reads <3.4 V → cuts power → unit goes dark ("Solid OFF / hibernate"). This is a clean mechanism for simultaneous death during a hot spell.
- PowerBoost 1000C — a hobby-grade boost/charger; efficiency and thermal headroom fall as it heats; possible thermal shutdown.
- 4G NTRIP Master modem — LTE modems self-heat and thermally throttle/drop; matches the client's "recurrent 4G NTRIP card failures."
The GNSS itself is not the weak link: the ZED-F9P is rated −40…+85 °C. So "the RTK died of heat" most likely means the battery, charger, or modem around it died — not the receiver.

Implication for the fix: you cannot firmware your way out of heat. The permanent fix is thermal-mechanical: the ASA inner box stays (it's adequate; optionally move to metal for extra heatsinking, or mount the PCB straight to the cabinet plate), remove/relocate the battery, replace the boost with an efficient 24 V→5 V buck, and conduct heat out to a metal plate/cabinet — not a bare internal fan, which does nothing in a sealed box (see hardware brief §2b). Firmware's job is to measure temperature (v2 logs MCU max temp) so the pulled units prove or kill this hypothesis.

4. Firmware's contribution to the chronic disconnects (H6)

Full bug list in 03-Architecture-Hardware/firmware-review.md. The two that most plausibly create Wialon "desconexiones recurrentes":

The 5-minute soft-reset is aimed at the wrong subsystem. v1 reacts to lost corrections by sending a GNSS hot-reset (UBX-CFG-RST). That restarts the ZED-F9P only — it cannot revive a hung 4G/NTRIP modem, which is the client's #1 reported failure. Five minutes are wasted on a no-op.
The 10-minute hard reset has no backoff → reboot-loop flapping. If the 4G card is genuinely stuck (or the unit is parked where it can't see sky), v1 power-cycles the whole stack every ~10 minutes, forever. Each cycle is a multi-minute Wialon disconnect. A bank of units doing this looks exactly like "recurrent disconnections."

Both are fixed in v2: recover by power-cycling the 4G stack (the thing that actually restarts the modem), detect NMEA silence in ~60 s, apply exponential backoff, and after repeated failures drop into LIMP mode that keeps reporting standalone GPS instead of vanishing. A unit with a dead modem now still shows on Wialon (degraded) rather than disconnecting.

Note: v1 does not mistreat RTK float (quality 5) as a failure — it correctly treats fixed(4) and float(5) as healthy. The chronic-disconnect driver is the recovery strategy, not the fix-quality threshold.

5. The client's hardware indictment is correct (H3/H4)

ICAVE's review is, frankly, a fair description of a lab prototype shipped to a brutal industrial environment. Each point maps to a real failure mechanism:

Client point	Mechanism	Permanent fix
Dupont connectors	vibration backs them out → intermittent open	locking connectors (JST-XH/Molex), no jumpers
Solder-joint stress	flex + heat-cycling cracks joints	PCB with strain relief; no dead-bug wiring
No vibration design	resonance loosens everything	PCB + standoffs + grommets + conformal coat
Loose terminals/antennas	SMA + screw terminals work loose	torqued bulkhead SMA, retained connectors
4G NTRIP card failures	thermal + mechanical + no recovery	thermal fix + v2 recovery + qualified modem
Hard SIM swaps	SIM buried in the stack	front-panel SIM access / eSIM
"Not apt for production"	hand-wired one-offs	designed PCB, documented, repeatable

None of this is solved by firmware. It's the core of the PCB + ruggedization work in 03-Architecture-Hardware/hardware-respec-and-pcb-brief.md.

6. Evidence still needed (fill this in)

Constraint (2026-06-19): all units are pulled and sitting in Veracruz; team is in Querétaro. Hands-on items below are gated on shipping to QRO or remote-hands — see 06-Testing-Validation/remote-diagnostic-protocol.md. These units are v1 (no MCU thermal log), so thermal evidence = physical inspection + bench power test, not firmware logs.

Historical logs (do from QRO now): battery V, fix quality, reset timestamps; and the htracker-online vs Wialon-offline split per unit
Backend: anti-jump warnings spamming a cell_id + netstat -su UDP drops around the death window (02-Architecture-Software/backend-review.md)
Caster: HelgenSRV NTRIP availability log
Carrier: per-SIM data balance / suspension status (rules H2 in/out fast)
Fuse: installed rating vs measured worst-case + inrush (H5)
4 pulled units (3 dead + 1 working-ish control): bench current-draw on power-up, LED + USB-dashboard capture, photos of LiPo (swelling?), boost (discolour?), solder, SMA. Genuinely-dead units → teardown in QRO (modem power-cycle test, LiPo capacity, thermal-soak)
Thermal quantification: thermocouple in a sealed box in full sun (can be done with a spare enclosure in VER or QRO) to get a real interior-temp number

7. Provisional conclusion

The evidence now points to at least two independent causes, not one:

Backend (H2/H2b) — confirmed in code. The blocking retranslator can drop the whole fleet at once on any upstream hiccup, and the anti-jump filter can permanently black-hole an individual unit. Some — possibly much — of the "deaths" and "recurrent disconnects" may be server-side, with healthy radios in the field. This is testable for free (htracker-online vs Wialon-offline) and must be done before/alongside the teardown.
Hardware/thermal (H1/H7/H3/H4). A heat event on the LiPo/modem/boost, on top of a vibration-fragile prototype, remains the leading physical cause for units that are genuinely dead — confirmed once units are pulled (swelling, discolour, cracked joints) and from the v2 temperature log.
Firmware (H6) amplifies real blips into long outages — fixed in v2.

The permanent fix is deliberately robust to all of them: backend hardening (async + cell_id identity + fixed anti-jump) + thermal-mechanical redesign + ruggedized interconnect (PCB) + smarter firmware (v2) + verified fusing. Do not spend the whole budget on a thermal redesign before confirming how much of the problem is the server.

13 KiB Raw Blame History Unescape Escape