# RCA — HBOX RTK Field Failures (Veracruz / ICAVE) **Status:** DRAFT — open hypotheses, awaiting field logs **Date:** 2026-06-19 · **Owner:** Leo Rodriguez (Helgen) **Scope:** 5 HBOX units at Hutchison Ports ICAVE (RTG crane, forklift, electric truck, 2 reachstackers). "Working-ish," then a near-simultaneous death; chronic recurrent disconnects in Wialon. Client escalation 2026-06-17. > ⚠️ This RCA is **hypothesis-ranked**, not concluded. The single most useful > next input is the field log set (battery voltage, MCU temp, fix quality, > reconnect timestamps) plus the backend/SIM/caster logs. Update §6 when they > arrive. --- ## 1. The two failures are probably *different* failures Do not collapse these into one cause: - **(A) Sudden, near-simultaneous death of the fleet.** Multiple independent units failing at ~the same time points to a **common-mode** stressor, not five coincidental hardware faults. Candidates: a heat spell, a backend/caster/SIM outage, or a server/retranslator problem. - **(B) Chronic "recurrent disconnections" in Wialon over time.** This is the drip-drip the client has been logging for weeks. Candidates: vibration-induced intermittents (Dupont/SMA/solder), 4G NTRIP card hangs, and a watchdog that recovers slowly and noisily. Treating (A) as "heat" and stopping there risks missing a cheap backend fix — and treating (B) as "heat" risks shipping a thermal redesign that doesn't stop the vibration intermittents. Both need addressing; the permanent fix does. --- ## 2. Differential diagnosis (check cheap/common before exotic) | # | Hypothesis | Explains A? | Explains B? | How to confirm (fast) | Prior | |---|---|:--:|:--:|---|:--:| | H1 | **Thermal (component-level).** A sealed **ASA** HBOX sits **inside a sealed ABS IP67 cabinet that also holds the Mean Well supply** (internal heat source), in full sun → cooks the **18650 / modem / boost**. The **ASA shell itself is fine** (UV-stable, softens ~95–100 °C). | ✓✓ | ✓ | Inspect pulled units for swollen **18650**, discoloured boost; thermocouple inside a representative cabinet over a sunny shift | **High** | | H2 | **Backend common-mode — CONFIRMED mechanism.** The production `retranslator.py` sends to Wialon and waits for the TCP ack (and 20 s connect) **inside the single UDP loop** → one slow upstream stalls the loop and **all units drop at once**. Plus caster/SIM-plan exhaustion. | ✓✓ | ✓ | Gateway log + `netstat -su` UDP drops in the death window; did all units drop within seconds (→ this) or one-by-one (→ H2b)? See `02-Architecture-Software/backend-review.md` | **High (and cheapest)** | | H2b | **Anti-jump filter lock-out — CONFIRMED mechanism.** On a rejected "jump" the filter never updates its reference and the reference survives reconnects → a healthy unit's packets are dropped **forever**; it transmits but never reaches Wialon. | ✗ | ✓✓ | The unit shows **ONLINE in htracker but OFFLINE in Wialon** (monitor marks online on UDP receipt, before the filter). Check this per unit. | **High** | | H3 | **Vibration intermittents** — Dupont/SMA/terminal back-out, cracked solder | ✗ | ✓✓ | Pulled-unit inspection: wiggle test, continuity under vibration, visual on joints/SMA torque | **High** | | H4 | **4G NTRIP card hang/failure** (client's #1 point) | ~ | ✓✓ | Pulled-unit bench: does modem re-enumerate after power cycle? thermal-soak test | **High** | | H5 | **Under-rated fuse** (client's one remaining power suspicion) | ✓ (if blown by inrush/load) | ~ | Check fuse rating vs measured worst-case + inrush current; look for blown/derated fuses | **Medium** | | H6 | **Watchdog logic** turning brief RTK loss into long outages / reboot loops | ✗ | ✓✓ | Code review (done — §4) + correlate Wialon gaps with ~5/10-min cadence | **High for B** | | H7 | **LiPo brownout** — hot/aged cell sags under load, watchdog cuts power at 3.4 V | ✓ | ✓ | Battery-voltage log; capacity test on pulled cells | **Medium-High** | | H8 | **Power-rail/EMI on machinery** — alternator/inverter transients, ground bounce on a crane | ~ | ✓ | Scope the 12–24 V input under operation; check for resets correlated with motion | **Medium** | The two **cheapest** checks (H2 backend, H5 fuse) should be done **first** — they can be confirmed or killed in an hour without pulling a single unit, and H2 in particular could explain the simultaneous death entirely. > **Backend code review is done (`02-Architecture-Software/backend-review.md`) and it materially > raised the priors on H2/H2b.** Two software defects can take healthy units off > Wialon: (1) synchronous Wialon send-and-ack in the single receive loop → > head-of-line stall → *simultaneous* fleet dropout; (2) an anti-jump filter that > can *permanently* black-hole one unit. The single best triage step is now free: > **compare Helgen's htracker monitor (online on UDP receipt) against the client's > Wialon (online on positions received).** Units that are online in one and > offline in the other are **backend victims, not dead hardware** — check > ZPM 020 / RS-SAN-001 this way before the teardown. --- ## 3. Leading physical hypothesis: thermal (H1/H7) Why heat is the prime suspect for the sudden death — and **the new component data makes it worse than first assumed**: - **It's a nested thermal trap, not one box.** The HBOX is a **3D-printed ASA** shell sealed **inside** a second sealed enclosure — the **ABS IP67 cabinet** — which **also contains the Mean Well 24 V supply** (60 W class, several watts of continuous self-heat). So the HBOX bakes in an enclosure that has **both** solar gain **and** an internal heat source, with **no airflow** (IP67). See `03-Architecture-Hardware/cabinet-electrical-bom.md`. - **The ASA enclosure is adequate — not the liability.** ASA is UV-stable and **softens at ~95–100 °C**, with margin over the cabinet's interior temperature, so the shell will **not sag/warp** at these temperatures. (This corrects an earlier assumption that the inner shell was PLA; both the ABS outer cabinet, Tg ~105 °C, and the inner **ASA** are fine — the thermal risk is to the **components**.) - **Environment:** Veracruz in June — coastal, hot, humid, full sun. Sealed enclosure interiors commonly run **20–35 °C above ambient**; with an internal supply added, **60–80 °C is realistic**. - **The weak components sit below the GNSS:** - **LiPo** — safe **charge only 0–45 °C**; above ~45 °C you get lithium plating, swelling, capacity loss, and (worst case) thermal runaway. Optimal is 20–25 °C. A hot LiPo also **sags under load** → the watchdog reads <3.4 V → cuts power → unit goes dark ("Solid OFF / hibernate"). This is a clean mechanism for *simultaneous* death during a hot spell. - **PowerBoost 1000C** — a hobby-grade boost/charger; efficiency and thermal headroom fall as it heats; possible thermal shutdown. - **4G NTRIP Master modem** — LTE modems self-heat and thermally throttle/drop; matches the client's "recurrent 4G NTRIP card failures." - **The GNSS itself is *not* the weak link:** the **ZED-F9P is rated −40…+85 °C**. So "the RTK died of heat" most likely means the **battery, charger, or modem** around it died — not the receiver. **Implication for the fix:** you cannot firmware your way out of heat. The permanent fix is thermal-mechanical: the **ASA inner box stays** (it's adequate; optionally move to metal for extra heatsinking, or mount the PCB straight to the cabinet plate), **remove/relocate the battery**, replace the boost with an efficient 24 V→5 V buck, and **conduct heat out to a metal plate/cabinet** — **not** a bare internal fan, which does nothing in a sealed box (see hardware brief §2b). Firmware's job is to **measure** temperature (v2 logs MCU max temp) so the pulled units prove or kill this hypothesis. --- ## 4. Firmware's contribution to the *chronic* disconnects (H6) Full bug list in `03-Architecture-Hardware/firmware-review.md`. The two that most plausibly create Wialon "desconexiones recurrentes": 1. **The 5-minute soft-reset is aimed at the wrong subsystem.** v1 reacts to lost corrections by sending a **GNSS hot-reset** (`UBX-CFG-RST`). That restarts the ZED-F9P only — it **cannot** revive a hung 4G/NTRIP modem, which is the client's #1 reported failure. Five minutes are wasted on a no-op. 2. **The 10-minute hard reset has no backoff → reboot-loop flapping.** If the 4G card is genuinely stuck (or the unit is parked where it can't see sky), v1 power-cycles the whole stack **every ~10 minutes, forever**. Each cycle is a multi-minute Wialon disconnect. A bank of units doing this looks exactly like "recurrent disconnections." Both are fixed in **v2**: recover by power-cycling the **4G stack** (the thing that actually restarts the modem), detect NMEA silence in ~60 s, apply **exponential backoff**, and after repeated failures drop into **LIMP mode that keeps reporting standalone GPS** instead of vanishing. A unit with a dead modem now still shows on Wialon (degraded) rather than disconnecting. > Note: v1 does **not** mistreat RTK **float** (quality 5) as a failure — it > correctly treats fixed(4) and float(5) as healthy. The chronic-disconnect > driver is the recovery *strategy*, not the fix-quality threshold. --- ## 5. The client's hardware indictment is correct (H3/H4) ICAVE's review is, frankly, a fair description of a lab prototype shipped to a brutal industrial environment. Each point maps to a real failure mechanism: | Client point | Mechanism | Permanent fix | |---|---|---| | Dupont connectors | vibration backs them out → intermittent open | locking connectors (JST-XH/Molex), no jumpers | | Solder-joint stress | flex + heat-cycling cracks joints | PCB with strain relief; no dead-bug wiring | | No vibration design | resonance loosens everything | PCB + standoffs + grommets + conformal coat | | Loose terminals/antennas | SMA + screw terminals work loose | torqued bulkhead SMA, retained connectors | | 4G NTRIP card failures | thermal + mechanical + no recovery | thermal fix + v2 recovery + qualified modem | | Hard SIM swaps | SIM buried in the stack | front-panel SIM access / eSIM | | "Not apt for production" | hand-wired one-offs | designed PCB, documented, repeatable | None of this is solved by firmware. It's the core of the **PCB + ruggedization** work in `03-Architecture-Hardware/hardware-respec-and-pcb-brief.md`. --- ## 6. Evidence still needed (fill this in) > **Constraint (2026-06-19):** all units are pulled and sitting in **Veracruz**; > team is in **Querétaro**. Hands-on items below are gated on **shipping to QRO** > or **remote-hands** — see `06-Testing-Validation/remote-diagnostic-protocol.md`. These units are > **v1 (no MCU thermal log)**, so thermal evidence = **physical inspection + bench > power test**, not firmware logs. - [ ] **Historical logs** (do from QRO now): battery V, fix quality, reset timestamps; and the **htracker-online vs Wialon-offline** split per unit - [ ] **Backend**: anti-jump warnings spamming a `cell_id` + `netstat -su` UDP drops around the death window (`02-Architecture-Software/backend-review.md`) - [ ] **Caster**: HelgenSRV NTRIP availability log - [ ] **Carrier**: per-SIM data balance / suspension status (rules H2 in/out fast) - [ ] **Fuse**: installed rating vs measured worst-case + inrush (H5) - [ ] **4 pulled units (3 dead + 1 working-ish control)**: bench **current-draw** on power-up, LED + USB-dashboard capture, photos of LiPo (swelling?), boost (discolour?), solder, SMA. Genuinely-dead units → **teardown in QRO** (modem power-cycle test, LiPo capacity, thermal-soak) - [ ] **Thermal quantification**: thermocouple in a sealed box in full sun (can be done with a spare enclosure in VER or QRO) to get a real interior-temp number --- ## 7. Provisional conclusion The evidence now points to **at least two independent causes, not one**: 1. **Backend (H2/H2b) — confirmed in code.** The blocking retranslator can drop the *whole fleet* at once on any upstream hiccup, and the anti-jump filter can *permanently* black-hole an individual unit. Some — possibly much — of the "deaths" and "recurrent disconnects" may be **server-side, with healthy radios in the field.** This is testable for free (htracker-online vs Wialon-offline) and must be done before/alongside the teardown. 2. **Hardware/thermal (H1/H7/H3/H4).** A heat event on the LiPo/modem/boost, on top of a vibration-fragile prototype, remains the leading *physical* cause for units that are genuinely dead — confirmed once units are pulled (swelling, discolour, cracked joints) and from the v2 temperature log. 3. **Firmware (H6)** amplifies real blips into long outages — fixed in v2. The permanent fix is deliberately robust to all of them: **backend hardening (async + cell_id identity + fixed anti-jump)** + thermal-mechanical redesign + ruggedized interconnect (PCB) + smarter firmware (v2) + verified fusing. Do not spend the whole budget on a thermal redesign before confirming how much of the problem is the server.