201 lines
13 KiB
Markdown
201 lines
13 KiB
Markdown
# RCA — HBOX RTK Field Failures (Veracruz / ICAVE)
|
||
|
||
**Status:** DRAFT — open hypotheses, awaiting field logs
|
||
**Date:** 2026-06-19 · **Owner:** Leo Rodriguez (Helgen)
|
||
**Scope:** 5 HBOX units at Hutchison Ports ICAVE (RTG crane, forklift, electric
|
||
truck, 2 reachstackers). "Working-ish," then a near-simultaneous death; chronic
|
||
recurrent disconnects in Wialon. Client escalation 2026-06-17.
|
||
|
||
> ⚠️ This RCA is **hypothesis-ranked**, not concluded. The single most useful
|
||
> next input is the field log set (battery voltage, MCU temp, fix quality,
|
||
> reconnect timestamps) plus the backend/SIM/caster logs. Update §6 when they
|
||
> arrive.
|
||
|
||
---
|
||
|
||
## 1. The two failures are probably *different* failures
|
||
|
||
Do not collapse these into one cause:
|
||
|
||
- **(A) Sudden, near-simultaneous death of the fleet.** Multiple independent
|
||
units failing at ~the same time points to a **common-mode** stressor, not five
|
||
coincidental hardware faults. Candidates: a heat spell, a backend/caster/SIM
|
||
outage, or a server/retranslator problem.
|
||
- **(B) Chronic "recurrent disconnections" in Wialon over time.** This is the
|
||
drip-drip the client has been logging for weeks. Candidates: vibration-induced
|
||
intermittents (Dupont/SMA/solder), 4G NTRIP card hangs, and a watchdog that
|
||
recovers slowly and noisily.
|
||
|
||
Treating (A) as "heat" and stopping there risks missing a cheap backend fix —
|
||
and treating (B) as "heat" risks shipping a thermal redesign that doesn't stop
|
||
the vibration intermittents. Both need addressing; the permanent fix does.
|
||
|
||
---
|
||
|
||
## 2. Differential diagnosis (check cheap/common before exotic)
|
||
|
||
| # | Hypothesis | Explains A? | Explains B? | How to confirm (fast) | Prior |
|
||
|---|---|:--:|:--:|---|:--:|
|
||
| H1 | **Thermal (component-level).** A sealed **ASA** HBOX sits **inside a sealed ABS IP67 cabinet that also holds the Mean Well supply** (internal heat source), in full sun → cooks the **18650 / modem / boost**. The **ASA shell itself is fine** (UV-stable, softens ~95–100 °C). | ✓✓ | ✓ | Inspect pulled units for swollen **18650**, discoloured boost; thermocouple inside a representative cabinet over a sunny shift | **High** |
|
||
| H2 | **Backend common-mode — CONFIRMED mechanism.** The production `retranslator.py` sends to Wialon and waits for the TCP ack (and 20 s connect) **inside the single UDP loop** → one slow upstream stalls the loop and **all units drop at once**. Plus caster/SIM-plan exhaustion. | ✓✓ | ✓ | Gateway log + `netstat -su` UDP drops in the death window; did all units drop within seconds (→ this) or one-by-one (→ H2b)? See `02-Architecture-Software/backend-review.md` | **High (and cheapest)** |
|
||
| H2b | **Anti-jump filter lock-out — CONFIRMED mechanism.** On a rejected "jump" the filter never updates its reference and the reference survives reconnects → a healthy unit's packets are dropped **forever**; it transmits but never reaches Wialon. | ✗ | ✓✓ | The unit shows **ONLINE in htracker but OFFLINE in Wialon** (monitor marks online on UDP receipt, before the filter). Check this per unit. | **High** |
|
||
| H3 | **Vibration intermittents** — Dupont/SMA/terminal back-out, cracked solder | ✗ | ✓✓ | Pulled-unit inspection: wiggle test, continuity under vibration, visual on joints/SMA torque | **High** |
|
||
| H4 | **4G NTRIP card hang/failure** (client's #1 point) | ~ | ✓✓ | Pulled-unit bench: does modem re-enumerate after power cycle? thermal-soak test | **High** |
|
||
| H5 | **Under-rated fuse** (client's one remaining power suspicion) | ✓ (if blown by inrush/load) | ~ | Check fuse rating vs measured worst-case + inrush current; look for blown/derated fuses | **Medium** |
|
||
| H6 | **Watchdog logic** turning brief RTK loss into long outages / reboot loops | ✗ | ✓✓ | Code review (done — §4) + correlate Wialon gaps with ~5/10-min cadence | **High for B** |
|
||
| H7 | **LiPo brownout** — hot/aged cell sags under load, watchdog cuts power at 3.4 V | ✓ | ✓ | Battery-voltage log; capacity test on pulled cells | **Medium-High** |
|
||
| H8 | **Power-rail/EMI on machinery** — alternator/inverter transients, ground bounce on a crane | ~ | ✓ | Scope the 12–24 V input under operation; check for resets correlated with motion | **Medium** |
|
||
|
||
The two **cheapest** checks (H2 backend, H5 fuse) should be done **first** — they
|
||
can be confirmed or killed in an hour without pulling a single unit, and H2 in
|
||
particular could explain the simultaneous death entirely.
|
||
|
||
> **Backend code review is done (`02-Architecture-Software/backend-review.md`) and it materially
|
||
> raised the priors on H2/H2b.** Two software defects can take healthy units off
|
||
> Wialon: (1) synchronous Wialon send-and-ack in the single receive loop →
|
||
> head-of-line stall → *simultaneous* fleet dropout; (2) an anti-jump filter that
|
||
> can *permanently* black-hole one unit. The single best triage step is now free:
|
||
> **compare Helgen's htracker monitor (online on UDP receipt) against the client's
|
||
> Wialon (online on positions received).** Units that are online in one and
|
||
> offline in the other are **backend victims, not dead hardware** — check
|
||
> ZPM 020 / RS-SAN-001 this way before the teardown.
|
||
|
||
---
|
||
|
||
## 3. Leading physical hypothesis: thermal (H1/H7)
|
||
|
||
Why heat is the prime suspect for the sudden death — and **the new component data
|
||
makes it worse than first assumed**:
|
||
|
||
- **It's a nested thermal trap, not one box.** The HBOX is a **3D-printed ASA**
|
||
shell sealed **inside** a second sealed enclosure — the **ABS IP67 cabinet** —
|
||
which **also contains the Mean Well 24 V supply** (60 W class, several watts of
|
||
continuous self-heat). So the HBOX bakes in an enclosure that has **both** solar
|
||
gain **and** an internal heat source, with **no airflow** (IP67). See
|
||
`03-Architecture-Hardware/cabinet-electrical-bom.md`.
|
||
- **The ASA enclosure is adequate — not the liability.** ASA is UV-stable and
|
||
**softens at ~95–100 °C**, with margin over the cabinet's interior temperature, so
|
||
the shell will **not sag/warp** at these temperatures. (This corrects an earlier
|
||
assumption that the inner shell was PLA; both the ABS outer cabinet, Tg ~105 °C,
|
||
and the inner **ASA** are fine — the thermal risk is to the **components**.)
|
||
- **Environment:** Veracruz in June — coastal, hot, humid, full sun. Sealed
|
||
enclosure interiors commonly run **20–35 °C above ambient**; with an internal
|
||
supply added, **60–80 °C is realistic**.
|
||
- **The weak components sit below the GNSS:**
|
||
- **LiPo** — safe **charge only 0–45 °C**; above ~45 °C you get lithium
|
||
plating, swelling, capacity loss, and (worst case) thermal runaway. Optimal
|
||
is 20–25 °C. A hot LiPo also **sags under load** → the watchdog reads
|
||
<3.4 V → cuts power → unit goes dark ("Solid OFF / hibernate"). This is a
|
||
clean mechanism for *simultaneous* death during a hot spell.
|
||
- **PowerBoost 1000C** — a hobby-grade boost/charger; efficiency and thermal
|
||
headroom fall as it heats; possible thermal shutdown.
|
||
- **4G NTRIP Master modem** — LTE modems self-heat and thermally throttle/drop;
|
||
matches the client's "recurrent 4G NTRIP card failures."
|
||
- **The GNSS itself is *not* the weak link:** the **ZED-F9P is rated −40…+85 °C**.
|
||
So "the RTK died of heat" most likely means the **battery, charger, or modem**
|
||
around it died — not the receiver.
|
||
|
||
**Implication for the fix:** you cannot firmware your way out of heat. The
|
||
permanent fix is thermal-mechanical: the **ASA inner box stays** (it's adequate;
|
||
optionally move to metal for extra heatsinking, or mount the PCB straight to the
|
||
cabinet plate), **remove/relocate the battery**,
|
||
replace the boost with an efficient 24 V→5 V buck, and **conduct heat out to a
|
||
metal plate/cabinet** — **not** a bare internal fan, which does nothing in a sealed box
|
||
(see hardware brief §2b). Firmware's job is to **measure** temperature (v2 logs MCU
|
||
max temp) so the pulled units prove or kill this hypothesis.
|
||
|
||
---
|
||
|
||
## 4. Firmware's contribution to the *chronic* disconnects (H6)
|
||
|
||
Full bug list in `03-Architecture-Hardware/firmware-review.md`. The two that most plausibly create
|
||
Wialon "desconexiones recurrentes":
|
||
|
||
1. **The 5-minute soft-reset is aimed at the wrong subsystem.** v1 reacts to lost
|
||
corrections by sending a **GNSS hot-reset** (`UBX-CFG-RST`). That restarts the
|
||
ZED-F9P only — it **cannot** revive a hung 4G/NTRIP modem, which is the
|
||
client's #1 reported failure. Five minutes are wasted on a no-op.
|
||
2. **The 10-minute hard reset has no backoff → reboot-loop flapping.** If the 4G
|
||
card is genuinely stuck (or the unit is parked where it can't see sky), v1
|
||
power-cycles the whole stack **every ~10 minutes, forever**. Each cycle is a
|
||
multi-minute Wialon disconnect. A bank of units doing this looks exactly like
|
||
"recurrent disconnections."
|
||
|
||
Both are fixed in **v2**: recover by power-cycling the **4G stack** (the thing
|
||
that actually restarts the modem), detect NMEA silence in ~60 s, apply
|
||
**exponential backoff**, and after repeated failures drop into **LIMP mode that
|
||
keeps reporting standalone GPS** instead of vanishing. A unit with a dead modem
|
||
now still shows on Wialon (degraded) rather than disconnecting.
|
||
|
||
> Note: v1 does **not** mistreat RTK **float** (quality 5) as a failure — it
|
||
> correctly treats fixed(4) and float(5) as healthy. The chronic-disconnect
|
||
> driver is the recovery *strategy*, not the fix-quality threshold.
|
||
|
||
---
|
||
|
||
## 5. The client's hardware indictment is correct (H3/H4)
|
||
|
||
ICAVE's review is, frankly, a fair description of a lab prototype shipped to a
|
||
brutal industrial environment. Each point maps to a real failure mechanism:
|
||
|
||
| Client point | Mechanism | Permanent fix |
|
||
|---|---|---|
|
||
| Dupont connectors | vibration backs them out → intermittent open | locking connectors (JST-XH/Molex), no jumpers |
|
||
| Solder-joint stress | flex + heat-cycling cracks joints | PCB with strain relief; no dead-bug wiring |
|
||
| No vibration design | resonance loosens everything | PCB + standoffs + grommets + conformal coat |
|
||
| Loose terminals/antennas | SMA + screw terminals work loose | torqued bulkhead SMA, retained connectors |
|
||
| 4G NTRIP card failures | thermal + mechanical + no recovery | thermal fix + v2 recovery + qualified modem |
|
||
| Hard SIM swaps | SIM buried in the stack | front-panel SIM access / eSIM |
|
||
| "Not apt for production" | hand-wired one-offs | designed PCB, documented, repeatable |
|
||
|
||
None of this is solved by firmware. It's the core of the **PCB + ruggedization**
|
||
work in `03-Architecture-Hardware/hardware-respec-and-pcb-brief.md`.
|
||
|
||
---
|
||
|
||
## 6. Evidence still needed (fill this in)
|
||
|
||
> **Constraint (2026-06-19):** all units are pulled and sitting in **Veracruz**;
|
||
> team is in **Querétaro**. Hands-on items below are gated on **shipping to QRO**
|
||
> or **remote-hands** — see `06-Testing-Validation/remote-diagnostic-protocol.md`. These units are
|
||
> **v1 (no MCU thermal log)**, so thermal evidence = **physical inspection + bench
|
||
> power test**, not firmware logs.
|
||
|
||
- [ ] **Historical logs** (do from QRO now): battery V, fix quality, reset
|
||
timestamps; and the **htracker-online vs Wialon-offline** split per unit
|
||
- [ ] **Backend**: anti-jump warnings spamming a `cell_id` + `netstat -su` UDP
|
||
drops around the death window (`02-Architecture-Software/backend-review.md`)
|
||
- [ ] **Caster**: HelgenSRV NTRIP availability log
|
||
- [ ] **Carrier**: per-SIM data balance / suspension status (rules H2 in/out fast)
|
||
- [ ] **Fuse**: installed rating vs measured worst-case + inrush (H5)
|
||
- [ ] **4 pulled units (3 dead + 1 working-ish control)**: bench **current-draw**
|
||
on power-up, LED + USB-dashboard capture, photos of LiPo (swelling?), boost
|
||
(discolour?), solder, SMA. Genuinely-dead units → **teardown in QRO**
|
||
(modem power-cycle test, LiPo capacity, thermal-soak)
|
||
- [ ] **Thermal quantification**: thermocouple in a sealed box in full sun (can be
|
||
done with a spare enclosure in VER or QRO) to get a real interior-temp number
|
||
|
||
---
|
||
|
||
## 7. Provisional conclusion
|
||
|
||
The evidence now points to **at least two independent causes, not one**:
|
||
|
||
1. **Backend (H2/H2b) — confirmed in code.** The blocking retranslator can drop
|
||
the *whole fleet* at once on any upstream hiccup, and the anti-jump filter can
|
||
*permanently* black-hole an individual unit. Some — possibly much — of the
|
||
"deaths" and "recurrent disconnects" may be **server-side, with healthy radios
|
||
in the field.** This is testable for free (htracker-online vs Wialon-offline)
|
||
and must be done before/alongside the teardown.
|
||
2. **Hardware/thermal (H1/H7/H3/H4).** A heat event on the LiPo/modem/boost, on
|
||
top of a vibration-fragile prototype, remains the leading *physical* cause for
|
||
units that are genuinely dead — confirmed once units are pulled (swelling,
|
||
discolour, cracked joints) and from the v2 temperature log.
|
||
3. **Firmware (H6)** amplifies real blips into long outages — fixed in v2.
|
||
|
||
The permanent fix is deliberately robust to all of them: **backend hardening
|
||
(async + cell_id identity + fixed anti-jump)** + thermal-mechanical redesign +
|
||
ruggedized interconnect (PCB) + smarter firmware (v2) + verified fusing. Do not
|
||
spend the whole budget on a thermal redesign before confirming how much of the
|
||
problem is the server.
|