13 KiB
| title | created | revised | owner | status | type | scope | source_logs | tooling | tags | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Drogheda HBOX — Bench Log + Config Analysis (20 Jun 2026) | 2026-06-20 | 2026-06-20 (rev 2 — corrected swap labels + config files) | Leo Rodriguez (Helgen) | draft-for-review | test-evidence | 7 swap-matrix bench captures + NTRIP-caster & u-blox config files, v1 firmware | Drive: 06-Testing-Validation/Logs Prueba 20JUN26 (*_GPS.ubx, *_NTRIP.log, 4G_NTRIP_HelgenSRV-NMEA.txt, configuration_GNSS.txt) | 06-Testing-Validation/log-analysis/analyze_logs.py + dashboard.html |
|
Drogheda HBOX — Bench Log + Config Analysis
One-line verdict (rev 3): Of 7 bench captures, 4 pass the comms chain and
only 1 is a confirmed fault — RTK21NTRIP21 (master 21: modem won't register,
511 AT errors). The two "no-forward" captures are indeterminate, not failed: the
NTRIP-master console can't prove forwarding either way (the healthy RTK14 baseline
logs the same sparse console). The confirmed fault reproduces cold → not heat. The
u-blox config is correct (GGA on UART2 @1 Hz); corrections come from the free public
rtk2go caster, and no unit reached RTK fix (SPS only — bench sky + corrections rarely
applied).
Rev-3 correction (post-baseline, 2026-06-20). A healthy RTK14 capture (
RTK14NTRIP14_440) — proven forwarding 1 Hz to the server by tcpdump and live in Wialon — logs the same sparse NTRIP-master console (onlyNET; noState main/UDP Bytes sent) as the captures I had called "no-forward." Absence of those console lines does not mean a unit failed. So the earlier "master-14 ingest fault" is withdrawn (master 14 demonstrably forwards — it is the baseline). OnlyRTK21NTRIP21is a confirmed fault;RTK14NTRIP14andRTK21NTRIP14are indeterminate. §§3–4 below predate this and are superseded on the no-forward point. The structural fix: make the Blackpill log fix + packets + state so health is unambiguous — see04-Source-Code/firmware/docs/LOGGING-SPEC.md.
Baseline (RTK14NTRIP14_440), 3 views: receiver
.ubx= 341 epochs, 100% valid, 12 sats, HDOP 0.75, SPS (C/N0 ~20); server tcpdump = 984 pkts, steady 1 Hz, max gap 0.86 s, zero dropouts to…:5010; master console = registered only. Healthy end-to-end but SPS, not RTK (corrections not applied). Use as the healthy-GNSS baseline; raise the bar to a fix≥4 capture once corrections work.
Rev-2 note: two captures were re-labelled by their swap pair —
RTK04NTRIP24 → RTK24NTRIP04andRTK14NTRIP21 → RTK21NTRIP14(byte-identical, md5-verified). This moved the no-forward fault from "board 14" to "master 14" (the element common to both no-forward captures). Earlier drafts said board 14.
1. Executive summary
Two log streams were captured per unit on the Querétaro bench (20 Jun 2026):
*_GPS.ubx (the ZED-F9P over USB, via u-center) and *_NTRIP.log (the 4G NTRIP
Master console). Filenames RTK<bb>NTRIP<mm> = RTK board bb + NTRIP master mm,
forming a swap matrix that isolates board vs master faults. Two config files
were also analysed: the 4G master config and the F9P config dump.
What the bench can / cannot show: it validates the integration/comms chain (boot → GNSS position → cell registration → UDP forwarding). It does not validate RTK accuracy (needs open sky + live corrections) and cannot measure heat (v1 has no temperature channel; the bench runs cold).
| Outcome | Captures | Why |
|---|---|---|
| Healthy (bench) | RTK17NTRIP17, RTK24NTRIP04, RTK24NTRIP24, RTK25NTRIP25 | boot → GNSS alive → TELCEL registered → UDP forwarding |
| Failed — no forwarding | RTK14NTRIP14, RTK21NTRIP14 | GNSS valid on USB, modem registered, but 0 UDP and no state machine — both use master 14 |
| Failed — modem | RTK21NTRIP21 | modem never registers (0xFFFFF, 511× AT-parser errors), stuck in init |
Isolated faults:
- NTRIP master 14 — GGA ingest. With two different GPS boards (14 and 21), master 14 forwarded nothing while the F9P produced valid NMEA on USB. The common element is the master → master 14's serial input / NMEA ingest, not the board. (See config note: GGA is enabled on UART2, so it is not a message-rate error.)
- NTRIP master 21 — LTE modem. Will not register on Telcel; continuous AT-parser errors. The GPS feeding it (board 21) is fine.
Neither is the GNSS receiver; neither needs heat.
2. The heat question (headline for the deck)
These logs do not prove the field failures are thermal — and that is useful. They are room-temperature bench captures with no thermal instrumentation, yet they reproduce hard faults cold (master-14 ingest, master-21 modem). A fault you can reproduce cold is not caused by heat → the fleet problem is not a single thermal cause, matching the RCA's "don't collapse everything into heat."
This neither confirms nor refutes the field thermal hypothesis (18650 / boost / modem in a sealed cabinet in Veracruz sun, RCA H1/H7) — that needs a thermocouple soak of a representative cabinet and/or v2 MCU-temp logging.
3. Per-device findings
Bench context is common: indoor/obstructed sky → C/N0 ≈ 17–21 dB-Hz, high HDOP, zero RTK fixes anywhere. Read the GNSS columns as "is the receiver alive and emitting valid NMEA," not as a positioning grade.
| Capture | Board / Master | Status | GNSS (USB) | Cell | UDP out | dBm | AT err | Headline |
|---|---|---|---|---|---|---|---|---|
| RTK24NTRIP04 | 24 / 04 | 🟢 Healthy | alive, SPS, 98% valid | TELCEL | 1555 dg | −51 | 0 | full chain OK (master 04 good) |
| RTK24NTRIP24 | 24 / 24 | 🟢 Healthy | alive, SPS, 85% valid | TELCEL | 1831 dg | −51 | 0 | full chain OK |
| RTK25NTRIP25 | 25 / 25 | 🟢 Healthy | alive, SPS, 100% valid | TELCEL | 2413 dg | −51 | 0 | best forwarder |
| RTK17NTRIP17 | 17 / 17 | 🟢 Healthy | alive, 12% valid (poor sky this run) | TELCEL | 2267 dg | −63 | 0 | comms healthiest; GNSS had no sky |
| RTK14NTRIP14 | 14 / 14 | 🔴 Failed | alive, SPS, 100% valid | TELCEL | 0 | −75 | 0 | registered but forwards nothing |
| RTK21NTRIP14 | 21 / 14 | 🔴 Failed | alive, SPS, 100% valid, 10 sats | TELCEL | 0 | −83 | 0 | same master-14 symptom, other board |
| RTK21NTRIP21 | 21 / 21 | 🔴 Failed | alive, SPS, 97% valid | NOT reg | 0 | −113 | 511 | modem never attaches |
4. Swap-matrix analysis (isolation)
By NTRIP master:
| Master | Captures | Result |
|---|---|---|
| 04, 17, 24, 25 | one each | healthy — forwards position |
| 14 | RTK14NTRIP14 (board 14) and RTK21NTRIP14 (board 21) | both fail to forward despite valid USB NMEA |
| 21 | RTK21NTRIP21 (board 21) | modem not registered, AT errors |
→ Master 14 fails with two different boards → fault is master 14's NMEA ingest (serial input), confirmed common factor. Master 21 is a modem-registration fault. Masters 04/17/24/25 are good.
By RTK board:
| Board | Captures | Read |
|---|---|---|
| 24 | NTRIP04 (✓), NTRIP24 (✓) | healthy with two masters → board 24 GNSS + UART2 output good |
| 17, 25 | one each (✓) | healthy |
| 21 | NTRIP14 (master-14 fault), NTRIP21 (master-21 fault) | GNSS alive on USB; its failures belong to the masters it was paired with |
| 14 | NTRIP14 only (master-14 fault) | GNSS alive on USB; board-14 UART2 output unproven — only ever paired with the bad master 14 |
Open item: test board 14 (and board 21) with a known-good master to clear/convict their UART2 output; test master 14 with a known-good board to confirm the ingest fault.
5. Configuration findings (from the two config files)
5.1 4G NTRIP Master (4G_NTRIP_HelgenSRV-NMEA.txt)
| Item | Value | Note |
|---|---|---|
| NTRIP caster | rtk2go.com : 2101, mountpoint HelgenTestStation | free public community caster — best-effort, no SLA |
| APN | internet.itelcel.com (Telcel) | correct |
| UART baud (to F9P) | 115200 | matches F9P UART2 baud |
| NTRIP restart timer | 300 s | matches "NTRIP DELAY n/300" in the logs |
| Telemetry out | UDP → Helgen server :5010, device H0014 | position forwarding path |
| PointPerfect | configured but disabled | not in use |
Flag: corrections from a free public caster (rtk2go) is a reliability risk for a production fleet and a plausible reason the correction client rarely connected and no unit reached RTK. Move production to a dedicated/own caster (HelgenSRV).
5.2 ZED-F9P (configuration_GNSS.txt, UBX CFG-VALGET dump)
| Item | Value | Note |
|---|---|---|
| Model / FW | ZED-F9P, HPG 1.32 | rover-grade |
| Mode | Rover (TMODE=0) | correct (not a base) |
| Update rate | 1 Hz (1000 ms) | correct |
| GGA on UART2 | ENABLED (1/epoch) | the feed the master needs — correctly configured |
| RMC on UART2 | ENABLED (1/epoch) | not required by NTRIP, harmless |
| UART2 | enabled · NMEA-out + RTCM3-in · 115200 | textbook 4G-NTRIP rover wiring |
| USB | full NMEA (GGA/GLL/GSA/GSV/RMC/VTG) | matches the .ubx captures |
| Dump layer | RAM only | flash persistence NOT verified |
Two implications:
- The message config is correct — GGA is on UART2 @1 Hz. So master 14's dead feed is not a missing-GGA setting → it's the master's ingest (or a per-unit deviation), reinforcing §4.
- The dump is RAM-layer only. A RAM-only config is lost on every power cycle (the classic "forgot Save Config to flash" trap). Confirm the FLASH layer on each unit — a unit that boots without GGA-on-UART2 would feed the master nothing.
6. Recommendations / next steps
- Master 14 — bench the ingest. Feed master 14 a known-good GGA stream (or pair
with a healthy board) and watch for
State main+UDP Bytes sent. If still zero, the master's serial-in / firmware is the fault → swap/repair the master. - Master 21 — modem triage. Re-test registration with a known-good SIM + antenna;
511 AT-parser errors +
0xFFFFFpoint at the LTE modem / its AT link / power. Swap the modem to confirm. - Persist & verify the F9P config to FLASH on every unit (u-center → Save Config), then dump the flash layer to prove GGA-on-UART2 survives a power cycle. Check the simpleRTK2B Pro XBEE↔UART2 switch too (UART2 is off by default).
- Move corrections off rtk2go to a dedicated caster (HelgenSRV) with uptime, then re-run outdoors with open sky to actually grade RTK (quality 4/5).
- Close the heat question separately — thermocouple soak of a sealed cabinet in full sun and/or v2 MCU-temp logging.
- Retranslator log (separate effort) — pair this unit-side analysis with the backend (HOL-stall / anti-jump) evidence for the full RCA picture.
7. Method & reproducibility
- Tool:
log-analysis/analyze_logs.py(Python stdlib). Parses NMEA (GGA/RMC/GSV) + UBX from.ubx, and the GPTXT diagnostic stream (State main,NET,UDP Bytes sent,NTRIP DELAY,SM_REG, AT errors) from.log. Emits per-device metrics, a swap-matrix with common-factor isolation, and per-device time-series. - Dashboard:
log-analysis/dashboard.html— dark (GitHub Dark-Dimmed), self-contained: config summary, signal-flow diagram (marks the break + names the common master), zoomable time-series, status board, swap matrix. - Config decode: 4G master
$$$AS+params; F9P UBX CFG-VALGET keys (CFG-MSGOUT-NMEA_ID_GGA_UART2, CFG-UART2*, CFG-RATE*, CFG-TMODE-MODE). - Re-run:
python3 analyze_logs.py --in raw_cur --out out && python3 build_dashboard.py - Validation: classification spot-checked vs raw logs; the two relabels were md5-verified byte-identical to the originals.
Caveats: ~10-min captures; indoor bench sky; v1 (no temperature); F9P config dump
is RAM-layer only; ntr/tcp/pp enum values inferred from behavior. None affect the
pass/fail conclusions, which rest on presence/absence of UDP forwarding, cellular
registration, and the swap-matrix common factor.
8. Glossary
- GGA fix quality: 0 no-fix · 1 SPS · 2 DGPS · 4 RTK-fixed · 5 RTK-float.
- C/N0 (dB-Hz): satellite carrier-to-noise; ~35–50 open sky, ~20 indoors.
- NTRIP master state
main,ntr,tcp,pp,ntr_srv,udp: main=4 operational / 1 init; udp=1 forwarding position out; ntr = correction-client state (0 idle … 2 active). NET,<op>,<dBm>,<MCCMNC>: TELCEL = registered;0xFFFFF= not registered.- NTRIP DELAY n/300: seconds since last correction vs the 300 s restart watchdog.
- GGA→UART2: whether the F9P's GGA is reaching the 4G master over UART2 (XBee socket). Inferred from the master running its loop + forwarding UDP.