239 lines
13 KiB
Markdown
239 lines
13 KiB
Markdown
---
|
||
tags: [project-drogheda, hardware, firmware, watchdog, telemetry, nmea]
|
||
project: Project Drogheda
|
||
author: Helgen CTO
|
||
date: 2026-06-20
|
||
status: Analysis — review & bench-test before any wiring/firmware change
|
||
---
|
||
|
||
# Watchdog Health Telemetry — Feasibility & Improvement Analysis
|
||
|
||
> **The question:** can the STM32F411 watchdog transmit its health (temp, reset
|
||
> cause, link state, recovery count…) over UART so it **lands in our server
|
||
> message**, e.g. as a custom `$GPTXT`? You've seen mixed information. This
|
||
> resolves it against the **as-built wiring** and proposes a clean design.
|
||
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
- **As wired today: No.** The watchdog sits on the F9P's **UART1** (it reads NMEA
|
||
and sends UBX rate commands). The data that reaches the server flows out the
|
||
F9P's **UART2 → 4G NTRIP Master → server**. The watchdog is **not on that path**,
|
||
so nothing it prints can reach the backend.
|
||
- The `$GPTXT,NET,<id>,TELCEL,<rssi>,<cell>` you see at the server is the **4G
|
||
modem's** own network-status line (the `<id>` is the modem's station ID set via
|
||
`AS+K`, which the backend treats as the `cell_id`). It is **not** produced by the
|
||
F9P or the watchdog.
|
||
- **It is achievable** — but it requires putting the watchdog **in the modem's
|
||
NMEA-uplink path** (a relay/injector) and emitting a **proprietary sentence**
|
||
(`$PHLG…`, *not* `$GPTXT` — see the collision note). This is a **wiring change**,
|
||
clean on the Phase-2 PCB, and it must be **bench-validated** against one unknown:
|
||
*does the ArduSimple 4G NTRIP Master forward arbitrary/proprietary NMEA, or only a
|
||
whitelist?* That single question is the source of the "mixed information."
|
||
- **Lower-risk interim (no rewiring):** the watchdog already has a full health
|
||
record — ship the **v2.1 parseable log** (`docs/LOGGING-SPEC.md`) over its console.
|
||
It's authoritative per-unit health, captured on pulled units; it just isn't *live
|
||
to the server*. In-band-to-server telemetry is the Phase-2 upgrade.
|
||
|
||
---
|
||
|
||
## 1. The as-built data path (why the answer is "not yet")
|
||
|
||
```text
|
||
┌──────────────── simpleRTK2B Pro (ZED-F9P) ─────────────────┐
|
||
│ │
|
||
UART1 ◄──┤ TX1 → NMEA (GGA…) ──► STM32 PA3 (RX) [watchdog: READS] │
|
||
│ RX1 ← UBX rate ◄── STM32 PA2 (TX) [watchdog: WRITES]│
|
||
│ │
|
||
UART2 ◄──┤ TX2 → NMEA/GGA @1Hz ──► 4G NTRIP Master (XBee) ──► SERVER │ ← server path
|
||
│ RX2 ← RTCM3 ◄── 4G NTRIP Master (NTRIP caster) │
|
||
└────────────────────────────────────────────────────────────┘
|
||
│
|
||
4G modem also emits its OWN $GPTXT,NET,<AS+K id>,TELCEL,…
|
||
(network status + station id = the backend's cell_id)
|
||
```
|
||
|
||
Key facts (confirmed in firmware + the 20-Jun bench analysis):
|
||
|
||
- Firmware: `HardwareSerial GNSS(PA3, PA2)` → **F9P UART1**. `parseNMEA()` only
|
||
*reads* GGA (for `fixQuality`/`satellites`/`rxCount`); `applyRate()` only *writes*
|
||
UBX. On power-off it **floats** PA2/PA3. The watchdog never transmits toward the
|
||
modem.
|
||
- The modem is on the Pro's **XBee socket = UART2** (the "XBEE↔UART2 switch"); GGA is
|
||
configured out **UART2 @ 1 Hz**. The server stream and the NTRIP GGA-uplink both
|
||
ride UART2.
|
||
- Therefore the watchdog (UART1) and the server path (UART2) are **electrically
|
||
separate**. No firmware change alone can bridge them.
|
||
|
||
---
|
||
|
||
## 2. The three theoretical ways in (and which one to use)
|
||
|
||
1. **Make the F9P emit it.** ✗ Not possible. The ZED-F9P has no command to inject an
|
||
arbitrary NMEA string into its UART2 output. It emits only its own sentences + UBX
|
||
replies.
|
||
2. **Tell the 4G modem to carry it.** ✗ No runtime path. The modem's `$GPTXT` is
|
||
network status; `AS+K` (station id) is *set-once* config, not a data channel the
|
||
MCU can stuff health into per-cycle.
|
||
3. **Put the watchdog in the uplink NMEA path (relay/injector).** ✓ **The viable
|
||
design.** Route the F9P→modem NMEA *through* the STM32:
|
||
|
||
```text
|
||
F9P TX2 (NMEA) ──► STM32 RX ──[relay verbatim]──► STM32 TX ──► modem RX
|
||
└─[inject $PHLG health @ ~1/30s in the idle gap]
|
||
modem ──► forwards EVERYTHING (GGA/RMC/GPTXT + $PHLG) ──► server
|
||
RTCM from the modem can still go straight to the F9P RX2 (corrections path intact).
|
||
```
|
||
|
||
The backend already maps **(ip,port) → cell_id** (from the modem's `$GPTXT`), so a
|
||
`$PHLG` arriving on the same socket is **auto-associated to the right unit** — no
|
||
need to put the id in the health sentence. (This reuses the CGNAT-safe identity work
|
||
shipped in `retranslator.py`.)
|
||
|
||
> **Retrofit variant (no full relay):** diode-OR / open-drain the STM32 TX onto the
|
||
> modem's RX and inject only during the F9P's idle gaps (at 115200 baud, 1 Hz NMEA
|
||
> leaves the line idle >90 % of the time). Cheaper to bolt on, but collision-prone and
|
||
> harder to guarantee — **prefer the clean relay on the PCB.**
|
||
|
||
---
|
||
|
||
## 3. The proprietary sentence (don't reuse `$GPTXT`)
|
||
|
||
**Collision warning:** the backend parses `$GPTXT` by taking **field 2 as the
|
||
`cell_id`**. A health `$GPTXT` would be mis-read as a device id. Use a **proprietary
|
||
talker** instead — `$PHLG` (`P` = proprietary, `HLG` = Helgen):
|
||
|
||
```text
|
||
$PHLG,<fw>,<state>,<link>,<fix>,<sat>,<rx>,<vbat>,<v5v>,<tc>,<tmax>,<rec>,<limp>,<nmea_age_ms>,<reset>*CS
|
||
e.g. $PHLG,2.2,RUN,DEGR,1,12,34192,3.96,5.01,58,69,0,0,180,POR*11
|
||
(checksum is the standard NMEA XOR over the chars between $ and *, computed at
|
||
runtime by buildPHLG(); -127 in tc/tmax = "n/a". The earlier *4F here was a wrong
|
||
hand-written placeholder — the correct XOR for that body is *49.)
|
||
```
|
||
|
||
Fields mirror the v2.1 `TLM` record (state, link, fix, sats, rx, battery, 5 V,
|
||
**MCU temp + session max**, recovery count, limp latch, NMEA age, **reset cause**).
|
||
`CS` = standard NMEA XOR checksum.
|
||
|
||
**Backend side** is a ~15-line addition to the hardened `retranslator.py`: a branch
|
||
that recognises `$PHLG`, verifies the checksum, and stores the fields per `cell_id`
|
||
(keyed by the existing `(ip,port)` map). Then the monitor can surface, **per unit and
|
||
live**: MCU temp / `tMax` (the thermal-hypothesis evidence, without pulling the unit),
|
||
reset cause (reboot-loop forensics), recovery count, and LIMP state — and it ships to
|
||
Loki for free.
|
||
|
||
### Backend `$PHLG` branch (concrete — review before deploy)
|
||
|
||
Drop-in for `retranslator.py`. `cell_id` is already resolved from `(ip,port)` by the
|
||
existing CGNAT-safe map; this only adds recognition + checksum + store. Tolerant of the
|
||
**extra trailing fields** the v3 board will append (air/RH/battery/fan).
|
||
|
||
```python
|
||
# --- $PHLG health branch (v2.2) — REVIEW BEFORE DEPLOY ------------------------
|
||
def _nmea_ok(s: str) -> bool:
|
||
if not s.startswith('$') or '*' not in s:
|
||
return False
|
||
body, _, cs = s[1:].partition('*')
|
||
x = 0
|
||
for ch in body:
|
||
x ^= ord(ch)
|
||
return f"{x:02X}" == cs.strip()[:2].upper()
|
||
|
||
PHLG_FIELDS = ("fw","state","link","fix","sat","rx","vbat","v5v",
|
||
"tc","tmax","rec","limp","nmea_age_ms","reset")
|
||
|
||
def handle_phlg(line: str, cell_id: str, store: dict) -> bool:
|
||
if not line.startswith('$PHLG,') or not _nmea_ok(line):
|
||
return False
|
||
body = line[1:].split('*', 1)[0]
|
||
parts = body.split(',')[1:] # drop the 'PHLG' talker
|
||
rec = dict(zip(PHLG_FIELDS, parts)) # extra v3 trailing fields are simply ignored
|
||
rec["ts"] = time.time()
|
||
store[cell_id] = rec # latest health per unit (also ship to Loki)
|
||
log.info("PHLG %s temp=%s tmax=%s reset=%s limp=%s rec=%s",
|
||
cell_id, rec.get("tc"), rec.get("tmax"), rec.get("reset"),
|
||
rec.get("limp"), rec.get("rec"))
|
||
return True
|
||
|
||
# In the inbound loop, BEFORE the GGA/position path:
|
||
# if handle_phlg(line, cell_id, PHLG_BY_CELL):
|
||
# continue # health record, NOT a position fix — don't forward to Wialon
|
||
# -----------------------------------------------------------------------------
|
||
```
|
||
|
||
---
|
||
|
||
## 4. What to bench-test before committing (resolves the "mixed info")
|
||
|
||
1. **Does the 4G NTRIP Master forward a proprietary `$P…` sentence?**
|
||
✅ **RESOLVED 2026-06-22 — YES.** Enabling `$PUBX` (a u-blox proprietary `$P…` talker)
|
||
on the F9P UART2 → it arrives at the server `udp/5010` **intact**, alongside GGA/RMC. The
|
||
modem is a transparent forwarder; **no whitelist**, so the `$GPTXT,HLT` fallback is not
|
||
needed. (Detail: `03-Architecture-Hardware/watchdog-pcb-v3-design.md` §7a.) Original note:
|
||
inject one on UART2-RX on the bench and watch the server `tcpdump`. *If it whitelists talkers:*
|
||
fall back to a `$GPTXT,HLT,…` subtype and special-case it in the backend by **field
|
||
1** (`HLT` vs `NET`) so it never collides with the cell-id parse. **This is the one
|
||
experiment that settles the whole question.**
|
||
2. **GGA integrity for NTRIP.** The modem relays the F9P's GGA *up to the caster* for
|
||
network RTK. The relay must be **byte-exact and low-latency** and only inject in
|
||
idle gaps — never delay or mangle GGA, or corrections degrade.
|
||
3. **Rate / cost.** Health at **1 / 30–60 s** keeps cellular + backend load trivial.
|
||
4. **Checksum** must be correct or the backend (and possibly the modem) drops it.
|
||
|
||
---
|
||
|
||
## 5. Other watchdog improvements (independent of the telemetry question)
|
||
|
||
### 🔴 5a. The "no-RTK ≠ dead-modem" thrash risk — *act on this now*
|
||
v2 treats `fixQuality >= 4` (RTK) as the only healthy link, and after
|
||
`NTRIP_GRACE` (120 s) of `< 4` it **power-cycles the stack**. But the 20-Jun bench
|
||
reality is **every unit is SPS (quality 1), no RTK** (caster/FLASH/sky issue, *not* a
|
||
modem hang). With the watchdog connected as-is, a unit that is **GNSS-healthy and
|
||
forwarding fine at SPS** would be **power-cycled every ~2 min + backoff** — manufacturing
|
||
the very disconnects we're trying to kill. **Fix:** only power-cycle on evidence of a
|
||
*link/modem* hang (NMEA silence is already handled); treat *stable SPS with incrementing
|
||
`rx`* as a reportable **DEGRADED** state, **not** a recovery trigger — at least until the
|
||
corrections pipeline (move off rtk2go, save F9P config to FLASH, open-sky retest) is
|
||
fixed. This is the highest-value firmware change right now.
|
||
|
||
### 🟠 5b. Ship the v2.1 parseable log (`docs/LOGGING-SPEC.md`)
|
||
The ANSI dashboard overwrites itself and has no timestamps/reset-cause → useless as a
|
||
field record. The v2.1 spec (BOOT/COLS/TLM/EVT, reset cause, ~60 additive lines) makes
|
||
the watchdog the **authoritative per-unit health record**. It's also the exact data the
|
||
`$PHLG` sentence would carry — **do this first**, so every later change is observable.
|
||
|
||
### 🟠 5c. Independent watchdog (IWDG) — watch the watcher
|
||
Nothing currently recovers the Blackpill itself if *it* hangs. Enabling the STM32 IWDG
|
||
makes an MCU lockup self-reset and show `reset=IWDG` next boot — closing the last
|
||
"silent vanish" gap for a device whose whole job is fail-safe.
|
||
|
||
### 🟠 5d. External temp sensor + persisted counters
|
||
v2 logs **MCU** temp, but the heat casualties are the **18650 / 4G modem**, not the die.
|
||
Add a small external sensor near the cell/modem (PCB brief already flags this), and
|
||
**persist** `tMax` / `recoveryCount` / reset history to RTC backup registers or flash so
|
||
a pulled unit's forensics survive the power-cycle. Directly serves the thermal RCA.
|
||
|
||
---
|
||
|
||
## 6. Recommendation (phased)
|
||
|
||
| When | Action | Risk |
|
||
|---|---|---|
|
||
| **Now** (firmware only) | 5a (stop SPS-driven power-cycling) + 5b (v2.1 log) + 5c (IWDG). Bench-test, bump v2.1, then flash. | Low — additive / behavioural guard. |
|
||
| **Bench experiment** | Test #1: does the modem forward `$P…`? Decide `$PHLG` vs `$GPTXT,HLT`. | None — measurement. |
|
||
| **Phase 2 (PCB)** | Watchdog as NMEA relay/injector on the UART2 uplink + external temp sensor; backend `$PHLG` parser + monitor/Loki surfacing. | Medium — relay must preserve GGA/NTRIP; validate before fleet. |
|
||
|
||
**Bottom line for the deck:** *Yes, we can put unit health (incl. temperature) into the
|
||
live server feed — it needs the watchdog inserted into the modem's uplink path (a
|
||
Phase-2 PCB change) and one bench test of the modem's forwarding. Until then the same
|
||
health is captured on-unit via the v2.1 log. Separately, before we connect the watchdog
|
||
we must stop it from power-cycling healthy SPS units.*
|
||
|
||
## Related
|
||
- [[03-Architecture-Hardware/watchdog-schematic|watchdog-schematic]] ·
|
||
[[03-Architecture-Hardware/firmware-review|firmware-review]] ·
|
||
[[03-Architecture-Hardware/hardware-respec-and-pcb-brief|PCB brief]] ·
|
||
`04-Source-Code/firmware/docs/LOGGING-SPEC.md` ·
|
||
`04-Source-Code/firmware/blackpillrtkcode_v2/blackpillrtkcode_v2.ino`
|