Watchdog Health Telemetry — Feasibility & Improvement Analysis

The question: can the STM32F411 watchdog transmit its health (temp, reset cause, link state, recovery count…) over UART so it lands in our server message, e.g. as a custom $GPTXT? You've seen mixed information. This resolves it against the as-built wiring and proposes a clean design.

TL;DR

As wired today: No. The watchdog sits on the F9P's UART1 (it reads NMEA and sends UBX rate commands). The data that reaches the server flows out the F9P's UART2 → 4G NTRIP Master → server. The watchdog is not on that path, so nothing it prints can reach the backend.
The $GPTXT,NET,<id>,TELCEL,<rssi>,<cell> you see at the server is the 4G modem's own network-status line (the <id> is the modem's station ID set via AS+K, which the backend treats as the cell_id). It is not produced by the F9P or the watchdog.
It is achievable — but it requires putting the watchdog in the modem's NMEA-uplink path (a relay/injector) and emitting a proprietary sentence ($PHLG…, not $GPTXT — see the collision note). This is a wiring change, clean on the Phase-2 PCB, and it must be bench-validated against one unknown: does the ArduSimple 4G NTRIP Master forward arbitrary/proprietary NMEA, or only a whitelist? That single question is the source of the "mixed information."
Lower-risk interim (no rewiring): the watchdog already has a full health record — ship the v2.1 parseable log (docs/LOGGING-SPEC.md) over its console. It's authoritative per-unit health, captured on pulled units; it just isn't live to the server. In-band-to-server telemetry is the Phase-2 upgrade.

1. The as-built data path (why the answer is "not yet")

          ┌──────────────── simpleRTK2B Pro (ZED-F9P) ─────────────────┐
          │                                                            │
 UART1 ◄──┤ TX1 → NMEA (GGA…)  ──►  STM32 PA3 (RX)   [watchdog: READS] │
          │ RX1 ← UBX rate     ◄──  STM32 PA2 (TX)   [watchdog: WRITES]│
          │                                                            │
 UART2 ◄──┤ TX2 → NMEA/GGA @1Hz ──►  4G NTRIP Master (XBee) ──► SERVER │  ← server path
          │ RX2 ← RTCM3        ◄──  4G NTRIP Master (NTRIP caster)     │
          └────────────────────────────────────────────────────────────┘
                                         │
                       4G modem also emits its OWN  $GPTXT,NET,<AS+K id>,TELCEL,…
                       (network status + station id = the backend's cell_id)

Key facts (confirmed in firmware + the 20-Jun bench analysis):

Firmware: HardwareSerial GNSS(PA3, PA2) → F9P UART1. parseNMEA() only reads GGA (for fixQuality/satellites/rxCount); applyRate() only writes UBX. On power-off it floats PA2/PA3. The watchdog never transmits toward the modem.
The modem is on the Pro's XBee socket = UART2 (the "XBEE↔UART2 switch"); GGA is configured out UART2 @ 1 Hz. The server stream and the NTRIP GGA-uplink both ride UART2.
Therefore the watchdog (UART1) and the server path (UART2) are electrically separate. No firmware change alone can bridge them.

2. The three theoretical ways in (and which one to use)

Make the F9P emit it. ✗ Not possible. The ZED-F9P has no command to inject an arbitrary NMEA string into its UART2 output. It emits only its own sentences + UBX replies.
Tell the 4G modem to carry it. ✗ No runtime path. The modem's $GPTXT is network status; AS+K (station id) is set-once config, not a data channel the MCU can stuff health into per-cycle.
Put the watchdog in the uplink NMEA path (relay/injector). ✓ The viable design. Route the F9P→modem NMEA through the STM32:
```
F9P TX2 (NMEA) ──► STM32 RX  ──[relay verbatim]──► STM32 TX ──► modem RX
                               └─[inject $PHLG health @ ~1/30s in the idle gap]
modem ──► forwards EVERYTHING (GGA/RMC/GPTXT + $PHLG) ──► server
RTCM from the modem can still go straight to the F9P RX2 (corrections path intact).
```
The backend already maps (ip,port) → cell_id (from the modem's $GPTXT), so a $PHLG arriving on the same socket is auto-associated to the right unit — no need to put the id in the health sentence. (This reuses the CGNAT-safe identity work shipped in retranslator.py.)

Retrofit variant (no full relay): diode-OR / open-drain the STM32 TX onto the modem's RX and inject only during the F9P's idle gaps (at 115200 baud, 1 Hz NMEA leaves the line idle >90 % of the time). Cheaper to bolt on, but collision-prone and harder to guarantee — prefer the clean relay on the PCB.

3. The proprietary sentence (don't reuse `$GPTXT`)

Collision warning: the backend parses $GPTXT by taking field 2 as the cell_id. A health $GPTXT would be mis-read as a device id. Use a proprietary talker instead — $PHLG (P = proprietary, HLG = Helgen):

$PHLG,<fw>,<state>,<link>,<fix>,<sat>,<rx>,<vbat>,<v5v>,<tc>,<tmax>,<rec>,<limp>,<nmea_age_ms>,<reset>*CS
e.g.  $PHLG,2.2,RUN,DEGR,1,12,34192,3.96,5.01,58,69,0,0,180,POR*11
        (checksum is the standard NMEA XOR over the chars between $ and *, computed at
         runtime by buildPHLG(); -127 in tc/tmax = "n/a". The earlier *4F here was a wrong
         hand-written placeholder — the correct XOR for that body is *49.)

Fields mirror the v2.1 TLM record (state, link, fix, sats, rx, battery, 5 V, MCU temp + session max, recovery count, limp latch, NMEA age, reset cause). CS = standard NMEA XOR checksum.

Backend side is a ~15-line addition to the hardened retranslator.py: a branch that recognises $PHLG, verifies the checksum, and stores the fields per cell_id (keyed by the existing (ip,port) map). Then the monitor can surface, per unit and live: MCU temp / tMax (the thermal-hypothesis evidence, without pulling the unit), reset cause (reboot-loop forensics), recovery count, and LIMP state — and it ships to Loki for free.

Backend `$PHLG` branch (concrete — review before deploy)

Drop-in for retranslator.py. cell_id is already resolved from (ip,port) by the existing CGNAT-safe map; this only adds recognition + checksum + store. Tolerant of the extra trailing fields the v3 board will append (air/RH/battery/fan).

# --- $PHLG health branch (v2.2) — REVIEW BEFORE DEPLOY ------------------------
def _nmea_ok(s: str) -> bool:
    if not s.startswith('$') or '*' not in s:
        return False
    body, _, cs = s[1:].partition('*')
    x = 0
    for ch in body:
        x ^= ord(ch)
    return f"{x:02X}" == cs.strip()[:2].upper()

PHLG_FIELDS = ("fw","state","link","fix","sat","rx","vbat","v5v",
               "tc","tmax","rec","limp","nmea_age_ms","reset")

def handle_phlg(line: str, cell_id: str, store: dict) -> bool:
    if not line.startswith('$PHLG,') or not _nmea_ok(line):
        return False
    body = line[1:].split('*', 1)[0]
    parts = body.split(',')[1:]              # drop the 'PHLG' talker
    rec = dict(zip(PHLG_FIELDS, parts))      # extra v3 trailing fields are simply ignored
    rec["ts"] = time.time()
    store[cell_id] = rec                     # latest health per unit (also ship to Loki)
    log.info("PHLG %s temp=%s tmax=%s reset=%s limp=%s rec=%s",
             cell_id, rec.get("tc"), rec.get("tmax"), rec.get("reset"),
             rec.get("limp"), rec.get("rec"))
    return True

# In the inbound loop, BEFORE the GGA/position path:
#     if handle_phlg(line, cell_id, PHLG_BY_CELL):
#         continue        # health record, NOT a position fix — don't forward to Wialon
# -----------------------------------------------------------------------------

4. What to bench-test before committing (resolves the "mixed info")

Does the 4G NTRIP Master forward a proprietary $P… sentence? ✅ RESOLVED 2026-06-22 — YES. Enabling $PUBX (a u-blox proprietary $P… talker) on the F9P UART2 → it arrives at the server udp/5010 intact, alongside GGA/RMC. The modem is a transparent forwarder; no whitelist, so the $GPTXT,HLT fallback is not needed. (Detail: 03-Architecture-Hardware/watchdog-pcb-v3-design.md §7a.) Original note: inject one on UART2-RX on the bench and watch the server tcpdump. If it whitelists talkers: fall back to a $GPTXT,HLT,… subtype and special-case it in the backend by field 1 (HLT vs NET) so it never collides with the cell-id parse. This is the one experiment that settles the whole question.
GGA integrity for NTRIP. The modem relays the F9P's GGA up to the caster for network RTK. The relay must be byte-exact and low-latency and only inject in idle gaps — never delay or mangle GGA, or corrections degrade.
Rate / cost. Health at 1 / 30–60 s keeps cellular + backend load trivial.
Checksum must be correct or the backend (and possibly the modem) drops it.

5. Other watchdog improvements (independent of the telemetry question)

🔴 5a. The "no-RTK ≠ dead-modem" thrash risk — act on this now

v2 treats fixQuality >= 4 (RTK) as the only healthy link, and after NTRIP_GRACE (120 s) of < 4 it power-cycles the stack. But the 20-Jun bench reality is every unit is SPS (quality 1), no RTK (caster/FLASH/sky issue, not a modem hang). With the watchdog connected as-is, a unit that is GNSS-healthy and forwarding fine at SPS would be power-cycled every ~2 min + backoff — manufacturing the very disconnects we're trying to kill. Fix: only power-cycle on evidence of a link/modem hang (NMEA silence is already handled); treat stable SPS with incrementing rx as a reportable DEGRADED state, not a recovery trigger — at least until the corrections pipeline (move off rtk2go, save F9P config to FLASH, open-sky retest) is fixed. This is the highest-value firmware change right now.

🟠 5b. Ship the v2.1 parseable log (`docs/LOGGING-SPEC.md`)

The ANSI dashboard overwrites itself and has no timestamps/reset-cause → useless as a field record. The v2.1 spec (BOOT/COLS/TLM/EVT, reset cause, ~60 additive lines) makes the watchdog the authoritative per-unit health record. It's also the exact data the $PHLG sentence would carry — do this first, so every later change is observable.

🟠 5c. Independent watchdog (IWDG) — watch the watcher

Nothing currently recovers the Blackpill itself if it hangs. Enabling the STM32 IWDG makes an MCU lockup self-reset and show reset=IWDG next boot — closing the last "silent vanish" gap for a device whose whole job is fail-safe.

🟠 5d. External temp sensor + persisted counters

v2 logs MCU temp, but the heat casualties are the 18650 / 4G modem, not the die. Add a small external sensor near the cell/modem (PCB brief already flags this), and persist tMax / recoveryCount / reset history to RTC backup registers or flash so a pulled unit's forensics survive the power-cycle. Directly serves the thermal RCA.

6. Recommendation (phased)

When	Action	Risk
Now (firmware only)	5a (stop SPS-driven power-cycling) + 5b (v2.1 log) + 5c (IWDG). Bench-test, bump v2.1, then flash.	Low — additive / behavioural guard.
Bench experiment	Test #1: does the modem forward `$P…`? Decide `$PHLG` vs `$GPTXT,HLT`.	None — measurement.
Phase 2 (PCB)	Watchdog as NMEA relay/injector on the UART2 uplink + external temp sensor; backend `$PHLG` parser + monitor/Loki surfacing.	Medium — relay must preserve GGA/NTRIP; validate before fleet.

Bottom line for the deck: Yes, we can put unit health (incl. temperature) into the live server feed — it needs the watchdog inserted into the modem's uplink path (a Phase-2 PCB change) and one bench test of the modem's forwarding. Until then the same health is captured on-unit via the v2.1 log. Separately, before we connect the watchdog we must stop it from power-cycling healthy SPS units.

03-Architecture-Hardware/watchdog-schematic · 03-Architecture-Hardware/firmware-review · 03-Architecture-Hardware/hardware-respec-and-pcb-brief · 04-Source-Code/firmware/docs/LOGGING-SPEC.md · 04-Source-Code/firmware/blackpillrtkcode_v2/blackpillrtkcode_v2.ino

13 KiB Raw Blame History Unescape Escape