diff --git a/02-Architecture-Software/code-review-scale-readiness.md b/02-Architecture-Software/code-review-scale-readiness.md new file mode 100644 index 0000000..d0dbdaf --- /dev/null +++ b/02-Architecture-Software/code-review-scale-readiness.md @@ -0,0 +1,109 @@ +--- +tags: [project-drogheda, backend, code-review, scale] +project: Project Drogheda +author: Helgen CTO +date: 2026-06-20 +status: Review — for the "add more units" test campaign +--- + +# Code Review — retranslator + monitor (scale readiness) + +> Review of the **hardened** `retranslator.py`, `web_monitor.py`, and the +> `gps_logger.py` they depend on, ahead of bringing more units online. The three +> 🔴 field-killers (HOL blocking, anti-jump black-hole, CGNAT identity) are already +> fixed — this pass is about **what else bites as the fleet grows**. Nothing here is +> on fire; the top items are the ones you'll actually hit while testing. + +Severity: 🔴 blocks/badly misleads at scale · 🟠 should fix before the fleet grows · +🟡 polish. + +> **Status (2026-06-20):** **R1, R2, G1, G2 are implemented and verified** — smoke +> (10/10, no regression), an R1/R2 focused test (7/7), and a standalone index+prune +> test all pass. Remaining: **M1** (dashboard auth — an nginx change, server-side) and +> the 🟡 polish items. New env knob: `GPS_RETENTION_DAYS` (default 30, 0 = off). + +## Summary + +The hardened code is solid and the concurrency model is sound. The remaining risks +are mostly **operational at scale**: a new unit that isn't in `device_config.json` +silently won't forward (and the monitor won't tell you why); the SQLite history has +no index and no retention (it will scan slowly and grow until the small disk fills); +the dashboard has no auth and re-renders 500 rows every 2 s; and a couple of latent +parsing/ack assumptions. No SQL injection (queries are parameterized) and no obvious +XSS. + +## 🔴 / 🟠 Issues to fix before adding units + +| # | File · location | Issue | Sev | +|---|---|---|---| +| R1 | `retranslator.py` · `_process_device_telemetry` (`wialon_id = device_cell_id_to_wialon_id.get(cell_id)`) | **An unconfigured `cell_id` is silently never forwarded.** A new unit's GPTXT marks it *online* (so it appears active in the monitor) but with no `wialon_id` mapping it's never sent to Wialon — and it shows as `wialon_state="idle"`, not a clear "unconfigured". This is the #1 thing you'll hit adding units. | 🔴 | +| G2 | `gps_logger.py` · whole file | **SQLite history grows unbounded.** ~288 rows/unit/day → ~144k/day at 500 units, no pruning. On the space-limited server this **fills the disk** over weeks. | 🔴 | +| G1 | `gps_logger.py` · `_create_table` | **No index** on `gps_positions`. `/api/availability` does `WHERE received_at … GROUP BY day,device` → full scan; at millions of rows the report crawls. | 🟠 | +| M1 | `web_monitor.py` / nginx | **No auth on the dashboard or API.** `monitor.helgen.dev` exposes live fleet positions + `/api/status` + `/api/availability` to anyone with the URL. Client-sensitive. | 🟠 | +| R2 | `retranslator.py` · `main()` UDP loop (`raw = udp_data.decode(...).strip()`) | **One NMEA sentence per UDP datagram is assumed.** If a unit ever batches sentences in one packet (`$GPTXT…\r\n$GNGGA…`), only the first is parsed. Latent today (firmware sends one per datagram) but a trap for the planned `$PHLG` injection. | 🟠 | +| R3 | `retranslator.py` · `WialonForwarder._deliver` (`response = s.recv(1024)`) | **Wialon data-ack isn't validated.** Only the *login* ack (`#AL#1`) is checked; the per-packet ack is read and discarded, so a **rejected** packet still counts as `acked` → monitor shows the unit healthy when Wialon isn't accepting it. | 🟠 | + +## 🟡 Worth doing + +| # | File · location | Suggestion | +|---|---|---| +| R4 | `retranslator.py` · config load | New units need a **restart** (`device_config.json` read once). Add a `SIGHUP` handler to reload the map live — no fleet-wide blip every time you add a test unit. | +| M2 | `web_monitor.py` · `renderTable()` (`tbody.innerHTML = rows.map(...)`) | Re-renders **all 500 rows every 2 s**, churning the DOM and dropping scroll/selection. Diff rows, or only re-render on change. | +| R5 | `retranslator.py` · `_build_data_packet` (`datetime.utcnow()`) | Wialon timestamp = **server receive time**, not the GGA fix time. Sub-second for a slow fleet, but parse the GGA UTC if you want exact fix times. | +| R6 | `retranslator.py` · `WialonForwarder.forget` | A `forget()` on a full worker queue is silently dropped → a socket can leak until the next send error. Bounded; low impact. | +| R7 | `retranslator.py` · `_deliver` ack read | Residual per-worker HOL: one slow ack blocks that worker's ~8 devices for up to `TCP_RW_TIMEOUT` (15 s). Far better than global; consider lowering the read timeout. | +| G3 | `gps_logger.py` · `_insert` | `commit()` per row. Fine at the 300 s throttle; revisit if the throttle drops or inserts burst. | +| M3 | `web_monitor.py` · `sse_events` | Re-parses the whole status JSON every 2 s only to validate. Trivial now; grows with the device count. | + +## What looks good ✅ + +- **Concurrency model** — UDP ingest fully decoupled from Wialon I/O; per-worker + queues with drop-oldest backpressure; stats behind a lock; `list()` snapshots in + the stats/cleanup threads (no "dict changed size"). +- **Identity** — `(ip,port)` transport + `cell_id` state is the right CGNAT-safe shape. +- **Anti-jump** — time-aware + self-healing + cleared on disconnect; can't black-hole. +- **Safety** — atomic status-file write (`.tmp` + `rename`); fail-safe drops; SQL writer + is a bounded background thread that never blocks the loop. +- **Security hygiene** — availability SQL is **parameterized** (`?`), DB opened + read-only, free-text escaped (`esc()`), SSE degrades to "gateway down" cleanly. + +## Before you test more units — do these first + +1. **R1 — make unconfigured units obvious.** Two cheap parts: log a one-time + `WARNING` per unknown `cell_id`, and surface a distinct `wialon_state="unconfigured"` + in the monitor so a new unit that isn't forwarding is self-explanatory. + ```python + # in _process_device_telemetry, where wialon_id is looked up: + wialon_id = device_cell_id_to_wialon_id.get(cell_id) + if not wialon_id: + if cell_id not in _warned_unconfigured: # module-level set() + _warned_unconfigured.add(cell_id) + log.warning(f"[CONFIG] cell_id {cell_id} sin wialon_id — " + f"agrégalo a device_config.json o no se reenvía.") + # still record telemetry/odometer; just don't forward + ``` +2. **G2 — cap the SQLite growth.** A nightly prune keeps the disk safe: + ```sql + DELETE FROM gps_positions WHERE received_at < datetime('now','-30 days'); + ``` + (run from the writer on a daily timer, or a cron `sqlite3` one-liner + `VACUUM`). +3. **G1 — add the index** (one line in `_create_table`, idempotent): + ```sql + CREATE INDEX IF NOT EXISTS idx_gps_recv_dev + ON gps_positions(received_at, device_cell_id); + ``` +4. **M1 — gate the dashboard.** Fastest is nginx Basic Auth on `monitor.helgen.dev` + (one `auth_basic` + htpasswd), or an IP allowlist for the office/VPN. +5. **R2 — split multi-sentence datagrams** (`for line in raw.split('\n'): …`) so the + loop is robust before you wire any new sender (incl. `$PHLG`). + +## Verdict + +**Approve, with the R1/G1/G2 items recommended before the test campaign.** The core is +production-grade; the gaps are operational (config ergonomics, DB lifecycle, dashboard +auth) and quick to close. R1 and G2 are the two that will actually bite you while +adding units. + +## Related +- [[02-Architecture-Software/backend-review|backend-review]] (the original 🔴 findings + resolution) · + `04-Source-Code/htracker/retranslator.py` · `web_monitor.py` · `gps_logger.py` diff --git a/03-Architecture-Hardware/watchdog-connections.xlsx b/03-Architecture-Hardware/watchdog-connections.xlsx new file mode 100644 index 0000000..aab7f8f Binary files /dev/null and b/03-Architecture-Hardware/watchdog-connections.xlsx differ diff --git a/03-Architecture-Hardware/watchdog-health-telemetry.md b/03-Architecture-Hardware/watchdog-health-telemetry.md index 5f0e144..79d99eb 100644 --- a/03-Architecture-Hardware/watchdog-health-telemetry.md +++ b/03-Architecture-Hardware/watchdog-health-telemetry.md @@ -106,7 +106,10 @@ talker** instead — `$PHLG` (`P` = proprietary, `HLG` = Helgen): ```text $PHLG,,,,,,,,,,,,,,*CS -e.g. $PHLG,2.1,RUN,LIMP,1,12,34192,4.01,5.02,63,71,2,1,420,IWDG*4F +e.g. $PHLG,2.2,RUN,DEGR,1,12,34192,3.96,5.01,58,69,0,0,180,POR*11 + (checksum is the standard NMEA XOR over the chars between $ and *, computed at + runtime by buildPHLG(); -127 in tc/tmax = "n/a". The earlier *4F here was a wrong + hand-written placeholder — the correct XOR for that body is *49.) ``` Fields mirror the v2.1 `TLM` record (state, link, fix, sats, rx, battery, 5 V, @@ -120,12 +123,55 @@ live**: MCU temp / `tMax` (the thermal-hypothesis evidence, without pulling the reset cause (reboot-loop forensics), recovery count, and LIMP state — and it ships to Loki for free. +### Backend `$PHLG` branch (concrete — review before deploy) + +Drop-in for `retranslator.py`. `cell_id` is already resolved from `(ip,port)` by the +existing CGNAT-safe map; this only adds recognition + checksum + store. Tolerant of the +**extra trailing fields** the v3 board will append (air/RH/battery/fan). + +```python +# --- $PHLG health branch (v2.2) — REVIEW BEFORE DEPLOY ------------------------ +def _nmea_ok(s: str) -> bool: + if not s.startswith('$') or '*' not in s: + return False + body, _, cs = s[1:].partition('*') + x = 0 + for ch in body: + x ^= ord(ch) + return f"{x:02X}" == cs.strip()[:2].upper() + +PHLG_FIELDS = ("fw","state","link","fix","sat","rx","vbat","v5v", + "tc","tmax","rec","limp","nmea_age_ms","reset") + +def handle_phlg(line: str, cell_id: str, store: dict) -> bool: + if not line.startswith('$PHLG,') or not _nmea_ok(line): + return False + body = line[1:].split('*', 1)[0] + parts = body.split(',')[1:] # drop the 'PHLG' talker + rec = dict(zip(PHLG_FIELDS, parts)) # extra v3 trailing fields are simply ignored + rec["ts"] = time.time() + store[cell_id] = rec # latest health per unit (also ship to Loki) + log.info("PHLG %s temp=%s tmax=%s reset=%s limp=%s rec=%s", + cell_id, rec.get("tc"), rec.get("tmax"), rec.get("reset"), + rec.get("limp"), rec.get("rec")) + return True + +# In the inbound loop, BEFORE the GGA/position path: +# if handle_phlg(line, cell_id, PHLG_BY_CELL): +# continue # health record, NOT a position fix — don't forward to Wialon +# ----------------------------------------------------------------------------- +``` + --- ## 4. What to bench-test before committing (resolves the "mixed info") -1. **Does the 4G NTRIP Master forward a proprietary `$P…` sentence?** Inject one on - UART2-RX on the bench and watch the server `tcpdump`. *If it whitelists talkers:* +1. **Does the 4G NTRIP Master forward a proprietary `$P…` sentence?** + ✅ **RESOLVED 2026-06-22 — YES.** Enabling `$PUBX` (a u-blox proprietary `$P…` talker) + on the F9P UART2 → it arrives at the server `udp/5010` **intact**, alongside GGA/RMC. The + modem is a transparent forwarder; **no whitelist**, so the `$GPTXT,HLT` fallback is not + needed. (Detail: `03-Architecture-Hardware/watchdog-pcb-v3-design.md` §7a.) Original note: + inject one on UART2-RX on the bench and watch the server `tcpdump`. *If it whitelists talkers:* fall back to a `$GPTXT,HLT,…` subtype and special-case it in the backend by **field 1** (`HLT` vs `NET`) so it never collides with the cell-id parse. **This is the one experiment that settles the whole question.** diff --git a/03-Architecture-Hardware/watchdog-pcb-v3-blockdiagram.svg b/03-Architecture-Hardware/watchdog-pcb-v3-blockdiagram.svg index 5390b02..18e5e64 100644 --- a/03-Architecture-Hardware/watchdog-pcb-v3-blockdiagram.svg +++ b/03-Architecture-Hardware/watchdog-pcb-v3-blockdiagram.svg @@ -1,6 +1,6 @@ HBOX Watchdog Carrier PCB v3 — power and signal map -24 V cabinet feed passes reverse-polarity protection, then splits: to a new dedicated fan buck feeding two 5 V fan drivers and the extraction fans; and to the external Mean Well 5 V supply that powers the STM32F411 Blackpill, the RTK-stack load switch (to the external simpleRTK2B + 4G stack), and sense circuits. New I2C SHT40 + TMP102 temperature sensors and the fan drivers connect to the MCU. +24 V cabinet feed passes reverse-polarity protection, then splits: to a new dedicated fan buck feeding a 5 V fan driver and the extraction fan; and to the external Mean Well 5 V supply that powers the STM32F411 Blackpill, the RTK-stack load switch (to the external simpleRTK2B + 4G stack), and sense circuits. New I2C SHT40 + TMP102 temperature sensors and the fan driver connect to the MCU. @@ -28,21 +28,16 @@ Fan buck U2 24→5 V dedicated rail · JP1 -Fan drv 1 +Fan driver Q2 · flyback - -Fan drv 2 -Q3 · flyback -2× 5 V fans +1× 5 V fan extraction · ext 24 V - 5V_FAN - STM32F411 Blackpill watchdog · on headers @@ -75,7 +70,7 @@ PB0 -PWM + tach PA6/7 · PB10/12 +PWM + tach PA6 · PB10 BAT sense PA0 diff --git a/03-Architecture-Hardware/watchdog-pcb-v3-bom.xlsx b/03-Architecture-Hardware/watchdog-pcb-v3-bom.xlsx index 062f759..05afc70 100644 Binary files a/03-Architecture-Hardware/watchdog-pcb-v3-bom.xlsx and b/03-Architecture-Hardware/watchdog-pcb-v3-bom.xlsx differ diff --git a/03-Architecture-Hardware/watchdog-pcb-v3-design.md b/03-Architecture-Hardware/watchdog-pcb-v3-design.md index 1e6ebe8..2f5cca6 100644 --- a/03-Architecture-Hardware/watchdog-pcb-v3-design.md +++ b/03-Architecture-Hardware/watchdog-pcb-v3-design.md @@ -23,9 +23,10 @@ supersedes_section: hardware-respec-and-pcb-brief.md §4 (watchdog carrier detai | # | Decision | Consequence | |---|---|---| | 1 | **Scope = watchdog + thermal *carrier*** | Blackpill plugs on via 2×20 headers (reuse v2.1 firmware, no MCU bring-up). Mean Well SD-15B-05 and the battery stay **external modules**. Lowest-risk first board. | -| 2 | **5 V fans** | Driven low-side (MOSFET PWM) + tach. ⚠ They load the same 5 V the modem bursts on — mitigated with a **dedicated fan rail** (see §0a). | +| 2 | **5 V fan ×1** | One extraction fan, low-side (MOSFET PWM) + tach. ⚠ Loads the same 5 V the modem bursts on — mitigated with a **dedicated fan rail** (§0a). Needs a **passive filtered inlet** to pull air through, and has **no redundancy** — a fan failure falls back to conduction + thermal derate (tach-fault flags it). | | 3 | **I²C temp + humidity** | SHT40 (board ambient + RH) + TMP102 probe (battery/modem hot spot) on one I²C bus. | | 4 | **JLCPCB with assembly** | Parts chosen from JLC/LCSC stock; we output Gerbers + BOM + CPL. Footprints biased to JLC-placeable + hand-reworkable. | +| 5 | **Live telemetry ($PHLG): bench-test first, NOT built into v3** (2026-06-22) | The 4G modem keeps its proven home on the F9P **XBee socket = UART2**; there is **no spare UART** to tap (see §2a). v3 ships health on the **USB STREAM log** (already works). PA9/PA10 header is **reserved/DNP** only. The forwarding question is settled by an isolated **bench experiment** (§7a), not by a board change. | --- @@ -33,15 +34,15 @@ supersedes_section: hardware-respec-and-pcb-brief.md §4 (watchdog carrier detai ### 0a. 5 V fans must NOT share the RTK stack rail The Mean Well **SD-15B-05** gives **5 V @ 3 A**. The RTK+4G stack already peaks -**~2–2.5 A on LTE-TX bursts** — the brownout the whole redesign is fixing. Two -5 V fans add **~0.3–0.5 A plus inrush**. Coincident worst case ≈ 2.8–3.0 A sits -right on the supply limit, and **fan inrush + a modem burst at the same instant -is exactly the sag that drops the link.** So the board gives the fans their **own +**~2–2.5 A on LTE-TX bursts** — the brownout the whole redesign is fixing. One +5 V fan adds **~0.15–0.25 A plus inrush**. Coincident worst case ≈ 2.7 A still +nears the supply limit, and **fan inrush + a modem burst at the same instant +is exactly the sag that drops the link.** So the board gives the fan its **own 5 V** from a small **24→5 V buck** tapped off the cabinet 24 V (which is already -inside the HBOX). A **0 Ω jumper (JP1)** lets you instead bridge fans to the main -5 V if you later upsize the supply — one board, both strategies. **Do not power -fans from the *switched* (load-switch) stack 5 V** — fans must run when hot even -while the stack is power-cycling. +inside the HBOX). A **0 Ω jumper (JP1)** lets you instead bridge the fan to the main +5 V — with a single ~0.2 A fan that's more defensible than with two, but the +dedicated rail stays the safe default. **Do not power the fan from the *switched* +(load-switch) stack 5 V** — it must run when hot even while the stack is power-cycling. ### 0b. Fans + IP67 is a real tradeoff — own it **Moving air in/out of the box breaks the IP67 seal.** A fan only helps if it @@ -85,7 +86,7 @@ fragile hardware**. Keep the brains, fix the body. | LED + dividers on Dupont | mechanical | all on-board, 0603, conformal-coatable | | `1.5 kΩ "????"` mystery R | undocumented | resolved on this schematic (it was the **LED series R**; dividers are 10k/10k — see §6) | | MCU temp only | heat casualties are the **18650/modem**, not the die | **external** I²C sensors at the hot spot (§5) | -| No thermal *actuation* | firmware can derate but can't cool | **2× fan drivers + closed-loop control** (§4, §7) | +| No thermal *actuation* | firmware can derate but can't cool | **1 fan driver + closed-loop control** (§4, §7) | | No SWD/header bring-out | can't debug a sealed unit | **SWD + USB + reserve-USART1** headers (§6) | --- @@ -103,11 +104,11 @@ fragile hardware**. Keep the brains, fix the body. PA5 IGN_SENSE GPIO in ignition via DFR1015 (INPUT_PULLDOWN) PB0 EXT_LED GPIO out external status LED (low-side MOSFET) PC13 LED GPIO out onboard LED (active-LOW) - ── NEW for v3 (fans + temperature) ─────────────────────────────────────── - PA6 FAN1_PWM TIM3_CH1 25 kHz PWM → fan-1 low-side gate - PA7 FAN2_PWM TIM3_CH2 25 kHz PWM → fan-2 low-side gate - PB10 FAN1_TACH EXTI fan-1 tach (open-collector, 10k pull-up to 3V3) - PB12 FAN2_TACH EXTI fan-2 tach (open-collector, 10k pull-up to 3V3) + ── NEW for v3 (fan + temperature) ──────────────────────────────────────── + PA6 FAN_PWM TIM3_CH1 25 kHz PWM → fan low-side gate + PB10 FAN_TACH EXTI fan tach (open-collector, 10k pull-up to 3V3) + PA7 (spare) TIM3_CH2 reserved — 2nd fan PWM if ever needed + PB12 (spare) EXTI reserved — 2nd fan tach if ever needed PB6 I2C1_SCL I2C1 SHT40 + TMP102 (4.7k pull-ups to 3V3) PB7 I2C1_SDA I2C1 SHT40 + TMP102 ── reserved (do not populate now; future in-band $PHLG telemetry relay) ─── @@ -117,10 +118,29 @@ fragile hardware**. Keep the brains, fix the body. PA11 USB_DM / PA12 USB_DP Blackpill micro-USB = STREAM log capture ``` -No conflicts: TIM3_CH1/CH2 (PA6/PA7) do fan PWM; PB0 (TIM3_CH3) stays plain-GPIO -LED. The reserved USART1 pair future-proofs the board for the +No conflicts: TIM3_CH1 (PA6) does fan PWM; PB0 (TIM3_CH3) stays plain-GPIO +LED. The reserved USART1 pair (H3) **future-proofs** the board for the [[03-Architecture-Hardware/watchdog-health-telemetry|`$PHLG` health-in-server]] upgrade -without a respin. +without a respin — but in v3 it is **left unpopulated; no relay is wired** (see §2a + the +decision row above). **PA10 caveat:** PA10 is USART1-RX *and* a known Blackpill DFU-entry +pin — a load/float here can block flash mode. Keep H3 unpopulated, or fit a **~22 kΩ +pull-up on PA10 to 3V3**, so this board never inherits the flaky-flashing gremlin. + +### 2a. Why there is no spare UART (the telemetry reality) + +The simpleRTK2B **Pro** breaks out **both** F9P UARTs, but only one is free: + +| F9P port | Header pins | Used by | Free? | +|---|---|---|---| +| **UART1** | TX1 / RX1 | **watchdog** (NMEA read + UBX rate) — PA2/PA3 | taken | +| **UART2** | TX2 / RX2 **and the XBee socket** | **4G NTRIP Master** (server uplink) | **no — shared bus** | + +`TX2` = F9P-UART2-TX = the **modem's DIN** (the wire the modem reads and forwards to the +server). The F9P already drives it, so the watchdog can't simply "connect a second wire" to +send `$PHLG` — that's two transmitters on one line. Getting health to the server therefore +needs a **relay** (interpose the MCU between F9P-UART2 and the modem), which is a deliberate +architecture change **deferred out of v3**. To *learn whether it's even possible*, run the +isolated bench experiment in **§7a** — it touches no PCB. --- @@ -139,10 +159,8 @@ without a respin. 18650 / PowerBoost (EXTERNAL) ──► P4 [BAT+ , BAT_SENSE] ─► PA0 divider FAN rail: 24V ─► U2 buck (MP1584EN) ─► 5V_FAN (JP1: alt-bridge to MAIN 5V) - 5V_FAN ─┬─► Fan1 (+) ───┐ ┌─── Fan1 tach ─10k─► PB10 - │ Fan1 (−) ─ Q2 (AO3400) ◄PWM PA6 │ (D2 SS34 flyback ∥ fan) - └─► Fan2 (+) ───┘ └─── Fan2 tach ─10k─► PB12 - Fan2 (−) ─ Q3 (AO3400) ◄PWM PA7 + 5V_FAN ───► Fan (+) ┌─── Fan tach ─10k─► PB10 + Fan (−) ─ Q2 (AO3400) ◄PWM PA6 ┘ (D2 SS34 flyback ∥ fan) I²C1 (PB6/7) ─┬─ SHT40 (U3, 0x44) board ambient temp + RH └─ TMP102 (U4, 0x48) on P5 pigtail → battery/modem hot spot @@ -202,19 +220,19 @@ solid, sourceable SMD, plus a resettable fuse for the missing current limit: eFuse (e.g., **AP2553W6-7** SOT-23-6, EN active-high, FLAGn). Lay the footprint *and* the discrete; populate one. Verify LCSC stock at order; **default = discrete**. -### 4.4 Fan drivers ×2 (low-side PWM + tach) — Q2/Q3 -Per fan (identical for fan-1 and fan-2): +### 4.4 Fan driver ×1 (low-side PWM + tach) — Q2 +Single extraction fan: ``` - 5V_FAN ──► Fan(+) Fan tach ──[R_t 1k]──┬──► PBxx (EXTI) - Fan(−) ──┬──► Q2/Q3 Drain │ - D2/D3 SS34 ∥ fan │ (AO3400A, N-MOS) [R 10k pull-up → 3V3] + 5V_FAN ──► Fan(+) Fan tach ──[R_t 1k]──┬──► PB10 (EXTI) + Fan(−) ──┬──► Q2 Drain │ + D2 SS34 ∥ fan │ (AO3400A, N-MOS) [R 10k pull-up → 3V3] (cathode→5V_FAN) │ Source ── GND PWM PAx ─[Rg 100Ω]─ Gate ──[Rgs 100k → GND] ``` -- **Q2/Q3 = AO3400A** logic-level N-MOSFET (SOT-23): full-on from 3.3 V PWM, 5.7 A / +- **Q2 = AO3400A** logic-level N-MOSFET (SOT-23): full-on from 3.3 V PWM, 5.7 A / 30 V — vast margin over a <0.5 A fan. -- **D2/D3 = SS34** Schottky **flyback** across each fan (motor is inductive — protects +- **D2 = SS34** Schottky **flyback** across the fan (motor is inductive — protects the FET at PWM turn-off). Cathode to 5V_FAN, anode to drain. - **Rg 100 Ω** gate series (tames edges/EMI), **Rgs 100 kΩ** gate→GND pull-down = **fans OFF until the MCU drives PWM** (fail-safe; consistent with the ethos — if the @@ -266,8 +284,7 @@ ADC stays ≤3.3 V with margin (verified in firmware-review). Firmware | P3 | JST-XH 4-pos | 4 | SW_5V, GND, GNSS_TX(PA2), GNSS_RX(PA3) → simpleRTK2B | | P4 | JST-XH 2-pos | 2 | BAT+ (to PA0 divider), GND (18650/PowerBoost) | | P5 | JST-XH 4-pos | 4 | 3V3, GND, SDA, SCL → TMP102 probe pigtail | -| P6 | JST-XH 4-pos | 4 | 5V_FAN, Fan1−(PWM), GND, Tach1 → fan 1 | -| P7 | JST-XH 4-pos | 4 | 5V_FAN, Fan2−(PWM), GND, Tach2 → fan 2 | +| P6 | JST-XH 4-pos | 4 | 5V_FAN, Fan−(PWM), GND, Tach → fan | | P8 | JST-XH 2-pos | 2 | EXT_LED+, EXT_LED− (panel LED) | | H1 | 1×4 0.1″ | 4 | SWD: 3V3, SWDIO, SWCLK, GND | | H2 | (Blackpill micro-USB) | — | STREAM log capture / flashing | @@ -284,9 +301,8 @@ ADC stays ≤3.3 V with margin (verified in firmware-review). Firmware The v2.1 `.ino` changes are **small and additive** (the ethos: bench first; the sandbox can't build STM32). New globals + functions, no state-machine surgery. -1. **Pins**: add `PIN_FAN1_PWM PA6`, `PIN_FAN2_PWM PA7`, `PIN_FAN1_TACH PB10`, - `PIN_FAN2_TACH PB12`; init TIM3 CH1/CH2 PWM at **25 kHz**; attach EXTI counters on - the tach pins. +1. **Pins**: add `PIN_FAN_PWM PA6`, `PIN_FAN_TACH PB10` (PA7/PB12 stay spare); + init TIM3 CH1 PWM at **25 kHz**; attach an EXTI counter on the tach pin. 2. **I²C temps**: `Wire.begin()` on PB6/PB7; read SHT40 (0x44, temp+RH) and TMP102 (0x48). Add `tAirC`, `rhPct`, `tBatC` to telemetry. Keep MCU-internal as cross-check. 3. **Closed-loop fan curve** (input = `max(tAirC, tBatC)`): @@ -298,15 +314,55 @@ sandbox can't build STM32). New globals + functions, no state-machine surgery. 4. **Tach fault**: once/30 s, command a fan to **100 % for ~300 ms**, count tach edges, compute RPM. `0 RPM while commanded > 0` → latch `FAN_FAULT`, log `EVT`, raise it in telemetry. A dead fan in a hot box must be **visible**, not silent. -5. **Telemetry**: extend the v2.1 `TLM`/`$PHLG` record with - `tair,rh,tbat,fan1_rpm,fan2_rpm,fan_pwm,fan_fault`. These are exactly the fields the - `$PHLG`-to-server upgrade will carry — capture them on-unit now via the STREAM log. +5. **Telemetry (on-unit only in v3)**: extend the v2.1 `TLM` record with + `tair,rh,tbat,fan_rpm,fan_pwm,fan_fault`, captured to the **USB STREAM log**. + These are the same fields a future `$PHLG`-to-server sentence would carry — but **v3 does + NOT push them live to the server** (no UART relay; see §2a + decision row 5). Live + telemetry is gated on the bench experiment below. 6. **IWDG-safe**: fan/tach/I²C work is non-blocking; keep petting the dog. I²C read uses a short timeout so a stuck sensor can't hang the loop. Pin map and control law live here; the implementation lands in `blackpillrtkcode_v2.ino` **after** bench validation on the prototype board. +### 7a. The one bench experiment that decides live telemetry (no PCB change) + +> **✅ RESULT — 2026-06-22: PASS (no extra hardware needed).** Enabled `$PUBX` (a u-blox +> **proprietary** `$P…` talker — same class + checksum format as `$PHLG`) on the F9P +> **UART2** and watched the server `udp/5010` capture: `$PUBX,00,…` arrives **end-to-end, +> intact**, alongside GGA/RMC. → **The 4G NTRIP Master forwards proprietary `$P…` +> sentences, so `$PHLG` health-in-server is confirmed viable.** No XBee-breakout test +> needed. Remaining work = the **UART2 relay** (interpose the watchdog to inject `$PHLG`) +> \+ backend parser — a deliberate later board, still out of v3 scope. +> +> *Incidental (flagged for GNSS follow-up):* the same capture caught two live units — +> bench **H0014 (Querétaro) at SPS (GGA quality 1), RSSI −93**, and field +> **H0017 (Veracruz/ICAVE) at DGPS (quality 2, RMC mode `D`, diff base id 0133), RSSI −51**. +> Per Leo the bench is in a **basement** (poor sky + weak cellular) — which alone explains +> the bench SPS, since **RTK fix needs open sky (carrier-phase) regardless of +> corrections/config**. But note H0017 has *good* sky+signal and still only reaches **DGPS, +> not RTK (4/5)** → the **corrections pipeline** (free rtk2go caster + RAM-only F9P config) +> is a *separate* ceiling from antenna placement. One **open-sky test with a proper caster + +> FLASH-saved config** separates the two. (Reinforces the v2.1 SPS-thrash guard: a healthy +> basement/field unit may legitimately sit at SPS/DGPS — do **not** power-cycle it.) + +Settles "does the 4G NTRIP Master forward a proprietary `$P…` sentence?" by **isolating the +modem** — the F9P is not involved, so there's no bus collision: + +1. Remove the **4G NTRIP Master** from the F9P XBee socket; mount it on a bench/XBee adapter + with **3.3 V power, the SIM, and the LTE antenna**. +2. Wire **Blackpill PA9 (USART1 TX) → modem DIN** (the `TX2`/F9P-UART2-TX net in the socket) + and **modem DOUT → Blackpill PA10 (RX)** *or* a USB-UART so a PC can watch it. +3. Match the modem's UART baud (ArduSimple default **115200**, 3.3 V). Let it register. +4. Emit `$PHLG,2.1,RUN,LIMP,1,12,34192,4.01,5.02,63,71,2,1,420,IWDG*CS\r\n` and watch the + **server `tcpdump`**. + - **Appears →** modem forwards proprietary sentences → `$PHLG` is viable → *then* plan a + relay-capable board. + - **Doesn't →** modem whitelists talkers → fall back to a `$GPTXT,HLT,…` subtype + (special-cased by **field 1** so it never collides with the cell-id parse). + +Either way, **v3 ships unchanged** — this experiment only informs a *later* decision. + --- ## 8. Layout, stackup & DFM (for a relative beginner — the rules that matter) @@ -342,8 +398,8 @@ Pin map and control law live here; the implementation lands in blocks a flipped input, TVS not hot. 2. **Load switch**: toggle PA1 → SW_5V on/off; confirm OFF at reset; PPTC holds the measured stack current, trips on a deliberate short. -3. **Fans**: ramp PA6/PA7 PWM 0→100 %, scope the gate + fan node, confirm flyback, - read **tach RPM** vs PWM; pull a fan → confirm `FAN_FAULT`. +3. **Fan**: ramp PA6 PWM 0→100 %, scope the gate + fan node, confirm flyback, + read **tach RPM** vs PWM; pull the fan → confirm `FAN_FAULT`. 4. **Sensors**: I²C scan finds 0x44 + 0x48; heat the TMP102 probe → fan ramps per curve. 5. **Power budget (the §0a check)**: with **JP1 open**, scope MAIN 5 V during an LTE burst **and** a fan ramp simultaneously — confirm the stack rail no longer sags. @@ -359,7 +415,7 @@ Pin map and control law live here; the implementation lands in - [ ] **IP rating target** with ICAVE: IP67 (no vents → conduction only) vs IP54/55 (filtered louvers + fans). Determines whether the fan section is populated. -- [ ] **Fan choice**: 5 V **3-wire** (tach) confirmed? Size (40/50/60 mm) + measured +- [ ] **Fan choice (×1)**: 5 V **3-wire** (tach), size (40/50/60 mm), + a **filtered passive inlet** so the single extraction fan can pull air through. Measured static-pressure need through the chosen duct/filter. Source 2–3 candidates. - [ ] **U2 buck vs upsize supply** (§4.2): lay the buck (default) or go 5 V/5 A external + JP1-closed. Recommend laying the buck for rail isolation. diff --git a/03-Architecture-Hardware/watchdog-pcb-v3-kicad-guide.md b/03-Architecture-Hardware/watchdog-pcb-v3-kicad-guide.md index b10c447..571c362 100644 --- a/03-Architecture-Hardware/watchdog-pcb-v3-kicad-guide.md +++ b/03-Architecture-Hardware/watchdog-pcb-v3-kicad-guide.md @@ -96,11 +96,11 @@ For each, place the symbols, wire locally, and tie to power/net labels: L2, C22, C23, the FB divider R20/R21, and **JP1** (solder-jumper symbol). - **Load switch (§4.3):** Q1 AO3401A + Q5 MMBT3904 + R1/R2 + **F1 PPTC** + C3. Net the drain to **SW_5V**. (Optional U1 eFuse footprint — leave it **DNP**.) -- **Fan drivers ×2 (§4.4):** Q2/Q3 AO3400A, D2/D3 SS34 flyback, Rg/Rgs gate parts, tach - pull-up + series into **PB10 / PB12**. PWM gates from **PA6 / PA7**. +- **Fan driver ×1 (§4.4):** Q2 AO3400A, D2 SS34 flyback, Rg/Rgs gate parts, tach + pull-up + series into **PB10**. PWM gate from **PA6** (PA7/PB12 left as spares). - **Sensors (§5):** U3 SHT40 + U4 TMP102 on **PB6/PB7**, 4.7k pull-ups, 100nF each. - **Sense + LED (§4.5–4.6):** the three 10k/10k dividers (PA0/PA4/PA5), D5 clamp, Q4 + D1 + R3. -- **Connectors (§6):** P1–P8, H1, H3 (DNP), and the two 2×20 sockets. +- **Connectors (§6):** P1–P6 + P8, H1, H3 (DNP), and the two 2×20 sockets. ### 4.4 Annotate + ERC - **Tools → Annotate** (gives refs R1, U2, …). @@ -140,7 +140,7 @@ on **easyeda2kicad** / the JLC part page, grab the EasyEDA footprint and import. **mounting holes** so the board can bolt to the cabinet **metal plate** (design-doc §8 — that's your conduction path). 3. **Place by functional block** (keep each §4 sub-circuit together): power-in + buck in one - corner, load switch near P3, fan drivers near P6/P7, sensors near an air edge / P5, the + corner, load switch near P3, fan driver near P6, sensors near an air edge / P5, the Blackpill sockets central, connectors on the edges. 4. **Stackup:** 2-layer is fine (cheaper); **4-layer** is better for RF/buck noise. Add a **ground pour** (filled zone = GND) on the bottom (and an inner layer if 4-layer). @@ -204,7 +204,7 @@ pigtail when it arrives. - **Rotations**: JLC's 0° reference often differs from KiCad's. Always eyeball the placement preview; SOT-23 and electrolytics are the usual offenders. -- **Polarity**: double-check D10 TVS, C10 electrolytic, D2/D3 Schottky, the LED, and Q10/Q1 +- **Polarity**: double-check D10 TVS, C10 electrolytic, D2 Schottky, the LED, and Q10/Q1 source/drain before ordering. - **The buck (U2)** is the only "real" analog block — if you're nervous, **populate JP1 closed + upsize the 5 V supply** and leave U2 unpopulated for the first spin (design-doc diff --git a/03-Architecture-Hardware/watchdog-protoboard-bringup.md b/03-Architecture-Hardware/watchdog-protoboard-bringup.md new file mode 100644 index 0000000..0194f58 --- /dev/null +++ b/03-Architecture-Hardware/watchdog-protoboard-bringup.md @@ -0,0 +1,123 @@ +--- +tags: [project-drogheda, hardware, watchdog, prototype, breadboard, bom, bring-up] +project: Project Drogheda +author: Helgen CTO +date: 2026-06-22 +status: Active — shop list for the Blackpill bring-up +--- + +# Watchdog Bring‑up — Protoboard Shopping List (print me) + +> Goal: breadboard the v2.2 watchdog on your **existing Blackpill**, prove the +> state machine + sensing + load switch + fans + temp loop, **before** any PCB. +> All parts are **through‑hole / breadboard‑friendly** and stocked at a normal MX +> electronics shop (Steren / AG Electrónica / local). Quantities include spares. +> Pin map: [[03-Architecture-Hardware/watchdog-pcb-v3-design#2|design §2]] · +> firmware: `blackpillrtkcode_v2.ino` (v2.2). + +## 0. Already have — do NOT buy +STM32F411 Blackpill · USB‑C/micro cable · 18650 + PowerBoost · Mean Well SD‑15B‑05 +(or a bench PSU) · multimeter · the F9P + 4G master (only needed later, not for the +bring‑up). A **heat source** (hair dryer / lighter / hot water cup) to trigger the +thermal/fan test — household, no need to buy. + +--- + +## 1. Core — bring up the "brains" (buy this even if the shop is sparse) + +| # | Item | Qty | Value / part | ~MXN | For | +|---|---|---|---|---|---| +| 1 | Breadboard 830‑pt | 2 | half/full | 120 | build surface (2 = room for fans/sensors) | +| 2 | Jumper wires M‑M | 1 pk | 65 pcs | 60 | board wiring | +| 3 | Jumper wires M‑F | 1 pk | 40 pcs | 50 | to Blackpill / modules | +| 4 | Resistor assortment box | 1 | ¼ W (incl. 330Ω,1k,4.7k,10k,100k,220Ω) | 120 | dividers, gates, pull‑ups | +| 5 | LED (any) | 5 | 3/5 mm | 10 | status LED on PB0 | +| 6 | Ceramic cap 100 nF | 10 | 0.1 µF (104) | 20 | ADC filters + decoupling | +| 7 | Electrolytic 100 µF / ≥16 V | 4 | — | 20 | rail bulk / modem‑burst reservoir | +| 8 | **2N3904** (NPN, TO‑92) | 3 | load‑switch driver | 15 | PA1 → P‑FET gate | +| 9 | **IRF9540 / IRF9540N** (P‑MOSFET, TO‑220) | 2 | high‑side 5 V switch | 40 | switched 5 V to "stack" | +| 10 | **10 kΩ trim pot** | 2 | breadboard trimmer | 30 | sweep VBAT / ignition to test thresholds | +| 11 | Tactile button or slide switch | 2 | — | 20 | simulate ignition on/off | +| 12 | Male header strip | 1 | 1×40, 2.54 mm | 15 | solder onto any module that needs pins | + +**Core subtotal ≈ 520 MXN.** With this you can flash v2.2, watch the `$PHLG`/TLM log, +and validate the **load switch + the three sense dividers + LED** — the whole watchdog +logic — without the thermal parts. + +--- + +## 2. Thermal add‑ons — fans + temperature (the v3 new stuff) + +| # | Item | Qty | Value / part | ~MXN | For / note | +|---|---|---|---|---|---| +| 13 | 5 V fan, 40–60 mm, **3‑wire** | 1 | 3‑wire = has tach | 60 | one extraction fan; 3‑wire so PB10 reads RPM. Grab a 2nd as a spare if cheap | +| 14 | **IRLZ44N** (logic‑level N‑MOSFET, TO‑220) | 2 | fan low‑side PWM | 30 | 1 + spare; **must be logic‑level**; see substitute below | +| 15 | Schottky **1N5819** | 3 | fan flyback | 10 | one across the fan (1N4007 ok if out) | +| 16 | I²C temp+RH breakout | 1 | **SHT31 / SHT40** ideal | 120 | matches firmware (0x44). RH catches condensation | +| 17 | Analog temp **LM35** (TO‑92) | 2 | sure‑thing fallback | 25 | every shop has it; heat it → watch fans ramp | +| 18 | **DS18B20** waterproof probe | 1 | 1‑wire | 45 | great as the battery/modem hot‑spot probe | +| 19 | LM2596 buck module (adj.) | 1 | 24/12 → 5 V | 35 | dedicated **fan 5 V** rail (caveat §0a) if no 2nd bench PSU | + +**Thermal subtotal ≈ 260 MXN** (one fan + driver + one temp option). Grab #13–15 for the fan, and **one** temperature option +(#16 ideal, else #17/#18). + +--- + +## 3. Substitutions (shop stock varies — these all work) + +| If they're out of… | Buy instead | Catch | +|---|---|---| +| IRLZ44N (logic‑level FET) | **2N2222A** (TO‑92) or **TIP120** (Darlington) | BJT: add 1 kΩ base resistor; fine for a <0.5 A fan. **Plain IRF540/IRFZ44 won't fully turn on from 3.3 V — avoid.** | +| SHT31/SHT40 I²C breakout | **BME280/BMP280** (I²C, common) | address 0x76/0x77 → small firmware change | +| Any I²C sensor | **LM35** (analog) or **DS18B20** (1‑wire) | not I²C → firmware reads ADC / 1‑wire instead (I'll patch it) | +| IRF9540 P‑MOSFET | keep your original, or **2N2222 high‑side** is awkward — ask for any P‑channel ≥30 V | gate fully pulled to GND by the 2N3904, so a standard P‑FET is OK here | +| 1N5819 Schottky | **1N4007** | slightly slower; fine at fan PWM | + +--- + +## 4. Bring‑up order (so the list maps to actions) + +1. **Just USB** — flash v2.2, open serial @115200. Confirm `BOOT/COLS/TLM` lines **and a + `$PHLG,...*CS` every 30 s**. (No external parts — this proves the firmware.) +2. **LED on PB0** via 330 Ω → watch the blink codes change with state. +3. **Load switch** — 2N3904 + IRF9540 + 1 kΩ (base) + 10 kΩ (gate pull‑up) from a 5 V rail; + put an LED+330 Ω on the switched output. Toggle: confirm **OFF at reset**, ON when PA1 + drives. (This is the heart of recovery.) +4. **Sense dividers** — two 10 kΩ each (÷2) into **PA0** (battery via 18650) and **PA4** + (5 V rail); a 10 kΩ pot into **PA5** (ignition). Sweep them and watch `veh`/`ign` flip in + the TLM with hysteresis/debounce. +5. **Fan** — PA6 → gate (220 Ω) of IRLZ44N → fan(−); flyback diode across the fan; + 100 kΩ gate→GND. Watch it spin/ramp; tach wire → 10 kΩ pull‑up to 3V3 → PB10 → + RPM in the log; **unplug the fan → `FAN_FAULT`**. +6. **Temperature** — I²C breakout on PB6/PB7 (+4.7 kΩ pull‑ups), or LM35 on an ADC pin. + Heat it → fans ramp per the 45→70 °C curve; at 75 °C the existing thermal derate kicks in. + +--- + +## 5. Wiring cheat‑sheet (Blackpill side) + +``` +PA0 ← BAT ÷2 (10k/10k) from 18650+ PA6 → 220Ω → IRLZ44N gate; fan(−); 1N5819 ∥ fan +PA4 ← 5V ÷2 (10k/10k) from 5V rail PB10 ← fan tach, 10k pull-up → 3V3 (1k series) +PA5 ← IGN ÷2 (10k/10k) from pot/switch PB6/PB7 → I²C SCL/SDA (+4.7k each → 3V3) +PA1 → 1k → 2N3904 base; collector → IRF9540 PB0 → 330Ω → LED → GND + gate; 10k gate→5V; drain → switched (PA7/PB12 left free — 2nd fan if ever needed) + 5V → stack (use an LED load to test) +GND/5V/3V3 rails common; fan 5V from a SEPARATE 5V (LM2596 or 2nd PSU) — not the stack rail. +``` + +--- + +## 6. Note on the temp sensor + firmware +The v2.2 firmware reads **I²C SHT40 (0x44) + TMP102 (0x48)**. Whatever you bring back: +- **SHT31/SHT40** → works as‑is (SHT40) or a 1‑line address tweak (SHT31). +- **BME280** → change address + read call. +- **LM35 / DS18B20** → analog / 1‑wire; tell me which and I'll patch `buildPHLG()` + + the fan‑curve input to read it. For *tomorrow's* bring‑up the fan loop can also be + driven off the **MCU‑internal** temp (already in firmware) — heat the whole board and + the derate still triggers, so you can prove the fan path even before the sensor arrives. + +## Related +- [[03-Architecture-Hardware/watchdog-pcb-v3-bom|v3 BOM (production + bench, xlsx)]] · + [[03-Architecture-Hardware/watchdog-pcb-v3-design|v3 design (pin map, sub‑circuits)]] · + `04-Source-Code/firmware/blackpillrtkcode_v2/blackpillrtkcode_v2.ino` diff --git a/03-Architecture-Hardware/~$Drogheda-BOM-Maestro.xlsx b/03-Architecture-Hardware/~$Drogheda-BOM-Maestro.xlsx new file mode 100644 index 0000000..bb7c09c Binary files /dev/null and b/03-Architecture-Hardware/~$Drogheda-BOM-Maestro.xlsx differ diff --git a/03-Architecture-Hardware/~$watchdog-pcb-v3-bom.xlsx b/03-Architecture-Hardware/~$watchdog-pcb-v3-bom.xlsx new file mode 100644 index 0000000..bb7c09c Binary files /dev/null and b/03-Architecture-Hardware/~$watchdog-pcb-v3-bom.xlsx differ diff --git a/04-Source-Code/htracker b/04-Source-Code/htracker index f93a36d..ed51abe 160000 --- a/04-Source-Code/htracker +++ b/04-Source-Code/htracker @@ -1 +1 @@ -Subproject commit f93a36d3ae8d64ead1036f422a32024eb3cb9cbf +Subproject commit ed51abe602ba401350347b0be135197af333e027 diff --git a/05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.pdf b/05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.pdf new file mode 100644 index 0000000..9cfe241 Binary files /dev/null and b/05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.pdf differ diff --git a/05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.tex b/05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.tex new file mode 100644 index 0000000..0d0b2c6 --- /dev/null +++ b/05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.tex @@ -0,0 +1,195 @@ +% Helgen Drogheda — Manual de Consulta de Logs (Grafana / Loki · backend htracker) +% Compilar: latexmk -xelatex 2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.tex (XeLaTeX) +\documentclass[10pt]{article} +\newcommand{\sistema}{Helgen Drogheda} +\newcommand{\sistemapdf}{Manual de Consulta de Logs — Grafana / Loki} +\newcommand{\doctipo}{Manual de Operación} +\input{estilo-helgen.tex} +\usepackage{amssymb} + +\begin{document} + +\helgenportada{Consulta de Logs en Grafana}{Observabilidad del backend \texttt{htracker} — Grafana Loki (capa gratuita)}{% +\begin{tabular}{@{}ll@{}} +Documento & Manual de Operación\\ +Versión & 1.0\\ +Fecha & 22 de junio de 2026\\ +Autor & Dr.\ Leopoldo Rodríguez Salazar — Helgen Technologies\\ +Sistema & \texttt{htracker} (retranslator + monitor), servidor \texttt{HelgenServer}\\ +Fuente & \texttt{04-Source-Code/htracker/log-shipping/}\\ +Clasificación & Confidencial — Uso interno\\ +\end{tabular}} + +\tableofcontents +\vspace{4pt} + +\section{Propósito} +El servidor del backend tiene \textbf{muy poco disco}, por lo que el gateway \textbf{no} +guarda logs localmente. En su lugar, los registros se \textbf{envían fuera del servidor} +a \textbf{Grafana Loki} (capa gratuita) y se consultan desde \textbf{Grafana}. Este manual +explica cómo \emph{entrar} a Grafana y \emph{consultar} los logs de la flota — no requiere +acceso al servidor para leer. + +\begin{helgennotice}[En una frase] +Los logs del backend viven en \textbf{Grafana Loki}. Para verlos: abre Grafana +$\rightarrow$ \textbf{Explore} $\rightarrow$ fuente de datos \textbf{Loki} $\rightarrow$ +escribe \texttt{\{job="htracker"\}}. +\end{helgennotice} + +\section{Arquitectura del envío de logs} +El gateway escribe a la salida estándar; \texttt{systemd} la captura en \texttt{journald} +(un búfer en anillo \textbf{limitado a 200 MB}, que nunca llena el disco); un agente ligero +(\textbf{Vector}, o \textbf{Grafana Alloy}) lee el journal y lo empuja por HTTPS a Loki. + +\begin{center} +\begin{tikzpicture}[node distance=6mm and 8mm] + \node[caja] (rt) {retranslator\\\scriptsize(stdout)}; + \node[caja, right=of rt] (jd) {journald\\\scriptsize(tope 200 MB)}; + \node[caja, right=of jd] (vec) {Vector\\\scriptsize(o Alloy)}; + \node[cajaoscura, right=of vec] (loki) {Grafana\\Loki}; + \node[caja, right=of loki] (graf) {Grafana\\\scriptsize(UI)}; + \draw[flecha] (rt)--(jd); \draw[flecha] (jd)--(vec); + \draw[flechaacento] (vec)-- node[etiqueta, above]{HTTPS} (loki); \draw[flecha] (loki)--(graf); +\end{tikzpicture} +\end{center} + +\noindent\textbf{Por qué así:} el gateway nunca escribe un archivo que crece; \texttt{journald} +es un búfer pequeño y limitado (sobrevive cortes breves de Loki); el agente empuja con un +búfer acotado que \emph{descarta lo más nuevo} si Loki no responde — de modo que la ruta de +logs \textbf{jamás} bloquea ni tumba al gateway. + +\begin{helgennotice}[¿Vector o Alloy? — verifícalo una vez] +El \texttt{setup-logging.sh} instala \textbf{Vector} por defecto (etiqueta +\texttt{job="htracker"}). Si en su lugar se usó el \emph{onboarding de Linux} de Grafana +Cloud, el agente es \textbf{Alloy} y los logs aparecen por unidad de systemd +(\texttt{\{unit="htracker.service"\}}). Si una consulta no devuelve nada, usa el +\textbf{Label browser} de Grafana para ver qué etiquetas existen. +\end{helgennotice} + +\section{Acceso a Grafana} +\subsection{Grafana en la nube (nada que instalar)} +\begin{enumerate}[leftmargin=1.5em] + \item Entra a tu \textbf{stack de Grafana Cloud} (URL del tipo \texttt{https://.grafana.net}). + \item Menú lateral $\rightarrow$ \textbf{Explore}. + \item Arriba, selecciona la fuente de datos \textbf{Loki}. + \item Escribe la consulta (sección 4) y presiona \textbf{Run query}. Ajusta el rango de + tiempo arriba a la derecha. +\end{enumerate} + +\subsection{Grafana local (si prefieres correr la interfaz tú mismo)} +Los datos siguen en Grafana Cloud; sólo el visor es local. +\begin{verbatim} +docker run -d --name grafana -p 3000:3000 grafana/grafana-oss +# http://localhost:3000 (admin/admin) +# Connections -> Data sources -> Loki: +# URL: Auth: Basic user: password: +\end{verbatim} + +\begin{helgennotice}[Credenciales] +\texttt{LOKI\_URL}, \texttt{LOKI\_USER} y \texttt{LOKI\_TOKEN} están en +\texttt{/etc/vector/htracker.env} del servidor y en la consola de Grafana Cloud +(\emph{Connections} $\rightarrow$ \emph{Hosted logs}). \textbf{Son secretos} — viven en +\texttt{.env.vector} (ignorado por git); nunca se suben al repo. +\end{helgennotice} + +\section{Cómo consultar los logs (LogQL)} +En \textbf{Explore} con la fuente \textbf{Loki}. Loki indexa \textbf{etiquetas}, no el texto; +todo lo demás se filtra con \texttt{|=} (contiene) en el momento de la consulta. + +\begin{verbatim} +{job="htracker"} # todo el backend +{job="htracker", level="error"} # sólo errores +{job="htracker", unit="htracker.service"} # sólo el gateway (retranslator) +{job="htracker"} |= "Anti-Salto" # rechazos del filtro anti-salto +{job="htracker"} |= "victimas_backend" # el resumen 1/min (victimas de backend) +{job="htracker"} |= "ADDR-CHANGE" # cambios de direccion SIM/NAT +\end{verbatim} + +\noindent Si el envío es por \textbf{Alloy} (no Vector), usa la unidad de systemd: +\begin{verbatim} +{unit="htracker.service"} +{unit="htracker.service"} |= "victimas_backend" +\end{verbatim} + +\section{Qué se registra} +\begin{center}\footnotesize\renewcommand{\arraystretch}{1.3} +\begin{tabularx}{\linewidth}{@{}>{\bfseries\color{HelgenIndigo}}L{3.6cm} X@{}} +\toprule +Etiquetas (labels) & \texttt{job=htracker}, \texttt{host}, \texttt{unit} + (\texttt{htracker.service} / \texttt{-web} / \texttt{-notifier}), \texttt{level} + (\texttt{info}/\texttt{warn}/\texttt{error}/\texttt{debug})\\ +Nivel por defecto & \texttt{INFO}: sólo \textbf{eventos} + una línea de \textbf{resumen cada + 60 s} (\texttt{[SUMMARY]} / \texttt{victimas\_backend}). El detalle por paquete es + \texttt{DEBUG} y \textbf{no} se envía.\\ +Eventos clave & anti-salto (\texttt{Anti-Salto}), cambios de dirección SIM/NAT + (\texttt{ADDR-CHANGE}), víctimas de backend (unidad en línea en htracker pero ausente en + Wialon).\\ +\bottomrule +\end{tabularx} +\end{center} + +\section{Retención, límites y seguridad de disco} +\begin{itemize} + \item \textbf{Loki (capa gratuita):} 50 GB/mes, \textbf{retención de 14 días}, Grafana + incluido. Tras el recorte de verbosidad se usa una fracción mínima. + \item \textbf{journald (servidor):} \texttt{SystemMaxUse=200M}, retención 3 días + (\texttt{/etc/systemd/journald.conf.d/htracker.conf}). Aunque Loki esté caído días, + el disco \textbf{no} se llena. + \item \textbf{Búfer de Vector:} en memoria, descarta lo más nuevo si Loki no responde. +\end{itemize} + +\section{Verbosidad (nivel de log)} +\begin{itemize} + \item \texttt{GW\_LOG\_LEVEL} (por defecto \texttt{INFO}). Para depurar una unidad en vivo, + ponlo en \texttt{DEBUG} \emph{temporalmente}. + \item \texttt{GW\_LOG\_SUMMARY\_S} (por defecto 60) — cadencia del resumen. +\end{itemize} +\begin{helgennotice}[No dejes DEBUG encendido] +\texttt{DEBUG} registra cada paquete e inunda la tubería (y consume la cuota de Loki). +Úsalo unos minutos y regresa a \texttt{INFO}. +\end{helgennotice} + +\section{Salud del envío y diagnóstico} +En el servidor (\texttt{ssh HelgenServer}): +\begin{verbatim} +systemctl status vector # esta activo el agente? +journalctl -u vector -f # esta enviando? errores de Loki? +\end{verbatim} +\noindent \textbf{No aparecen logs en Grafana:} (1) confirma \texttt{vector} activo; (2) revisa +credenciales/endpoint en \texttt{/etc/vector/htracker.env}; (3) verifica el rango de tiempo en +Grafana; (4) revisa el \textbf{Label browser} (¿\texttt{job} o \texttt{unit}?, Vector vs Alloy). + +\section{Puesta en marcha / reinstalación} +Sólo si hay que reinstalar el envío (normalmente ya está activo). En el servidor, desde +\texttt{/home/itadmin/htracker/log-shipping}: +\begin{verbatim} +cp .env.vector.example .env.vector +nano .env.vector # pega LOKI_URL / LOKI_USER / LOKI_TOKEN +./setup-logging.sh # limita journald, --no-filelog, instala Vector, arranca +# ./setup-logging.sh --journald-only # si Grafana Alloy ya envia el journal +\end{verbatim} +\noindent \texttt{deploy.sh} ya sincroniza la carpeta \texttt{log-shipping/} al servidor. +Detalle completo: \texttt{04-Source-Code/htracker/log-shipping/README.md}. + +\section{Referencia rápida de consultas} +\begin{center}\footnotesize\renewcommand{\arraystretch}{1.3} +\begin{tabularx}{\linewidth}{@{}>{\ttfamily\color{HelgenIndigo}}L{6.3cm} X@{}} +\toprule +\normalfont\bfseries Consulta & \normalfont\bfseries Qué muestra\\ +\midrule +\{job="htracker"\} & Todo el backend\\ +\{job="htracker", level="error"\} & Sólo errores\\ +\{job="htracker", unit="htracker.service"\} & Sólo el gateway (retranslator)\\ +\{job="htracker"\} |= "victimas\_backend" & Resumen 1/min — víctimas de backend\\ +\{job="htracker"\} |= "Anti-Salto" & Rechazos del filtro anti-salto\\ +\{job="htracker"\} |= "ADDR-CHANGE" & Cambios de dirección SIM/NAT\\ +\bottomrule +\end{tabularx} +\end{center} + +\vfill +\begin{center}\scriptsize\color{gray} +Helgen Technologies · Project Drogheda · Observabilidad del backend · Documento interno · v1.0 (22 jun 2026) +\end{center} + +\end{document} diff --git a/05-Analytics-Reports/estilo-helgen.tex b/05-Analytics-Reports/estilo-helgen.tex new file mode 100644 index 0000000..328e489 --- /dev/null +++ b/05-Analytics-Reports/estilo-helgen.tex @@ -0,0 +1,117 @@ +% ---------- Estilo de marca Helgen (compartido, Project Drogheda) ---------- +% Reutiliza la identidad de Project Ennis (_estilo-helgen.tex). Cada documento define +% \sistema, \sistemapdf y \doctipo antes de \input{...}. Compilar con XeLaTeX. +% ---------- Geometría y base ---------- +\usepackage[letterpaper, margin=2.4cm, headheight=14pt]{geometry} +\usepackage{polyglossia} +\setmainlanguage[variant=mexican]{spanish} +\usepackage{xcolor} +\usepackage{colortbl} +\usepackage{graphicx} +\usepackage{booktabs} +\usepackage{array} +\usepackage{tabularx} +\usepackage{enumitem} +\usepackage{titlesec} +\usepackage{float} +\usepackage{fancyhdr} +\usepackage{tikz} +\usetikzlibrary{arrows.meta, positioning, shapes.misc, calc, fit, automata} +\usepackage{caption} +\captionsetup{font=small, labelfont={bf,color=HelgenIndigo}} +\usepackage{microtype} +\setlength{\emergencystretch}{3em}\tolerance=2000\hbadness=2000 +\usepackage{hyperref} +\usepackage{textcomp} +\usepackage{ragged2e} +\usepackage{xltabular} +\usepackage[export]{adjustbox} +\usepackage[most]{tcolorbox} + +\providecommand{\sistema}{Helgen Drogheda} +\providecommand{\sistemapdf}{Helgen Drogheda} +\providecommand{\doctipo}{Documento} + +% ---------- Paleta oficial Helgen ---------- +\definecolor{HelgenIndigo}{HTML}{2C3C5B} +\definecolor{HelgenCyan}{HTML}{51F3DA} +\definecolor{HelgenTeal}{HTML}{17A08C} +\definecolor{HelgenMist}{HTML}{EFF1F5} +\definecolor{HelgenLightCyan}{HTML}{E7FCF8} +\definecolor{HelgenLightIndigo}{HTML}{EDEFF4} + +\newtcolorbox{helgennotice}[1][Aviso]{enhanced,breakable,colback=HelgenLightCyan,colframe=HelgenIndigo,boxrule=0.6pt,arc=2pt,left=9pt,right=9pt,top=7pt,bottom=7pt,coltitle=white,colbacktitle=HelgenIndigo,fonttitle=\helgentitle\bfseries\small,title={#1}} +\newcommand{\hwtab}[1]{\begin{center}\footnotesize\renewcommand{\arraystretch}{1.25}#1\end{center}} + +\hypersetup{colorlinks=true, linkcolor=HelgenTeal, urlcolor=HelgenTeal, pdftitle={\sistemapdf{} - \doctipo}, pdfauthor={Dr. Leopoldo Rodríguez Salazar (Helgen Technologies)}} + +% ---------- Tipografía de marca (XeLaTeX) ---------- +\usepackage{iftex} +\ifXeTeX + \usepackage{fontspec} + \IfFontExistsTF{MADE Evolve Sans}{\setmainfont{MADE Evolve Sans}[AutoFakeSlant=0.18]}{\setmainfont{Latin Modern Sans}[AutoFakeSlant=0.18]} + \IfFontExistsTF{League Spartan}{\newfontfamily\helgentitle{League Spartan}[BoldFont={League Spartan SemiBold}]}{\newcommand{\helgentitle}{\sffamily\bfseries}} +\else + \usepackage[T1]{fontenc} + \renewcommand{\familydefault}{\sfdefault} + \newcommand{\helgentitle}{\sffamily\bfseries} +\fi + +% ---------- Color de texto global ---------- +\color{HelgenIndigo} + +% ---------- Títulos de sección ---------- +\titleformat{\section}{\helgentitle\Large\bfseries\color{HelgenIndigo}}{\thesection}{0.8em}{}[{\color{HelgenCyan}\titlerule[1.6pt]}] +\titleformat{\subsection}{\helgentitle\large\bfseries\color{HelgenIndigo}}{\thesubsection}{0.8em}{} +\titleformat{\subsubsection}{\helgentitle\normalsize\bfseries\color{HelgenIndigo}}{\thesubsubsection}{0.8em}{} +\titlespacing{\section}{0pt}{18pt}{10pt} + +% ---------- Encabezado / pie ---------- +\pagestyle{fancy} +\fancyhf{} +\fancyhead[L]{\helgentitle\small\color{HelgenIndigo}\textbf{\sistema} — \doctipo} +\fancyhead[R]{\helgentitle\small\color{HelgenIndigo}Hutchison Ports · ICAVE} +\fancyfoot[L]{\helgentitle\scriptsize\color{gray}Confidencial — Helgen Servicios Especializados S.A. de C.V.} +\fancyfoot[R]{\helgentitle\scriptsize\color{HelgenIndigo}\thepage} +\renewcommand{\headrulewidth}{0.4pt} +\renewcommand{\headrule}{\color{HelgenCyan}\hrule width\headwidth height 1.2pt} + +% ---------- Estilos TikZ Helgen ---------- +\tikzset{ + caja/.style={draw=HelgenIndigo, fill=HelgenMist, rounded corners=2pt, text=HelgenIndigo, + align=center, font=\small, inner sep=5pt, minimum height=9mm}, + cajaoscura/.style={draw=HelgenIndigo, fill=HelgenIndigo, rounded corners=2pt, text=white, + align=center, font=\small\bfseries, inner sep=5pt, minimum height=9mm}, + zona/.style={draw=HelgenIndigo!50, dashed, rounded corners=4pt, inner sep=8pt}, + etiqueta/.style={font=\scriptsize\bfseries, text=HelgenTeal}, + flecha/.style={-{Stealth[length=2.6mm]}, thick, draw=HelgenIndigo}, + flechaacento/.style={-{Stealth[length=2.6mm]}, very thick, draw=HelgenTeal}, +} + +\newcolumntype{L}[1]{>{\raggedright\arraybackslash}p{#1}} +\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}} + +% ---------- Listas con viñeta de marca ---------- +\setlist[itemize]{label={\textcolor{HelgenTeal}{\rule[0.18ex]{0.5em}{0.5em}}}, leftmargin=1.4em, itemsep=2pt} + +% ---------- Portada de marca (comando reutilizable) ---------- +% Uso: \helgenportada{Título}{Subtítulo cian}{Bloque de metadatos (tabular)} +\newcommand{\helgenportada}[3]{% +\begin{titlepage} +\begin{tikzpicture}[remember picture, overlay] + \fill[HelgenIndigo] (current page.south west) rectangle (current page.north east); + \draw[HelgenCyan, line width=2.5pt] ([xshift=2.4cm, yshift=-8.1cm]current page.north west) -- ([xshift=-2.4cm, yshift=-8.1cm]current page.north east); + \node[anchor=north west, text=white] at ([xshift=2.4cm, yshift=-3.2cm]current page.north west) + {\helgentitle\fontsize{30}{34}\selectfont Helgen}; + \node[anchor=north west, text=HelgenCyan] at ([xshift=2.4cm, yshift=-4.4cm]current page.north west) + {\helgentitle\small Ingeniería que produce, mantenimiento que protege}; + \node[anchor=north west, text=white, text width=16.5cm] at ([xshift=2.4cm, yshift=-9.0cm]current page.north west) + {\helgentitle\fontsize{24}{30}\selectfont #1\\[6pt] + \helgentitle\fontsize{14}{19}\selectfont\color{HelgenCyan} #2}; + \node[anchor=north west, text=white, text width=16.5cm] at ([xshift=2.4cm, yshift=-12.6cm]current page.north west) + {\helgentitle\fontsize{18}{22}\selectfont \sistema\\[5pt] + \normalsize Fleet RTK GPS — Hutchison Ports (ICAVE · TIMSA · EIT · LCT)}; + \node[anchor=south west, text=white] at ([xshift=2.4cm, yshift=2.2cm]current page.south west) + {\footnotesize\helgentitle #3}; +\end{tikzpicture} +\end{titlepage}} diff --git a/06-Testing-Validation/2026-06-20_Drogheda_GNSS-Findings-Summary.md b/06-Testing-Validation/2026-06-20_Drogheda_GNSS-Findings-Summary.md index 41565e8..8c9c915 100644 --- a/06-Testing-Validation/2026-06-20_Drogheda_GNSS-Findings-Summary.md +++ b/06-Testing-Validation/2026-06-20_Drogheda_GNSS-Findings-Summary.md @@ -41,6 +41,30 @@ tags: [drogheda, hbox, gnss, rtk, ntrip, findings, presentation] - So **"healthy delivery" ≠ "RTK."** The baseline is *healthy SPS + reliable delivery*; raise the bar to a fix-≥4 capture once corrections work. +## Update — 22 Jun 2026: live field capture + two ceilings on RTK + +> A fresh live `udp/5010` server capture (not the 20-Jun bench) caught two units at once. + +- **Telemetry forwarding CONFIRMED.** Enabling `$PUBX` (u-blox *proprietary* `$P…`, the + same class as our planned `$PHLG`) on the F9P **UART2** → it reaches the server **intact**, + next to GGA/RMC. The 4G NTRIP Master is a **transparent forwarder**, so **in-band unit + health (`$PHLG`) to the server is viable** — proven with zero extra hardware. +- **A natural A/B on RTK** (same capture): + - **H0014 (Querétaro bench)** — RSSI **−93**, **SPS (quality 1)**. It sits in a **basement**. + - **H0017 (Veracruz / ICAVE field)** — RSSI **−51**, **DGPS (quality 2, RMC mode `D`, + base id 0133)** — i.e. it **is** applying corrections. +- **Two independent ceilings on RTK** (refines §2's "RTK not achieved"): + 1. **Sky / antenna placement** — RTK fix is carrier-phase; a **basement caps it** no matter + how good the corrections are. (Per Leo: bench antenna placement — not fixable there.) + 2. **Corrections pipeline** — H0017 has good sky + signal yet only reaches **DGPS, not RTK + (4/5)**, pointing back at the **free rtk2go caster + RAM-only F9P config** as a *separate* + limit present everywhere. +- **Cleanest next test:** one **open-sky capture** (roof/outdoors) with a **proper caster + + FLASH-saved config** — that isolates the two ceilings and shows whether a fix ≥4 is reachable. +- **Watchdog implication:** a healthy unit can legitimately sit at SPS/DGPS (basement, or field + on a weak caster) → **never power-cycle on no-RTK alone** (already enforced by the v2.1 + SPS-thrash guard; the v2.2 `$PHLG` will make this visible per-unit). + ## 4 · The diagnostic trap (and a retraction) - The **4G-NTRIP-Master console is an unreliable health signal.** The provably-healthy diff --git a/CLAUDE.md b/CLAUDE.md index ea9afc6..b652d71 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -130,9 +130,9 @@ these before presenting; if it breaks one, say so and propose the compliant fix. | `00-Project-Management` | plan, ethos, hackathon, **financials**, **handover**, client comms, Drive tooling | [[00-Project-Management/PLAN\|PLAN]] · [[00-Project-Management/Engineering-Handover\|Engineering-Handover]] · `ethos.md` · `hackathon-plan.md` · `Financials.md` · `Drive-Upload/` · `client-comms/` | | `01-Requirements` | what we build | `SRS.md` · `HBOX-product-spec.md` | | `02-Architecture-Software` | backend/software analysis, ADRs | `backend-review.md` | -| `03-Architecture-Hardware` | HBOX/cabinet, wiring, BOM, **watchdog PCB v3** | `hardware-respec-and-pcb-brief.md` · `cabinet-electrical-bom.md` · `watchdog-schematic.md` · `watchdog-health-telemetry.md` · `firmware-review.md` · `BOM-master.md` · **`watchdog-pcb-v3-design.md`** (schematic spec: +2 fans, +I²C temp/RH) · `watchdog-pcb-v3-bom.xlsx` (proto+production, LCSC) · `watchdog-pcb-v3-kicad-guide.md` (KiCad→JLCPCB) · `watchdog-pcb-v3-blockdiagram.svg` | +| `03-Architecture-Hardware` | HBOX/cabinet, wiring, BOM, **watchdog PCB v3** | `hardware-respec-and-pcb-brief.md` · `cabinet-electrical-bom.md` · `watchdog-schematic.md` · `watchdog-health-telemetry.md` · `firmware-review.md` · `BOM-master.md` · **`watchdog-pcb-v3-design.md`** (schematic spec: +1 fan, +I²C temp/RH) · `watchdog-pcb-v3-bom.xlsx` (proto+production, LCSC) · `watchdog-pcb-v3-kicad-guide.md` (KiCad→JLCPCB) · `watchdog-pcb-v3-blockdiagram.svg` · `watchdog-protoboard-bringup.md` (shop list) · `watchdog-connections.xlsx` (pin/net connection list, PCB+proto) | | `04-Source-Code` | firmware + backend repos | `firmware/` (**own git repo `hbox-firmware`**: `blackpillrtkcode_v2/` deploy · `legacy/` v1 · `docs/LOGGING-SPEC.md` v2.1) · `htracker/` · `notifications/` · `rtk-config/` | -| `05-Analytics-Reports` | SQL views, dashboards (VERONICA/PowerBI) | _(placeholder)_ | +| `05-Analytics-Reports` | SQL views, dashboards (VERONICA/PowerBI), **observability** | `2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.pdf` (Grafana/Loki log-consultation manual) | | `06-Testing-Validation` | RCA, diagnostics, test evidence, **log analysis & tooling** | `RCA-2026-06-field-failures.md` · `remote-diagnostic-protocol.md` · `2026-06-20_Drogheda_Bench-Log-Analysis_v1.0.md` (rev-3) · `2026-06-20_Drogheda_GNSS-Findings-Summary.md` · `log-analysis/` (repo **`hbox-log-tools`**: `log-explorer.html` · `analyze_logs.py` · `build_dashboard.py` · `docs/manual.pdf`) | | `Marketing` | product-facing collateral (LaTeX→PDF) | `HBOX-Datasheet/` (industrial spec sheet) · `HBOX-Brochure-RTK/` (flyer) | | `99-Archive` | superseded raw inputs (read-only) | `situation_ORIGINAL.md` | @@ -185,6 +185,12 @@ Montacargas 12 V. **Target:** 30 units by August. `Project-Drogheda/` once synced. - **Backend repo:** `ssh://git.helgen.dev:6611/htracker` (retranslator + monitor), cloned at `04-Source-Code/htracker` (branch `feature/test500unidades`). +- **Observability (logs):** backend logs ship off-box to **Grafana Loki** (free tier, 50 GB/mo, + 14-day) via **Vector** (default) or **Grafana Alloy** reading journald (capped 200 MB) — + `04-Source-Code/htracker/log-shipping/` (`README.md` runbook · `setup-logging.sh` · `vector.toml`). + Read in Grafana → Explore → Loki: `{job="htracker"}` (Vector) or `{unit="htracker.service"}` + (Alloy). Consultation manual (PDF): `05-Analytics-Reports/2026-06-22_Drogheda_Manual-Logs-Grafana_v1.0.pdf`. + Creds in `/etc/vector/htracker.env` (server) — never in the repo. - **Firmware repo:** `ssh://git.helgen.dev:6611/hbox-firmware` (STM32F411 watchdog), lives at `04-Source-Code/firmware` (branch `main`) — its own git repo with README, `docs/PINOUT.md`, redacted RTK-config template, CHANGELOG. Push instructions in its README.