Helgen-Services/Drogheda
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2a7e5511c0
Drogheda/00-Project-Management/Engineering-Handover.md

22 KiB
Raw Blame History

tags project author date status audience
project-drogheda
handover
onboarding
rca
architecture
Project Drogheda Helgen CTO 2026-06-20 Active Helgen engineering (new + existing)

Project Drogheda — Engineering Handover & Deep Context

Read this if you're new to Drogheda, or returning after time away. It is the human-readable brain-dump: what the product is, why it failed in the field, what we are fixing, how the repo is laid out, and what's still open. The terse machine index is CLAUDE.md; current priorities are PLAN.md. When this doc and CLAUDE.md disagree, fix both — they should agree.

1. TL;DR (60 seconds)

HBOX is Helgen's GPS-RTK tracker for heavy port equipment. We deployed a first batch on Hutchison Ports ICAVE machines in Veracruz. They worked "ish," then failed. We are mid-crisis and hardening the product for scale — target 30 units operational by August, hundreds of cabinets planned across the Hutchison terminals (ICAVE, TIMSA, EIT, LCT).

The failure is not one bug. Three independent fronts, fixed in parallel:

  1. Backend (confirmed in code) — the production retranslator.py blocks the whole fleet on any upstream slowness (head-of-line), and an anti-jump filter can permanently black-hole a healthy unit. Some "dead" units are backend victims.
  2. Thermal (component-level) — an ASA HBOX (UV-stable, softens ~95–100 °C — the case is fine) sealed inside a sealed ABS IP67 cabinet that also holds the heat-generating Mean Well PSU, in full Veracruz sun → a nested heat trap for the battery / boost / modem.
  3. Prototype hardware — Dupont jumpers, hand-wiring, 18650 + PowerBoost. Fragile under continuous machine vibration. The client (ICAVE) called this out themselves.

Current physical reality: all units pulled from machines. 4 in hand (3 dead, 1 working-ish), physically in Veracruz; the team is in Querétaro → we debug remotely. Units run firmware v1 (no thermal log), so we cannot yet prove the thermal story from logs — hence the diagnostic protocol and v2 firmware.

Free triage you can do today: a unit that is ONLINE in htracker but OFFLINE in Wialon is a backend victim, not dead hardware. Check that before condemning a radio.

2. What the product actually is

Two-component RTK system:

  • Rover (HBOX) — rides on the machine (RTG crane, reach stacker, empty handler, tractoplana, forklift). Computes cm-class position.
  • Base Station — fixed reference (Ardusimple AS-PRO-CORS) that publishes RTCM3 corrections over NTRIP. One per terminal. Comes in 110 V and 230 V flavors.

The rover gets corrections over the 4G modem (NTRIP), not over the GNSS. This single fact drives the whole recovery design: if the link dies, you power-cycle the 4G stack — a GNSS reset never fixes lost NTRIP.

HBOX internals (per unit, 24 V production variant): simpleRTK2B Pro (u-blox ZED-F9P, L1/L2/L5/L6) · 4G NTRIP Master (XBee LTE) · STM32F411 Blackpill watchdog · Mean Well SD-15B-05 (24→5 V, 3 A) · PowerBoost 1000C + 18650 1S4P Li-ion backup · DFR1015 (ignition conditioning) · 2N3904 + IRF9540N + 2N7000 (watchdog power gate) · GX16-5 connector (HBOX ↔ cabinet) · U-blox multiband GNSS antenna + 2× LTE antennas · Telcel micro-SIM · ASA case.

3. Corrected system architecture

This is the corrected picture (the original BOM hid the cabinet/HBOX split). The cabinet does the power conditioning and surge/ignition isolation; the HBOX receives clean 24 V and makes its own rails.

 Machine power+ignition (12/24/48/110/230 V, per equipment)
   │
   ▼  ┌──────────────── Gabinete IP67 (ABS, Yetlebox P3901) ─────────────────┐
 breaker│ Phoenix SPD · fuse holder · DIN rail · SSR/opto (ignition isolation)│
 + SPD ►│ Mean Well DDR-60G-24 ──► regulated 24 Vdc ───┐                       │
        │                                              │  GX16-5 connector     │
        │   ┌────────── HBOX (3D-printed ASA) ─────────┼────┐                  │
        │   │ Mean Well SD-15B-05 (18–36V→5V,3A) ──► 5V rail │                 │
        │   │   ├► PowerBoost 1000C (+18650 Li-ion 1S4P) backup                │
        │   │   ├► STM32F411 Blackpill WATCHDOG ──UART(PA2)──► simpleRTK2B Pro  │
        │   │   │     └ switches RTK 5V via 2N3904→IRF9540N (PA1)              │
        │   │   ├► simpleRTK2B Pro (ZED-F9P) + 4G NTRIP Master (XBee LTE)      │
        │   │   └► ignition 24V ──► DFR1015 buck ──► STM32 PA5 (ign sense)     │
        │   └──────────────────────────────────────────────────────────────┘ │
        └───────────────────────────────────────────────────────────────────┘
            4G/LTE ▲ RTCM3 in (NTRIP)        │ NMEA out (UDP)
   Base station (Ardusimple AS-PRO-CORS) ────┘        ▼
            retranslator.py ─(anti-jump+speed filter)─► Wialon / Traccar ─► VERONICA

ICAVE machine voltages: RTG 24 V · Reach Stacker 24 V · Empty Handler 24 V · Tractoplana 12 V · Montacargas 12 V. 24 V is the production variant; the others are theoretical until built. Base station is 110/230 V.

4. Root-cause analysis (the three fronts, in depth)

Full write-up: 06-Testing-Validation/RCA-2026-06-field-failures.md. Branded client version (es-MX): 06-Testing-Validation/RCA-Informe-Falla/ → PDF.

4.1 Backend (confirmed in htracker code — not a hypothesis)

The retranslator is the funnel every unit's NMEA passes through on its way to Wialon. Three defects, all in 02-Architecture-Software/backend-review.md:

  • Head-of-line blocking. Single-threaded/synchronous forward: one slow upstream (Wialon/Traccar) stalls the entire fleet, not just one unit. At 500 units this is fatal. Fix → async retranslator, per-unit isolation; load-test with a deliberately slow upstream.
  • Anti-jump filter black-holes healthy units. On a rejected "jump," the reference position is not updated and the lockout persists across reconnect → a healthy unit can be silently excluded from Wialon forever. Fix → time/speed-based threshold, re-anchor after N rejects, clear last_pos on TTL.
  • Identity by source IP breaks under CGNAT. Telcel CGNAT means source IP is not a stable unit identity → units collide/swap. Fix → identity by cell_id (provisioned per unit), never source IP.
  • Live secret. The MSSQL sa password is committed in the repo (.env / .htracker_env). Rotate it and purge from git history. This is invariant #6.

4.2 Thermal (component-level; not yet log-proven because v1 has no sensor)

  • The ASA case is not the weak material. ASA is UV-stable and softens at ~95–100 °C, with margin over the cabinet interior. (Corrects an earlier PLA assumption — the case is fine.)
  • Nested heat trap. The ASA HBOX is sealed inside a sealed ABS IP67 cabinet that also holds the Mean Well (a heat source), under direct sun. Cabinet interior can run far above ambient. Design to measured cabinet-interior temperature, not spec-sheet ambient.
  • The GNSS is not the casualty either. ZED-F9P is rated −40…+85 °C. The heat victims are the 18650/PowerBoost (charge ≤45 °C; swells/degrades hot) and the 4G modem.
  • A bare fan inside a sealed box does not cool it — it just stirs hot air. Conduct heat out (metal plate / cabinet wall), or use an engineered waterproof duct + Blackpill- controlled fan (v3.0, preserves IP). Fixes: keep the ASA case (optional metal for extra heatsinking), thermal path to a metal plate, resolve the battery (drop / LiFePO4 / supercap), reduce solar load (reflective cabinet / sunshade / relocate).

4.3 Prototype hardware

Dupont + hand-wiring + 18650 + PowerBoost survive a bench, not a machine. Continuous vibration walks connectors loose. Fix = ruggedize now (retained connectors, strain relief, no Dupont) + a production PCB (Series 2) that integrates the watchdog gate, power, and ignition conditioning. Brief: 03-Architecture-Hardware/hardware-respec-and-pcb-brief.md.

5. The fixes (what "done" looks like)

5.1 Firmware v2 — 04-Source-Code/firmware/ (own git repo hbox-firmware)

The firmware now lives in its own git repo (ssh://git.helgen.dev:6611/hbox-firmware) at 04-Source-Code/firmware/, with README, docs/PINOUT.md, a redacted RTK-config template, and CHANGELOG. Canonical sketch: blackpillrtkcode_v2/blackpillrtkcode_v2.ino.

The watchdog is an STM32F411. v2 encodes the recovery doctrine:

  • Recover the link by power-cycling the 4G stack (gate via 2N3904→IRF9540N on PA1), never by resetting the GNSS.
  • Exponential backoff between recovery attempts — never an infinite fast-reboot loop.
  • LIMP mode — if corrections are lost, keep reporting standalone position rather than going silent. Never vanish, never loop.
  • NMEA-silence watchdog — detect a wedged receiver by absence of NMEA, not just power.
  • Thermal logging — v1 has no temperature log; v2 adds it so the next field batch proves the thermal story.
  • Power-off is the safe state — gate pull-up keeps the RTK stack OFF until the MCU asserts confidence.

Hard rule: bench-test firmware before any field flash. The sandbox cannot build STM32 — UBX checksums and logic were hand-verified, but a real build + bench cycle is mandatory before touching a field unit. Edit the _v2 file; v1 is reference only.

Firmware review + watchdog wiring: 03-Architecture-Hardware/firmware-review.md, 03-Architecture-Hardware/watchdog-schematic.md.

5.2 Backend — 04-Source-Code/htracker/ (branch feature/test500unidades)

Async retranslator (no HOL) · identity by cell_id · anti-jump rewrite (threshold + re-anchor + TTL) · 500-unit load test against a slow upstream · rotate + purge the MSSQL secret. Notifier (no secrets, reads $MATTERMOST_WEBHOOK_URL): 04-Source-Code/notifications/notify_mattermost.py.

5.3 Hardware / thermal (Series 2)

Keep ASA case (optional metal) · conduction path to metal · resolve battery · reduce solar load · production PCB integrating watchdog/power/ignition · retained connectors. Locked BOM + assembly doc required before the August lot.

6. Canonical invariants (never violate; check every output against these)

  1. Power-off is the safe state. RTK stack defaults OFF until the MCU asserts; loss of confidence → safe state, not best-guess.
  2. Never vanish, never loop. Keep reporting (standalone if needed); never enter an infinite fast-reboot loop.
  3. Corrections ride the 4G modem, not the GNSS — recover the link by power-cycling the stack; a GNSS reset is never a fix for lost NTRIP.
  4. No material that softens below ~80 °C (PLA-class) in the thermal path — the ASA enclosure complies. Design to measured cabinet-interior temperature.
  5. Backend reads/forwards without head-of-line blocking, and the anti-jump filter must never permanently lock out a healthy unit.
  6. No secrets in the repo — ever. (The committed MSSQL password is a live violation to remediate.)

Hard constraints: HBOX input is the cabinet's regulated 24 V + isolated ignition (don't re-implement the cabinet's surge/breaker/SSR on the HBOX) · IP67 maintained (conduction cooling only; no IP-breaking vents/fans) · bench-test before field flash · client-facing docs are Spanish, routed through Leo.

7. Repo map (Helgen English NN- standard)

Folder Holds Key files
00-Project-Management plan, ethos, hackathon, financials, this handover, client comms, Drive-upload tooling PLAN.md · ethos.md · hackathon-plan.md · Financials.md · Engineering-Handover.md · Drive-Upload/ · client-comms/
01-Requirements what we build SRS.md · HBOX-product-spec.md · branded SRS + ficha técnica (.tex→PDF) · requirements tracker (.xlsx)
02-Architecture-Software backend/software analysis backend-review.md
03-Architecture-Hardware HBOX/cabinet, wiring, BOM, PCB hardware-respec-and-pcb-brief.md · cabinet-electrical-bom.md · watchdog-schematic.md · firmware-review.md · BOM-master.md · BOM (.xlsx) · install/op manual (.tex→PDF)
04-Source-Code firmware + backend + tooling firmware/ (own repo hbox-firmware: blackpillrtkcode_v2/ deploy · legacy/ v1) · htracker/ · notifications/ · rtk-config/
05-Analytics-Reports SQL views, dashboards (VERONICA/PowerBI) (placeholder)
06-Testing-Validation RCA, diagnostics, evidence RCA-2026-06-field-failures.md · remote-diagnostic-protocol.md · branded RCA (.tex→PDF)
Marketing product-facing collateral HBOX-Datasheet/ (industrial spec sheet) · HBOX-Brochure-RTK/ (flyer)
_brand shared LaTeX brand style estilo-helgen.tex
99-Archive superseded raw inputs (read-only) situation_ORIGINAL.md · blackpillrtkcode_ORIGINAL_loose.ino
.claude/skills vendored project skill project-drogheda/

8. How to build / run

Firmware — its own git repo (hbox-firmware) at 04-Source-Code/firmware/; open blackpillrtkcode_v2/blackpillrtkcode_v2.ino in the STM32duino toolchain on a real machine (the sandbox cannot build STM32). Build/flash + push instructions are in that repo's README.md. Bench-test before any field flash.

Backend — 04-Source-Code/htracker/ (ssh://git.helgen.dev:6611/htracker, branch feature/test500unidades). See backend-review.md before changing the retranslator.

Branded documents (LaTeX) — each doc folder has main.tex + cuerpo.tex, sharing _brand/estilo-helgen.tex. Build with XeLaTeX (fonts: League Spartan titles + MADE Evolve Sans body — the latter is a personal-use license, needs a commercial license before client-facing embedded use):

cd 01-Requirements/HBOX-Ficha-Tecnica   # or any doc folder
latexmk -xelatex main.tex               # → main.pdf ; latexmk -c to clean aux

Trackers / BOM (xlsx) — built with openpyxl + Helgen brand colors; build scripts were generated in the scratchpad. The committed .xlsx are the source of truth for the data.

9. Deliverables produced this engagement (2026-06)

Branded LaTeX (customer-facing, es-MX), source in repo → PDF:

  • 01-Requirements/HBOX-Ficha-Tecnica/ — DRG-FT-001 (product spec sheet)
  • Marketing/HBOX-Datasheet/ — industrial datasheet (Mean Well-style, dense specs + full Ambiental/Térmica section + thermal operating-window chart + power/signal block diagram). Models the Canva "SPEC SHEET H-BOX". This is the brochure-as-datasheet.
  • 01-Requirements/Especificacion-Requerimientos/ — DRG-REQ-001 (SRS)
  • 03-Architecture-Hardware/Manual-Instalacion-Operacion/ — DRG-MAN-001 (field manual)
  • 06-Testing-Validation/RCA-Informe-Falla/ — DRG-RCA-001 (failure report)

Branded xlsx (Helgen colors, openpyxl):

  • 01-Requirements/Drogheda-Requirements-Tracker.xlsx — 34 reqs (FR/NFR/TR/BR), status + category summaries, open-points sheet (modeled on Project Ennis's GateLogic tracker).
  • 03-Architecture-Hardware/Drogheda-BOM-Maestro.xlsx — HBOX common + 24 V production + theoretical variants + cost summary + base station, one table per sheet.

Tooling: 00-Project-Management/Drive-Upload/drive_upload.py (+ README mapping) · 04-Source-Code/notifications/notify_mattermost.py.

10. Where things live (storage model — one home per artifact)

Home Role Holds
This repo (~/Documents/Claude/Projects/Drogheda) Working source of truth Editable .md/.tex, code, CLAUDE.md, vendored skill
Google Drive Drogheda (GPS RTK) Institutional / client-facing Finalized exports: branded PDFs, native Docs/Sheets
Second Brain vault 03 Projects/Helgen/Project-Drogheda/ Personal / thinking Distilled decisions, lessons, contacts

Flow: author in repo → export finalized deliverable to the matching Drive folder → distill the evergreen residue into the vault. A repo .tex and its Drive PDF are the same artifact in two formats — the only allowed redundancy.

Drive upload reality (so the next engineer doesn't relearn it): the Drive MCP connector can upload binaries (base64), but the bytes pass through the model as text, so it is only reliable for small files (≈≤15 KB; the Requirements Tracker and BOM .xlsx were uploaded this way and verified by exact byte-size). For the 30–50 KB PDFs, use Google Drive for Desktop (drag) or drive_upload.py (one-time OAuth, then it upserts all artifacts to the right folders). Folder→file mapping: 00-Project-Management/Drive-Upload/README.md. Never change Drive sharing/permissions — that is Leo's action.

11. Status snapshot (2026-06-20)

  • Units: all pulled from machines. 4 in hand: 3 dead, 1 working-ish, in Veracruz.
  • Team: in Querétaro → remote debug. Diagnostic protocol: 06-Testing-Validation/remote-diagnostic-protocol.md.
  • Firmware in field: v1 (no thermal log). v2 ready for bench → field once tested.
  • Commercial: productized across ICAVE/TIMSA/EIT/LCT; 30 units by August.
  • Naming: renamed Galway → Drogheda on 2026-06-20; repo migrated to Helgen English NN- layout. Legacy Spanish dirs removed locally.
  • Drive: Requirements Tracker + BOM .xlsx uploaded (verified); remaining branded PDFs pending drag/script upload.

12. Open items / next steps (priority order)

P0 (this week):

  1. Backend anti-jump hotfix — threshold + re-anchor + clear last_pos on TTL; stop black-holing healthy units. (Owner: Gibran)
  2. cell_id identity in the retranslator — kill the source-IP/CGNAT bug. (Gibran)
  3. Rotate + purge the MSSQL secret from the repo and git history. (Gibran)
  4. Bench-test firmware v2, then flash the working-ish unit; confirm 4G power-cycle recovery + thermal logging. (Leo)

P1 (hardening for August): 5. Async retranslator + 500-unit load test (slow upstream). (Gibran) 6. Thermal redesign — conduction path to metal + active waterproof-duct fan (v3.0); resolve battery, reduce solar load. (ASA case is adequate — no swap.) (Daniel) 7. Production PCB (Series 2) — integrate watchdog/power/ignition; retained connectors. (Daniel) 8. Lock BOM + assembly doc for the August lot. (Adriana)

Housekeeping: 9. Delete the legacy English folders + stray Google Docs in the old Drogheda Drive (Leo to do). 10. Finish mirroring branded PDFs to Drive (drag / drive_upload.py). 11. Replace datasheet "validar en banco" power figures with measured values once available.

13. People

Name Email Role
Leo Rodriguez leo@helgen.tech / ceo@helgen.tech Helgen — founder/CEO; routes client docs
Adriana Del Angel adriana@helgen.tech Helgen — implementation/plan
Paola Rodriguez paola@helgen.tech Helgen
Angel Daniel Martinez daniel@helgen.tech Helgen — hardware/thermal
Gibran Ortega gibran@helgen.tech Helgen — GPS/telemetry/backend
Agustín Alfonseca alfonseca.agustin@icave.com.mx Client — ICAVE Tech Research Supervisor
Leon David Piña pina.david@hutchisonports.com.mx Hutchison Ports

14. Glossary (don't relearn these)

  • HBOX — the rover unit (the product).
  • RTK — Real-Time Kinematic; cm-class GNSS using carrier-phase + corrections.
  • NTRIP / RTCM3 — protocol / format for streaming RTK corrections over IP (here, 4G).
  • ZED-F9P — u-blox multiband RTK receiver; rated −40…+85 °C (not the heat casualty).
  • NMEA GGA quality — ≥4 = healthy (4 = RTK-fixed, 5 = RTK-float); <4 = degraded.
  • LIMP mode — keep reporting standalone position when corrections are lost.
  • HOL (head-of-line) blocking — one slow consumer stalls everyone in a shared queue.
  • Anti-jump filter — backend logic that rejects implausible position jumps (currently over-aggressive; can lock out healthy units).
  • GX16-5 — 5-pin circular connector between HBOX and cabinet.
  • cell_id — stable per-unit identity (use this, not source IP, under CGNAT).
  • Series 1 / Series 2 — prototype (ASA case, hand-wired) / production (PCB, ruggedized).
  • VERONICA — downstream platform fed from Wialon/Traccar.

15. Pointers