Drogheda/00-Project-Management/Engineering-Handover.md

386 lines
22 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
tags: [project-drogheda, handover, onboarding, rca, architecture]
project: Project Drogheda
author: Helgen CTO
date: 2026-06-20
status: Active
audience: Helgen engineering (new + existing)
---
# Project Drogheda — Engineering Handover & Deep Context
> **Read this if you're new to Drogheda, or returning after time away.** It is the
> human-readable brain-dump: what the product is, why it failed in the field, what we
> are fixing, how the repo is laid out, and what's still open. The terse machine index
> is [`CLAUDE.md`](../CLAUDE.md); current priorities are [`PLAN.md`](PLAN.md). When this
> doc and `CLAUDE.md` disagree, fix both — they should agree.
---
## 1. TL;DR (60 seconds)
**HBOX** is Helgen's GPS-**RTK** tracker for heavy port equipment. We deployed a first
batch on **Hutchison Ports ICAVE** machines in **Veracruz**. They worked "ish," then
failed. We are **mid-crisis and hardening the product for scale** — target **30 units
operational by August**, hundreds of cabinets planned across the Hutchison terminals
(**ICAVE, TIMSA, EIT, LCT**).
The failure is **not one bug**. Three independent fronts, fixed in parallel:
1. **Backend** (confirmed in code) — the production `retranslator.py` blocks the whole
fleet on any upstream slowness (head-of-line), and an anti-jump filter can
**permanently black-hole a healthy unit**. Some "dead" units are backend victims.
2. **Thermal** (component-level) — an **ASA** HBOX (UV-stable, softens ~95–100 °C — the
case is fine) sealed inside a sealed **ABS IP67** cabinet that *also* holds the
heat-generating **Mean Well** PSU, in full Veracruz sun → a nested heat trap for the
battery / boost / modem.
3. **Prototype hardware** — Dupont jumpers, hand-wiring, 18650 + PowerBoost. Fragile
under continuous machine vibration. The client (ICAVE) called this out themselves.
**Current physical reality:** all units pulled from machines. **4 in hand (3 dead, 1
working-ish)**, physically in **Veracruz**; the team is in **Querétaro** → we debug
**remotely**. Units run **firmware v1** (no thermal log), so we cannot yet *prove* the
thermal story from logs — hence the diagnostic protocol and v2 firmware.
**Free triage you can do today:** a unit that is **ONLINE in htracker but OFFLINE in
Wialon** is a **backend victim**, not dead hardware. Check that before condemning a radio.
---
## 2. What the product actually is
Two-component RTK system:
- **Rover (HBOX)** — rides on the machine (RTG crane, reach stacker, empty handler,
tractoplana, forklift). Computes cm-class position.
- **Base Station** — fixed reference (Ardusimple **AS-PRO-CORS**) that publishes **RTCM3**
corrections over **NTRIP**. One per terminal. Comes in **110 V and 230 V** flavors.
The rover gets corrections **over the 4G modem (NTRIP), not over the GNSS**. This single
fact drives the whole recovery design: **if the link dies, you power-cycle the 4G stack —
a GNSS reset never fixes lost NTRIP.**
**HBOX internals (per unit, 24 V production variant):**
simpleRTK2B **Pro** (u-blox **ZED-F9P**, L1/L2/L5/L6) · **4G NTRIP Master** (XBee LTE) ·
**STM32F411 Blackpill** watchdog · **Mean Well SD-15B-05** (24→5 V, 3 A) · **PowerBoost
1000C** + **18650 1S4P** Li-ion backup · **DFR1015** (ignition conditioning) · 2N3904 +
IRF9540N + 2N7000 (watchdog power gate) · **GX16-5** connector (HBOX ↔ cabinet) · U-blox
multiband GNSS antenna + 2× LTE antennas · Telcel micro-SIM · ASA case.
---
## 3. Corrected system architecture
This is the **corrected** picture (the original BOM hid the cabinet/HBOX split). The
cabinet does the power conditioning and surge/ignition isolation; the HBOX receives
**clean 24 V** and makes its own rails.
```text
Machine power+ignition (12/24/48/110/230 V, per equipment)
│
▼ ┌──────────────── Gabinete IP67 (ABS, Yetlebox P3901) ─────────────────┐
breaker│ Phoenix SPD · fuse holder · DIN rail · SSR/opto (ignition isolation)│
+ SPD ►│ Mean Well DDR-60G-24 ──► regulated 24 Vdc ───┐ │
│ │ GX16-5 connector │
│ ┌────────── HBOX (3D-printed ASA) ─────────┼────┐ │
│ │ Mean Well SD-15B-05 (18–36V→5V,3A) ──► 5V rail │ │
│ │ ├► PowerBoost 1000C (+18650 Li-ion 1S4P) backup │
│ │ ├► STM32F411 Blackpill WATCHDOG ──UART(PA2)──► simpleRTK2B Pro │
│ │ │ └ switches RTK 5V via 2N3904→IRF9540N (PA1) │
│ │ ├► simpleRTK2B Pro (ZED-F9P) + 4G NTRIP Master (XBee LTE) │
│ │ └► ignition 24V ──► DFR1015 buck ──► STM32 PA5 (ign sense) │
│ └──────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────┘
4G/LTE ▲ RTCM3 in (NTRIP) │ NMEA out (UDP)
Base station (Ardusimple AS-PRO-CORS) ────┘ ▼
retranslator.py ─(anti-jump+speed filter)─► Wialon / Traccar ─► VERONICA
```
**ICAVE machine voltages:** RTG 24 V · Reach Stacker 24 V · Empty Handler 24 V ·
Tractoplana 12 V · Montacargas 12 V. **24 V is the production variant**; the others are
theoretical until built. Base station is 110/230 V.
---
## 4. Root-cause analysis (the three fronts, in depth)
Full write-up: [`06-Testing-Validation/RCA-2026-06-field-failures.md`](../06-Testing-Validation/RCA-2026-06-field-failures.md).
Branded client version (es-MX): `06-Testing-Validation/RCA-Informe-Falla/` → PDF.
### 4.1 Backend (confirmed in `htracker` code — not a hypothesis)
The retranslator is the funnel every unit's NMEA passes through on its way to Wialon.
Three defects, all in [`02-Architecture-Software/backend-review.md`](../02-Architecture-Software/backend-review.md):
- **Head-of-line blocking.** Single-threaded/synchronous forward: one slow upstream
(Wialon/Traccar) stalls **the entire fleet**, not just one unit. At 500 units this is
fatal. Fix → **async retranslator**, per-unit isolation; load-test with a deliberately
slow upstream.
- **Anti-jump filter black-holes healthy units.** On a rejected "jump," the reference
position is **not updated** and the lockout **persists across reconnect** → a healthy
unit can be silently excluded from Wialon forever. Fix → time/speed-based threshold,
**re-anchor** after N rejects, **clear `last_pos` on TTL**.
- **Identity by source IP breaks under CGNAT.** Telcel CGNAT means source IP is not a
stable unit identity → units collide/swap. Fix → identity by **`cell_id`** (provisioned
per unit), never source IP.
- **Live secret.** The MSSQL `sa` password is **committed in the repo** (`.env` /
`.htracker_env`). **Rotate it and purge from git history.** This is invariant #6.
### 4.2 Thermal (component-level; not yet log-proven because v1 has no sensor)
- **The ASA case is not the weak material.** ASA is UV-stable and softens at ~95–100 °C,
with margin over the cabinet interior. (Corrects an earlier PLA assumption — the case is fine.)
- **Nested heat trap.** The ASA HBOX is sealed inside a sealed ABS IP67 cabinet that also
holds the **Mean Well** (a heat source), under direct sun. Cabinet interior can run far
above ambient. **Design to measured cabinet-interior temperature, not spec-sheet ambient.**
- **The GNSS is not the casualty either.** ZED-F9P is rated **−40…+85 °C**. The heat victims
are the **18650/PowerBoost** (charge ≤45 °C; swells/degrades hot) and the **4G modem**.
- **A bare fan inside a sealed box does not cool it** — it just stirs hot air. **Conduct heat
out** (metal plate / cabinet wall), or use an **engineered waterproof duct + Blackpill-
controlled fan** (v3.0, preserves IP). Fixes: keep the ASA case (optional metal for extra
heatsinking), thermal path to a metal plate, resolve the battery (drop / LiFePO4 / supercap),
reduce solar load (reflective cabinet / sunshade / relocate).
### 4.3 Prototype hardware
Dupont + hand-wiring + 18650 + PowerBoost survive a bench, not a machine. Continuous
vibration walks connectors loose. Fix = **ruggedize now** (retained connectors, strain
relief, no Dupont) **+ a production PCB (Series 2)** that integrates the watchdog gate,
power, and ignition conditioning. Brief: [`03-Architecture-Hardware/hardware-respec-and-pcb-brief.md`](../03-Architecture-Hardware/hardware-respec-and-pcb-brief.md).
---
## 5. The fixes (what "done" looks like)
### 5.1 Firmware v2 — `04-Source-Code/firmware/` (own git repo `hbox-firmware`)
> The firmware now lives in its **own git repo** (`ssh://git.helgen.dev:6611/hbox-firmware`)
> at `04-Source-Code/firmware/`, with README, `docs/PINOUT.md`, a redacted RTK-config
> template, and CHANGELOG. Canonical sketch: `blackpillrtkcode_v2/blackpillrtkcode_v2.ino`.
The watchdog is an **STM32F411**. v2 encodes the recovery doctrine:
- **Recover the link by power-cycling the 4G stack** (gate via 2N3904→IRF9540N on PA1),
**never** by resetting the GNSS.
- **Exponential backoff** between recovery attempts — never an infinite fast-reboot loop.
- **LIMP mode** — if corrections are lost, keep reporting **standalone** position rather
than going silent. *Never vanish, never loop.*
- **NMEA-silence watchdog** — detect a wedged receiver by absence of NMEA, not just power.
- **Thermal logging** — v1 has no temperature log; v2 adds it so the next field batch
*proves* the thermal story.
- **Power-off is the safe state** — gate pull-up keeps the RTK stack OFF until the MCU
asserts confidence.
> **Hard rule: bench-test firmware before any field flash.** The sandbox cannot build
> STM32 — UBX checksums and logic were hand-verified, but a real build + bench cycle is
> mandatory before touching a field unit. Edit the **`_v2`** file; v1 is reference only.
Firmware review + watchdog wiring: [`03-Architecture-Hardware/firmware-review.md`](../03-Architecture-Hardware/firmware-review.md),
[`03-Architecture-Hardware/watchdog-schematic.md`](../03-Architecture-Hardware/watchdog-schematic.md).
### 5.2 Backend — `04-Source-Code/htracker/` (branch `feature/test500unidades`)
Async retranslator (no HOL) · identity by `cell_id` · anti-jump rewrite (threshold +
re-anchor + TTL) · 500-unit load test against a slow upstream · **rotate + purge the MSSQL
secret**. Notifier (no secrets, reads `$MATTERMOST_WEBHOOK_URL`):
`04-Source-Code/notifications/notify_mattermost.py`.
### 5.3 Hardware / thermal (Series 2)
Keep ASA case (optional metal) · conduction path to metal · resolve battery · reduce solar
load · production PCB integrating watchdog/power/ignition · retained connectors. Locked
BOM + assembly doc required before the August lot.
---
## 6. Canonical invariants (never violate; check every output against these)
1. **Power-off is the safe state.** RTK stack defaults OFF until the MCU asserts; loss of
confidence → safe state, not best-guess.
2. **Never vanish, never loop.** Keep reporting (standalone if needed); never enter an
infinite fast-reboot loop.
3. **Corrections ride the 4G modem, not the GNSS** — recover the link by power-cycling the
stack; a GNSS reset is never a fix for lost NTRIP.
4. **No material that softens below ~80 °C (PLA-class) in the thermal path** — the ASA
enclosure complies. Design to measured cabinet-interior temperature.
5. **Backend reads/forwards without head-of-line blocking**, and the anti-jump filter must
never permanently lock out a healthy unit.
6. **No secrets in the repo — ever.** (The committed MSSQL password is a live violation to
remediate.)
**Hard constraints:** HBOX input is the cabinet's regulated 24 V + isolated ignition
(don't re-implement the cabinet's surge/breaker/SSR on the HBOX) · **IP67 maintained**
(conduction cooling only; no IP-breaking vents/fans) · **bench-test before field flash** ·
client-facing docs are **Spanish**, routed through Leo.
---
## 7. Repo map (Helgen English `NN-` standard)
| Folder | Holds | Key files |
|---|---|---|
| `00-Project-Management` | plan, ethos, hackathon, financials, **this handover**, client comms, Drive-upload tooling | `PLAN.md` · `ethos.md` · `hackathon-plan.md` · `Financials.md` · `Engineering-Handover.md` · `Drive-Upload/` · `client-comms/` |
| `01-Requirements` | what we build | `SRS.md` · `HBOX-product-spec.md` · branded SRS + ficha técnica (`.tex`→PDF) · requirements tracker (`.xlsx`) |
| `02-Architecture-Software` | backend/software analysis | `backend-review.md` |
| `03-Architecture-Hardware` | HBOX/cabinet, wiring, BOM, PCB | `hardware-respec-and-pcb-brief.md` · `cabinet-electrical-bom.md` · `watchdog-schematic.md` · `firmware-review.md` · `BOM-master.md` · BOM (`.xlsx`) · install/op manual (`.tex`→PDF) |
| `04-Source-Code` | firmware + backend + tooling | `firmware/` (**own repo `hbox-firmware`**: `blackpillrtkcode_v2/` deploy · `legacy/` v1) · `htracker/` · `notifications/` · `rtk-config/` |
| `05-Analytics-Reports` | SQL views, dashboards (VERONICA/PowerBI) | _(placeholder)_ |
| `06-Testing-Validation` | RCA, diagnostics, evidence | `RCA-2026-06-field-failures.md` · `remote-diagnostic-protocol.md` · branded RCA (`.tex`→PDF) |
| `Marketing` | product-facing collateral | `HBOX-Datasheet/` (industrial spec sheet) · `HBOX-Brochure-RTK/` (flyer) |
| `_brand` | shared LaTeX brand style | `estilo-helgen.tex` |
| `99-Archive` | superseded raw inputs (read-only) | `situation_ORIGINAL.md` · `blackpillrtkcode_ORIGINAL_loose.ino` |
| `.claude/skills` | vendored project skill | `project-drogheda/` |
---
## 8. How to build / run
**Firmware** — its own git repo (`hbox-firmware`) at `04-Source-Code/firmware/`; open
`blackpillrtkcode_v2/blackpillrtkcode_v2.ino` in the STM32duino toolchain on a real machine
(the sandbox cannot build STM32). Build/flash + push instructions are in that repo's
`README.md`. **Bench-test before any field flash.**
**Backend** — `04-Source-Code/htracker/` (`ssh://git.helgen.dev:6611/htracker`, branch
`feature/test500unidades`). See `backend-review.md` before changing the retranslator.
**Branded documents (LaTeX)** — each doc folder has `main.tex` + `cuerpo.tex`, sharing
`_brand/estilo-helgen.tex`. Build with **XeLaTeX** (fonts: **League Spartan** titles +
**MADE Evolve Sans** body — the latter is a *personal-use* license, **needs a commercial
license before client-facing embedded use**):
```bash
cd 01-Requirements/HBOX-Ficha-Tecnica # or any doc folder
latexmk -xelatex main.tex # → main.pdf ; latexmk -c to clean aux
```
**Trackers / BOM (xlsx)** — built with openpyxl + Helgen brand colors; build scripts were
generated in the scratchpad. The committed `.xlsx` are the source of truth for the data.
---
## 9. Deliverables produced this engagement (2026-06)
**Branded LaTeX (customer-facing, es-MX), source in repo → PDF:**
- `01-Requirements/HBOX-Ficha-Tecnica/` — DRG-FT-001 (product spec sheet)
- `Marketing/HBOX-Datasheet/` — **industrial datasheet** (Mean Well-style, dense specs +
full **Ambiental/Térmica** section + thermal operating-window chart + power/signal block
diagram). Models the Canva "SPEC SHEET H-BOX". **This is the brochure-as-datasheet.**
- `01-Requirements/Especificacion-Requerimientos/` — DRG-REQ-001 (SRS)
- `03-Architecture-Hardware/Manual-Instalacion-Operacion/` — DRG-MAN-001 (field manual)
- `06-Testing-Validation/RCA-Informe-Falla/` — DRG-RCA-001 (failure report)
**Branded xlsx (Helgen colors, openpyxl):**
- `01-Requirements/Drogheda-Requirements-Tracker.xlsx` — 34 reqs (FR/NFR/TR/BR), status +
category summaries, open-points sheet (modeled on Project Ennis's GateLogic tracker).
- `03-Architecture-Hardware/Drogheda-BOM-Maestro.xlsx` — HBOX common + 24 V production +
theoretical variants + cost summary + base station, one table per sheet.
**Tooling:** `00-Project-Management/Drive-Upload/drive_upload.py` (+ README mapping) ·
`04-Source-Code/notifications/notify_mattermost.py`.
---
## 10. Where things live (storage model — one home per artifact)
| Home | Role | Holds |
|---|---|---|
| **This repo** (`~/Documents/Claude/Projects/Drogheda`) | Working **source of truth** | Editable `.md`/`.tex`, code, `CLAUDE.md`, vendored skill |
| **Google Drive** `Drogheda (GPS RTK)` | Institutional / client-facing | Finalized exports: branded PDFs, native Docs/Sheets |
| **Second Brain vault** `03 Projects/Helgen/Project-Drogheda/` | Personal / thinking | Distilled decisions, lessons, contacts |
**Flow:** author in repo → export finalized deliverable to the matching Drive folder →
distill the evergreen residue into the vault. A repo `.tex` and its Drive PDF are the same
artifact in two formats — the only allowed redundancy.
**Drive upload reality (so the next engineer doesn't relearn it):** the Drive MCP
connector *can* upload binaries (base64), but the bytes pass through the model as text, so
it is **only reliable for small files** (≈≤15 KB; the Requirements Tracker and BOM `.xlsx`
were uploaded this way and verified by exact byte-size). For the 30–50 KB **PDFs**, use
**Google Drive for Desktop (drag)** or **`drive_upload.py`** (one-time OAuth, then it
upserts all artifacts to the right folders). Folder→file mapping:
`00-Project-Management/Drive-Upload/README.md`. **Never change Drive sharing/permissions —
that is Leo's action.**
---
## 11. Status snapshot (2026-06-20)
- **Units:** all pulled from machines. **4 in hand: 3 dead, 1 working-ish**, in **Veracruz**.
- **Team:** in **Querétaro** → **remote** debug. Diagnostic protocol:
`06-Testing-Validation/remote-diagnostic-protocol.md`.
- **Firmware in field:** **v1** (no thermal log). v2 ready for bench → field once tested.
- **Commercial:** productized across ICAVE/TIMSA/EIT/LCT; **30 units by August**.
- **Naming:** renamed **Galway → Drogheda** on 2026-06-20; repo migrated to Helgen English
`NN-` layout. Legacy Spanish dirs removed locally.
- **Drive:** Requirements Tracker + BOM `.xlsx` uploaded (verified); remaining branded PDFs
pending drag/script upload.
---
## 12. Open items / next steps (priority order)
**P0 (this week):**
1. **Backend anti-jump hotfix** — threshold + re-anchor + clear `last_pos` on TTL; stop
black-holing healthy units. (Owner: Gibran)
2. **`cell_id` identity** in the retranslator — kill the source-IP/CGNAT bug. (Gibran)
3. **Rotate + purge the MSSQL secret** from the repo and git history. (Gibran)
4. **Bench-test firmware v2**, then flash the working-ish unit; confirm 4G power-cycle
recovery + thermal logging. (Leo)
**P1 (hardening for August):**
5. **Async retranslator** + 500-unit load test (slow upstream). (Gibran)
6. **Thermal redesign** — conduction path to metal + active waterproof-duct fan (v3.0);
resolve battery, reduce solar load. (ASA case is adequate — no swap.) (Daniel)
7. **Production PCB (Series 2)** — integrate watchdog/power/ignition; retained connectors. (Daniel)
8. **Lock BOM + assembly doc** for the August lot. (Adriana)
**Housekeeping:**
9. Delete the legacy English folders + stray Google Docs in the **old** Drogheda Drive
(Leo to do).
10. Finish mirroring branded PDFs to Drive (drag / `drive_upload.py`).
11. Replace datasheet "validar en banco" power figures with measured values once available.
---
## 13. People
| Name | Email | Role |
|---|---|---|
| Leo Rodriguez | leo@helgen.tech / ceo@helgen.tech | Helgen — founder/CEO; routes client docs |
| Adriana Del Angel | adriana@helgen.tech | Helgen — implementation/plan |
| Paola Rodriguez | paola@helgen.tech | Helgen |
| Angel Daniel Martinez | daniel@helgen.tech | Helgen — hardware/thermal |
| Gibran Ortega | gibran@helgen.tech | Helgen — GPS/telemetry/backend |
| **Agustín Alfonseca** | alfonseca.agustin@icave.com.mx | **Client** — ICAVE Tech Research Supervisor |
| Leon David Piña | pina.david@hutchisonports.com.mx | Hutchison Ports |
---
## 14. Glossary (don't relearn these)
- **HBOX** — the rover unit (the product).
- **RTK** — Real-Time Kinematic; cm-class GNSS using carrier-phase + corrections.
- **NTRIP / RTCM3** — protocol / format for streaming RTK corrections over IP (here, 4G).
- **ZED-F9P** — u-blox multiband RTK receiver; rated **−40…+85 °C** (not the heat casualty).
- **NMEA GGA quality** — **≥4 = healthy** (4 = RTK-fixed, 5 = RTK-float); <4 = degraded.
- **LIMP mode** — keep reporting standalone position when corrections are lost.
- **HOL (head-of-line) blocking** — one slow consumer stalls everyone in a shared queue.
- **Anti-jump filter** — backend logic that rejects implausible position jumps (currently
over-aggressive; can lock out healthy units).
- **GX16-5** — 5-pin circular connector between HBOX and cabinet.
- **cell_id** — stable per-unit identity (use this, not source IP, under CGNAT).
- **Series 1 / Series 2** — prototype (ASA case, hand-wired) / production (PCB, ruggedized).
- **VERONICA** — downstream platform fed from Wialon/Traccar.
---
## 15. Pointers
- Machine index for Claude: [`CLAUDE.md`](../CLAUDE.md)
- Current priorities: [`PLAN.md`](PLAN.md) · Weekend plan: [`hackathon-plan.md`](hackathon-plan.md)
- RCA (full): [`06-Testing-Validation/RCA-2026-06-field-failures.md`](../06-Testing-Validation/RCA-2026-06-field-failures.md)
- Backend review: [`02-Architecture-Software/backend-review.md`](../02-Architecture-Software/backend-review.md)
- Remote diagnostics: [`06-Testing-Validation/remote-diagnostic-protocol.md`](../06-Testing-Validation/remote-diagnostic-protocol.md)
- Hardware re-spec + PCB: [`03-Architecture-Hardware/hardware-respec-and-pcb-brief.md`](../03-Architecture-Hardware/hardware-respec-and-pcb-brief.md)
- Drive upload mapping: [`Drive-Upload/README.md`](Drive-Upload/README.md)