Helgen-Services/Drogheda
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
0dd3171ced
Drogheda/00-Project-Management/PLAN.md

12 KiB
Raw Blame History

PLAN.md — Project Drogheda URGENT Remediation

Updated: 2026-07-10 · Owner: Leo Rodriguez Situation: all HBOX units have been removed from the machines. We have 4 units in hand — 3 not working, 1 working-ish. They are physically in Veracruz; the team is in Querétaro → debugging is remote until they ship or someone on-site is our hands. Units run v1 firmware (no thermal log). Strategy: ruggedize now + PCB in parallel; near-term work pivots to remote backend analysis (doable from QRO today) + a remote diagnostic protocol (06-Testing-Validation/remote-diagnostic-protocol.md).

Current status & schedule (2026-07-10) — supersedes the dated remediation view below

Where we are. The June crisis response drove a full hardware redesign: EinarBoard 1.0 (Rev 1.1) — the production carrier that folds the watchdog, LTC4040 battery-backup, and locking I/O onto one board — is designed, audited (ERC 0 / DRC 0), and now in fabrication + PCBA at JLCPCB.

  • PCBA order #W2026070815023249 (SMT SMT026070860804) — in production. DFM/remark resolved with JLCPCB (Swee): J1/J2 left unpopulated, C50984 not purchased, edge rails removed → finished board 135.0 × 100.0 mm. Advanced-option payment (depanel + edge rail, $4.93) settled from JLC balance on 2026-07-10; awaiting dispatch estimate.
  • First lot schedule. Assembled boards land in Querétaro ~Jul 14 → final HBox assembly + validation Jul 14–17 → ship to Veracruz ~Jul 18 (client update sent by Adriana, 2026-07-10).
  • Client escalation (open). León David Piña (Hutchison) pushed back on the slip and asked whether a latest-version unit went to ICAVE for parallel testing (agreed Jul 2). It has not shipped yet — commitment: dispatch one of the 3 pre-assembled units to ICAVE by ~Jul 15 so on-site testing runs in parallel; results to follow. (Draft reply prepared for Leo, 2026-07-10.)
  • Target: 30 units operational by August across ICAVE, TIMSA, EIT, LCT.
Date Milestone Status
2026-07-10 EinarBoard PCBA advanced-option payment settled (JLC balance) ✅ done
~2026-07-15 Dispatch 1 pre-assembled unit to ICAVE for parallel testing ⬜ committed
~2026-07-14 Assembled PCBs received in Querétaro 🔄 in production
2026-07-14–17 Final HBox assembly + validation ⬜ upcoming
~2026-07-18 Ship first validated lot to Veracruz ⬜ upcoming
Aug 2026 30 units operational (ICAVE · TIMSA · EIT · LCT) ⬜ target

The P0–P2 remediation plan below (dated 2026-06-19) is retained for the still-open backend items (anti-jump hotfix, async retranslator, secret rotation) and the RCA/thermal work; hardware has since progressed from “ruggedize + PCB in parallel” to the EinarBoard production build above.

Legend: ⬜ todo · 🔄 in progress · ✅ done · 👤 owner TBD · ⛔ blocker

P0 — DO THIS WEEK (stop the bleeding, learn the truth)

Logistics — units are in Veracruz, we're in Querétaro (decide first)

  • ⬜ 👤 Ship the 4 units to Querétaro (recommended) — small parcels; remote- debugging dead electronics is slow and inconclusive vs 1–2 days for full bench access. Bring the working-ish unit too — it's the control. Confirm with ICAVE who owns the units / whether they may leave the terminal.
  • ⬜ 👤 If they can't ship: stand up remote-hands in Veracruz using 06-Testing-Validation/remote-diagnostic-protocol.md (tiered: photos+current draw → USB dashboard → remote desktop). Need a cooperative on-site tech with a bench PSU + laptop.

Diagnosis (cheapest first — could change everything)

  • ⬜ 👤 Backend triage — now the #1 lead (backend code review done, see 02-Architecture-Software/backend-review.md; two confirmed software defects can take healthy units off Wialon). Do these free checks first:
    • htracker-online vs Wialon-offline: any unit ONLINE in Helgen's monitor but OFFLINE in the client's Wialon = backend victim, not dead hardware. Check ZPM 020 + RS-SAN-001 this way before the teardown.
    • Confirm the live process (systemctl status htracker — service runs the blocking retranslator.py; run_retranslator.sh runs a different file).
    • Gateway log + netstat -su UDP drops in the death window; did all units drop within seconds (→ HOL stall) or one-by-one (→ anti-jump lockout)?
    • NTRIP caster (HelgenSRV) availability + per-SIM data balance/suspension.
  • ⬜ 👤 Pull the field logs (battery V, fix quality, reset timestamps) and correlate Wialon gaps with the ~5/10-min v1 reset cadence.
  • ⬜ 👤 Check the installed fuse rating vs measured worst-case + inrush (H5).

Client (ICAVE) — relationship is on the line

  • ✅ Units removed from the machines (all of them, not just ZPM 020 / RS-SAN-001).
  • ⬜ 👤 Reply to Agustín/ICAVE (Spanish): acknowledge each review point, share the plan, commit dates. Draft not yet written — ask Claude to draft it; Leo sends.
  • ⬜ 👤 Agree custody/shipping of the 4 pulled units with ICAVE (see Logistics).

Bench (unblocks the hardware spec)

  • ⬜ 👤 Measure the power budget — idle/1 Hz/LTE-TX-burst current + inrush at 12 V and 24 V (hardware brief §2). Picks the DC-DC and fuse.
  • ⬜ 👤 Bench-test firmware v2 on a Blackpill + simpleRTK2B stack: power-cycle recovery, NMEA-timeout, thermal read, LED patterns. Do not field-flash untested.

P1 — SHORT TERM (≈1–3 weeks): ruggedize the fleet (Phase 1)

  • ⬜ 👤 Quantify the heat: thermocouple/logger inside a representative cabinet for a full sunny shift → real interior-temp number to design against.
  • ⬜ 👤 Build a Phase-1 ruggedization kit and rework one reference unit:
    • the ASA inner box stays (UV-stable, ~95–100 °C — adequate); optionally mount the boards to the cabinet plate for better conduction
    • replace Dupont → locking JST/Molex connectors
    • replace PowerBoost 1000C → 24 V→5 V industrial buck ≥3 A (cabinet feeds 24 V)
    • verify the cabinet blade fuse (Phoenix PT 6-FSI/C) vs measured worst-case
    • resolve battery (relocate/replace/drop — see decision below)
    • vibration mounting (isolated standoffs), retain boards, torque SMAs
    • thermal: conduct heat to the metal plate, sunshade/reflective — NO internal fan in a sealed box (does nothing); keep a Gore vent for condensation
    • conformal coat; front-panel SIM access
  • ⬜ 👤 Flash firmware v2 on every unit (thermal logging + real 4G recovery).
  • ⬜ 👤 Triage the 4 pulled units (3 dead + 1 working-ish as control) via 06-Testing-Validation/remote-diagnostic-protocol.md: bench power-on with current-draw reading, LED + USB-dashboard capture, photos (swollen LiPo? discoloured boost? cracked solder? loose SMA?). Classify each against the decision tree; teardown the genuinely-dead ones in QRO. Update 06-Testing-Validation/RCA-2026-06-field-failures.md.
  • ⬜ 👤 Instrument one unit with an interior thermocouple for a full sunny shift — quantify the thermal problem with a number.
  • ⬜ 👤 Roll the validated Phase-1 build across the fleet.

Backend (parallel to the hardware work — see 02-Architecture-Software/backend-review.md)

  • ⬜ 👤 Anti-jump hotfix to the running retranslator.py: make the threshold time/speed-aware, re-anchor after N rejects (self-heal), and clear device_last_pos on TTL cleanup. Small, safe, stops permanent black-holing.
  • ⬜ 👤 Set SQL_THROTTLE_S back to its intended 300 s; bump UDP SO_RCVBUF.
  • ⬜ 👤 Rotate the MSSQL sa password (committed in README.md) and purge secrets (.env, .htracker_env) from git history.

P2 — PERMANENT FIX (≈3–8 weeks): the named incomplete tasks

⭐ Incomplete task #1 — Design the Watchdog PCB

  • ⬜ 👤 Schematic for the integrated carrier (protection front-end, ≥3 A buck, load switch/eFuse, STM32 + sense + SWD/debug, hold-up footprint, locking I/O, bulkhead SMA, front-panel/MFF2 SIM) — hardware brief §4.
  • ⬜ 👤 Layout (4-layer, industrial temp, GNSS RF kept clean of switcher noise).
  • ⬜ 👤 Fab + assemble 2–3 pilot boards; bring up with v2 firmware.
  • ⬜ 👤 Environmental test pilots (thermal soak + vibration) before fleet rollout.

⭐ Backend — migrate to the async retranslator (the real concurrency fix)

  • ⬜ 👤 Adopt retranslator_async.py (decoupled UDP→queue→workers, per-connection async) after porting in: the corrected anti-jump filter, cell_id-based identity (not source IP — CGNAT-safe), and realistic load testing.
  • ⬜ 👤 Rewrite load_test.py to measure end-to-end (packets forwarded to Wialon, ack latency, UDP drops) against a slow upstream, not just UDP sends.

⭐ Incomplete task #2 — Re-spec HBOX components for production

  • ⬜ 👤 Lock the BOM decisions (hardware brief §6): battery strategy, SIM strategy, MCU choice, enclosure (plastic+sunshade vs metal heatsink), modem.
  • ⬜ 👤 Produce a documented, repeatable production BOM + assembly doc ("apt for production" — directly answers the client).

Key decisions Leo must make (block design — see hardware brief §6)

  1. Logistics (now): ship the 4 units to QRO (recommended) vs remote-hands in Veracruz — gates how fast we can confirm the hardware root cause.
  2. Battery mode: keep (→ LiFePO4/relocated), ride-through only (→ supercap), or drop (vehicle-only)? Recommended: drop if ICAVE doesn't need parked tracking — removes the worst thermal/safety liability. Confirm with client.
  3. Enclosure: ASA inner box is adequate (UV-stable, ~95–100 °C) — no swap needed; keep ABS IP67 cabinet + sunshade + conduct heat to a metal plate, or go metal cabinet as heatsink. (A bare internal fan ≠ a fix; v3.0 uses a ducted fan.)
  4. SIM: swappable front-panel vs MFF2 eSIM.
  5. MCU: reuse F411 vs industrial-temp STM32.

🔴 Cross-cutting red item — FINANCIALS

  • ⬜ 👤 Model unit economics (BOM cost → price → margin), per-BU revenue/COGS, cashflow/billing, burn (PCB NRE + pilot), warranty reserve. Placeholder + open questions in 00-Project-Management/Financials. Detailed work happens in a separate conversation; keep it in sync here.

Deliverables in this repo (Helgen NN- layout, updated 2026-06-20)

File What
CLAUDE.md master index (inherits Helgen ethos)
00-Project-Management/ PLAN.md · ethos.md · hackathon-plan.md · Financials.md · client-comms/
01-Requirements/ SRS.md · HBOX-product-spec.md
02-Architecture-Software/backend-review.md htracker review (HOL block + anti-jump lockout)
03-Architecture-Hardware/ hardware-respec-and-pcb-brief.md · cabinet-electrical-bom.md · watchdog-schematic.md · firmware-review.md · BOM-master.md
04-Source-Code/ firmware/ (v2 deploy · v1 ref) · htracker/ · notifications/ (Mattermost)
06-Testing-Validation/ RCA-2026-06-field-failures.md · remote-diagnostic-protocol.md

Still open / not done here

  • ✅ htracker code review done → 02-Architecture-Software/backend-review.md.
  • ✅ Repo migrated to the Helgen English NN- standard; Mattermost notifier wired.
  • ⬜ Apply the anti-jump hotfix + secret rotation (P1 backend, above).
  • ⬜ Financials model (red item, above) — separate conversation.
  • Client reply (Spanish) — not drafted yet (Leo opted to hold); ask when ready.
  • Field logs not yet ingested — RCA stays DRAFT until they are.
  • 🧹 Delete empty legacy Spanish dirs locally (the mount can't): cd ~/Documents/Claude/Projects/Drogheda && rmdir 0?_* 99_Archivo docs hardware firmware repo emails backend skills 2>/dev/null