# PLAN.md — Project Drogheda URGENT Remediation

**Updated:** 2026-07-10 · **Owner:** Leo Rodriguez
**Situation:** **all HBOX units have been removed from the machines.** We have
**4 units in hand — 3 not working, 1 working-ish.** They are **physically in
Veracruz**; the team is in **Querétaro → debugging is remote** until they ship or
someone on-site is our hands. Units run **v1 firmware** (no thermal log).
Strategy: **ruggedize now + PCB in parallel**; near-term work pivots to **remote
backend analysis (doable from QRO today)** + a **remote diagnostic protocol**
(`06-Testing-Validation/remote-diagnostic-protocol.md`).

---

## Current status & schedule (2026-07-10) — supersedes the dated remediation view below

**Where we are.** The June crisis response drove a full hardware redesign:
**EinarBoard 1.0 (Rev 1.1)** — the production carrier that folds the watchdog,
LTC4040 battery-backup, and locking I/O onto one board — is **designed, audited
(ERC 0 / DRC 0), and now in fabrication + PCBA at JLCPCB.**

- **PCBA order `#W2026070815023249` (SMT `SMT026070860804`) — in production.**
  DFM/remark resolved with JLCPCB (Swee): J1/J2 left unpopulated, C50984 not
  purchased, edge rails removed → finished board **135.0 × 100.0 mm**.
  **Advanced-option payment (depanel + edge rail, $4.93) settled from JLC balance
  on 2026-07-10**; awaiting dispatch estimate.
- **First lot schedule.** Assembled boards land in **Querétaro ~Jul 14** → **final
  HBox assembly + validation Jul 14–17** → **ship to Veracruz ~Jul 18** (client
  update sent by Adriana, 2026-07-10).
- **Client escalation (open).** León David Piña (Hutchison) pushed back on the slip
  and asked whether a latest-version unit went to **ICAVE for parallel testing**
  (agreed Jul 2). It **has not shipped yet** — commitment: **dispatch one of the 3
  pre-assembled units to ICAVE by ~Jul 15** so on-site testing runs in parallel;
  results to follow. *(Draft reply prepared for Leo, 2026-07-10.)*
- **Target:** **30 units operational by August** across ICAVE, TIMSA, EIT, LCT.

| Date | Milestone | Status |
|---|---|---|
| 2026-07-10 | EinarBoard PCBA advanced-option payment settled (JLC balance) | ✅ done |
| ~2026-07-15 | Dispatch 1 pre-assembled unit to ICAVE for parallel testing | ⬜ committed |
| ~2026-07-14 | Assembled PCBs received in Querétaro | 🔄 in production |
| 2026-07-14–17 | Final HBox assembly + validation | ⬜ upcoming |
| ~2026-07-18 | Ship first validated lot to Veracruz | ⬜ upcoming |
| Aug 2026 | 30 units operational (ICAVE · TIMSA · EIT · LCT) | ⬜ target |

> The **P0–P2 remediation plan below** (dated 2026-06-19) is retained for the
> still-open **backend** items (anti-jump hotfix, async retranslator, secret
> rotation) and the RCA/thermal work; hardware has since progressed from
> “ruggedize + PCB in parallel” to the EinarBoard production build above.

---

Legend: ⬜ todo · 🔄 in progress · ✅ done · 👤 owner TBD · ⛔ blocker

---

## P0 — DO THIS WEEK (stop the bleeding, learn the truth)

### Logistics — units are in Veracruz, we're in Querétaro (decide first)
- ⬜ 👤 **Ship the 4 units to Querétaro** *(recommended)* — small parcels; remote-
  debugging dead electronics is slow and inconclusive vs 1–2 days for full bench
  access. **Bring the working-ish unit too — it's the control.** Confirm with ICAVE
  who **owns** the units / whether they may leave the terminal.
- ⬜ 👤 If they can't ship: stand up **remote-hands** in Veracruz using
  `06-Testing-Validation/remote-diagnostic-protocol.md` (tiered: photos+current draw → USB dashboard
  → remote desktop). Need a cooperative on-site tech with a bench PSU + laptop.

### Diagnosis (cheapest first — could change everything)
- ⬜ 👤 **Backend triage — now the #1 lead** (backend code review done, see
  `02-Architecture-Software/backend-review.md`; two confirmed software defects can take healthy units
  off Wialon). Do these free checks first:
  - **htracker-online vs Wialon-offline:** any unit ONLINE in Helgen's monitor but
    OFFLINE in the client's Wialon = **backend victim, not dead hardware.** Check
    ZPM 020 + RS-SAN-001 this way *before* the teardown.
  - Confirm the **live process** (`systemctl status htracker` — service runs the
    blocking `retranslator.py`; `run_retranslator.sh` runs a different file).
  - Gateway log + **`netstat -su`** UDP drops in the death window; did all units
    drop within seconds (→ HOL stall) or one-by-one (→ anti-jump lockout)?
  - NTRIP caster (HelgenSRV) availability + **per-SIM data balance/suspension**.
- ⬜ 👤 **Pull the field logs** (battery V, fix quality, reset timestamps) and
  correlate Wialon gaps with the ~5/10-min v1 reset cadence.
- ⬜ 👤 **Check the installed fuse** rating vs measured worst-case + inrush (H5).

### Client (ICAVE) — relationship is on the line
- ✅ Units **removed** from the machines (all of them, not just ZPM 020 / RS-SAN-001).
- ⬜ 👤 **Reply to Agustín/ICAVE** (Spanish): acknowledge each review point, share
  the plan, commit dates. *Draft not yet written — ask Claude to draft it; Leo sends.*
- ⬜ 👤 Agree **custody/shipping** of the 4 pulled units with ICAVE (see Logistics).

### Bench (unblocks the hardware spec)
- ⬜ 👤 **Measure the power budget** — idle/1 Hz/**LTE-TX-burst** current + inrush
  at 12 V and 24 V (`hardware brief §2`). Picks the DC-DC and fuse.
- ⬜ 👤 **Bench-test firmware v2** on a Blackpill + simpleRTK2B stack: power-cycle
  recovery, NMEA-timeout, thermal read, LED patterns. *Do not field-flash
  untested.*

---

## P1 — SHORT TERM (≈1–3 weeks): ruggedize the fleet (Phase 1)

- ⬜ 👤 **Quantify the heat:** thermocouple/logger inside a representative cabinet
  for a full sunny shift → real interior-temp number to design against.
- ⬜ 👤 Build a **Phase-1 ruggedization kit** and rework **one reference unit**:
  - **the ASA inner box stays** (UV-stable, ~95–100 °C — adequate); optionally
    mount the boards to the cabinet plate for better conduction
  - replace Dupont → locking JST/Molex connectors
  - replace PowerBoost 1000C → **24 V→5 V** industrial buck ≥3 A (cabinet feeds 24 V)
  - **verify the cabinet blade fuse** (Phoenix PT 6-FSI/C) vs measured worst-case
  - resolve battery (relocate/replace/drop — see decision below)
  - vibration mounting (isolated standoffs), retain boards, torque SMAs
  - **thermal: conduct heat to the metal plate, sunshade/reflective — NO internal
    fan in a sealed box (does nothing); keep a Gore vent for condensation**
  - conformal coat; front-panel SIM access
- ⬜ 👤 **Flash firmware v2** on every unit (thermal logging + real 4G recovery).
- ⬜ 👤 **Triage the 4 pulled units** (3 dead + 1 working-ish as control) via
  `06-Testing-Validation/remote-diagnostic-protocol.md`: bench power-on with **current-draw reading**,
  LED + USB-dashboard capture, photos (swollen LiPo? discoloured boost? cracked
  solder? loose SMA?). Classify each against the decision tree; **teardown the
  genuinely-dead ones in QRO**. Update `06-Testing-Validation/RCA-2026-06-field-failures.md`.
- ⬜ 👤 **Instrument one unit** with an interior thermocouple for a full sunny
  shift — quantify the thermal problem with a number.
- ⬜ 👤 Roll the validated Phase-1 build across the fleet.

### Backend (parallel to the hardware work — see `02-Architecture-Software/backend-review.md`)
- ⬜ 👤 **Anti-jump hotfix** to the running `retranslator.py`: make the threshold
  time/speed-aware, **re-anchor after N rejects (self-heal)**, and **clear
  `device_last_pos` on TTL cleanup**. Small, safe, stops permanent black-holing.
- ⬜ 👤 Set `SQL_THROTTLE_S` back to its intended 300 s; bump UDP `SO_RCVBUF`.
- ⬜ 👤 **Rotate the MSSQL `sa` password** (committed in `README.md`) and purge
  secrets (`.env`, `.htracker_env`) from git history.

---

## P2 — PERMANENT FIX (≈3–8 weeks): the named incomplete tasks

### ⭐ Incomplete task #1 — **Design the Watchdog PCB**
- ⬜ 👤 Schematic for the integrated carrier (protection front-end, ≥3 A buck,
  load switch/eFuse, STM32 + sense + SWD/debug, hold-up footprint, locking I/O,
  bulkhead SMA, front-panel/MFF2 SIM) — `hardware brief §4`.
- ⬜ 👤 Layout (4-layer, industrial temp, GNSS RF kept clean of switcher noise).
- ⬜ 👤 Fab + assemble **2–3 pilot boards**; bring up with v2 firmware.
- ⬜ 👤 Environmental test pilots (thermal soak + vibration) before fleet rollout.

### ⭐ Backend — migrate to the async retranslator (the real concurrency fix)
- ⬜ 👤 Adopt `retranslator_async.py` (decoupled UDP→queue→workers, per-connection
  async) **after** porting in: the corrected anti-jump filter, **cell_id-based
  identity** (not source IP — CGNAT-safe), and realistic load testing.
- ⬜ 👤 Rewrite `load_test.py` to measure **end-to-end** (packets forwarded to
  Wialon, ack latency, UDP drops) against a **slow upstream**, not just UDP sends.

### ⭐ Incomplete task #2 — **Re-spec HBOX components for production**
- ⬜ 👤 Lock the BOM decisions (`hardware brief §6`): **battery strategy**, SIM
  strategy, MCU choice, enclosure (plastic+sunshade vs metal heatsink), modem.
- ⬜ 👤 Produce a documented, repeatable production BOM + assembly doc ("apt for
  production" — directly answers the client).

---

## Key decisions Leo must make (block design — see hardware brief §6)

0. **Logistics (now):** **ship the 4 units to QRO** (recommended) vs **remote-hands
   in Veracruz** — gates how fast we can confirm the hardware root cause.
1. **Battery mode:** keep (→ LiFePO4/relocated), ride-through only (→ supercap),
   or **drop** (vehicle-only)? *Recommended: drop if ICAVE doesn't need parked
   tracking — removes the worst thermal/safety liability. Confirm with client.*
2. **Enclosure:** **ASA inner box is adequate** (UV-stable, ~95–100 °C) — no swap
   needed; keep ABS IP67 cabinet + sunshade + **conduct heat to a metal plate**, or go
   **metal cabinet as heatsink**. (A bare internal fan ≠ a fix; v3.0 uses a ducted fan.)
3. **SIM:** swappable front-panel vs **MFF2 eSIM**.
4. **MCU:** reuse F411 vs industrial-temp STM32.

---

## 🔴 Cross-cutting red item — FINANCIALS
- ⬜ 👤 Model **unit economics** (BOM cost → price → margin), per-BU revenue/COGS,
  cashflow/billing, burn (PCB NRE + pilot), warranty reserve. Placeholder +
  open questions in [[00-Project-Management/Financials|Financials.md]]. *Detailed
  work happens in a separate conversation; keep it in sync here.*

## Deliverables in this repo (Helgen `NN-` layout, updated 2026-06-20)

| File | What |
|---|---|
| `CLAUDE.md` | master index (inherits Helgen ethos) |
| `00-Project-Management/` | `PLAN.md` · `ethos.md` · `hackathon-plan.md` · `Financials.md` · `client-comms/` |
| `01-Requirements/` | `SRS.md` · `HBOX-product-spec.md` |
| `02-Architecture-Software/backend-review.md` | htracker review (HOL block + anti-jump lockout) |
| `03-Architecture-Hardware/` | `hardware-respec-and-pcb-brief.md` · `cabinet-electrical-bom.md` · `watchdog-schematic.md` · `firmware-review.md` · `BOM-master.md` |
| `04-Source-Code/` | `firmware/` (v2 deploy · v1 ref) · `htracker/` · `notifications/` (Mattermost) |
| `06-Testing-Validation/` | `RCA-2026-06-field-failures.md` · `remote-diagnostic-protocol.md` |

## Still open / not done here
- ✅ `htracker` **code review done** → `02-Architecture-Software/backend-review.md`.
- ✅ Repo migrated to the **Helgen English `NN-` standard**; Mattermost notifier wired.
- ⬜ Apply the **anti-jump hotfix** + secret rotation (P1 backend, above).
- ⬜ **Financials** model (red item, above) — separate conversation.
- **Client reply (Spanish)** — not drafted yet (Leo opted to hold); ask when ready.
- Field logs not yet ingested — RCA stays DRAFT until they are.
- 🧹 Delete empty legacy Spanish dirs locally (the mount can't): `cd ~/Documents/Claude/Projects/Drogheda && rmdir 0?_* 99_Archivo docs hardware firmware repo emails backend skills 2>/dev/null`