Remove Google Sheets mirror entirely (#13)

Postgres is now the sole source of truth: delete sheets.py and import_from_sheets.py, strip mirror_entity/mirror_async/_mirror_worker and POST /api/sync from app.py, drop the tab/mirror keys from db.py's TABLES. Re-point n8n/renewal-reminder.json at the CRM's own HTTP API (GET /api/clients, POST /api/activity_log) instead of the Sheets nodes, and drop SHEET_ID/GOOGLE_SA_JSON from deploy env/compose and requests from requirements.txt (PyJWT stays — still used by booking_api.py). Updates docs/README/playbooks accordingly and closes the old #5 (atomic mirror) as moot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-04 11:29:57 +02:00
parent 16500c4392
commit 319218ce21
18 changed files with 76 additions and 451 deletions
@@ -23,15 +23,15 @@ every client regardless of tier.
                         │                Postgres (smb-db)            │
                         │  clients · leads · projects · bookings ·    │
                         │  invoices · activity_log · credentials —    │
-                         │  SOURCE OF TRUTH (credentials never mirror) │
-                         └───────────────▲───────────────┬────────────┘
-                                         │ SQL             │ one-way mirror
-                          JSON API       │                 ▼
-                    ┌────────────────────┴───────┐   ┌───────────────────┐
-                    │   smb-crm (Flask, backoffice)│   │  Google Sheets    │
-                    │   onboard.mivanchenko.de/crm │   │  (human overview,  │
-                    │   dashboard + CRUD + iCal    │   │  Looker Studio)    │
-                    └────────────────────▲─────────┘   └───────────────────┘
+                         │  SOLE SOURCE OF TRUTH                       │
+                         └───────────────▲──────────────────────────────┘
+                                         │ SQL
+                          JSON API       │
+                    ┌────────────────────┴───────┐
+                    │   smb-crm (Flask, backoffice)│
+                    │   onboard.mivanchenko.de/crm │
+                    │   dashboard + CRUD + iCal    │
+                    └────────────────────▲─────────┘
                                         │ HTTP (CRM_API_TOKEN)
                                         │
                    ┌────────────────────┴─────────────────────────────┐
@@ -49,10 +49,9 @@ every client regardless of tier.
                 └───────────────────────────┘  └───────────────────────────┘
 ```

-**Postgres is the source of truth.** Google Sheets used to be authoritative; the back-office
-rewrite (2026-06-25) flipped that — the DB now owns writes, and Sheets is a best-effort,
-one-way projection kept for the human-readable overview and the Looker Studio dashboard. See
-`backoffice/app/db.py` (schema/contract), `backoffice/app/sheets.py` (mirror client).
+**Postgres is the sole source of truth.** Google Sheets used to be authoritative, then (from
+2026-06-25) became a best-effort downstream mirror; that mirror was removed entirely (#13) —
+there is no second data path anymore. See `backoffice/app/db.py` (schema/contract).

 Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a
 shared external `proxy` Docker network. Compose groups (§4) are independent projects that only
@@ -61,7 +60,7 @@ share that network — see `deploy/clients/README.md` for the full picture.
 ## 3. Data model

 Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into
-`backoffice/db/init.sql` (Postgres) and Google Sheets tabs (`sheets/SCHEMA.md`):
+`backoffice/db/init.sql` (Postgres):

 | Entity | PK | Purpose |
 |---|---|---|
@@ -71,11 +70,10 @@ Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 in
 | `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. |
 | `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). |
 | `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. |
-| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. **The one entity excluded from the Sheets mirror** (`mirror: False` in `TABLES`) so secrets never leave Postgres; reading it also requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone). Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |
+| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. Reading it requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone), so secrets never leave Postgres. Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |

 `db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps,
-numbers, booleans) so the CRUD API, the n8n ingest path and the one-time Sheets importer can never
-drift on types.
+numbers, booleans) so the CRUD API and the n8n ingest path can never drift on types.

 ## 4. Components

@@ -88,7 +86,6 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al
  `renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted.
 - `PATCH /api/<entity>/<id>` — partial update, same auth.
 - `DELETE /api/<entity>/<id>` — delete, same auth.
- `POST /api/sync` — force a full DB→Sheets resync of every tab (same auth).
 - `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't
  send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of
  the header token). Supports `?client_id=` to scope to one client.
@@ -98,15 +95,10 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al
  tab masks the `secret` column by default with a per-row 👁 reveal toggle and a 📋 copy-to-
  clipboard button).

-Every write is audit-logged to `activity_log` and enqueues an async, best-effort mirror of that
-entity (and of `activity_log` itself) into the linked Google Sheet — mirror failures never fail
-the DB write (`mirror_async` / `_mirror_worker` in `app.py`).
-
-`import_from_sheets.py` is a one-time, idempotent bootstrap (upsert by PK) used only to seed
-Postgres from the pre-existing Sheet; after that the Sheet is purely downstream.
+Every write is audit-logged to `activity_log`.

 Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from
-`backoffice/.env.example` + a service-account JSON mounted at `./secrets/gcp-sa.json`).
+`backoffice/.env.example`).

 ### 4.2 `templates/landing/` — landing pages & demos
 Static, self-contained HTML (no build step). `templates/landing/index.html` is the public
@@ -160,19 +152,13 @@ keeps the 14 most recent dumps in `/home/mivanchenko/backups/smb-crm/`.
 | `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. |
 | `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients` → `POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA username → `POST /api/credentials` with the EA username **and password**. Fully automates "sign a client" end to end, including capturing the generated password so it isn't lost (it used to be discarded after the EA API call — see `TODO.md`). |
 | `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. |
-| `renewal-reminder.json` | daily 08:00 schedule | Read `Clients` from Sheets → find renewals due soon → Telegram notify → append a row to `Activity Log`. **Note:** still reads from the Sheets mirror rather than the DB directly — safe today because the mirror is kept current, but a re-point to the DB would remove that indirection. |
+| `renewal-reminder.json` | daily 08:00 schedule | `GET /api/clients` → find renewals due soon → Telegram notify → `POST /api/activity_log`. |

 n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a
 separate, already-running compose stack on the homelab; only the exported workflow definitions
 live here.

-### 4.6 `sheets/SCHEMA.md` — Sheets mirror spec
-Describes the Google Sheets workbook (`SMB-Online — CRM`) that Postgres mirrors into: one tab per
-entity, headers matching column names (n8n / the mirror map by header). Also specifies the
-**Looker Studio** dashboard built on top of it (pipeline by status, MRR, renewals due in 30 days,
-recent leads) — the dashboard itself lives in Google, not in this repo.
-
-### 4.7 `playbooks/` — operator procedure
+### 4.6 `playbooks/` — operator procedure
 - `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out
  (email/DM/phone scripts in German).
 - `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the
@@ -187,7 +173,7 @@ its own `.env.example` to copy from:

 | File | Fills |
 |---|---|
-| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN`, `SHEET_ID` (+ a service-account JSON mounted at `secrets/gcp-sa.json`, not example-tracked) |
+| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN` |
 | `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` |
 | `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) |

@@ -196,7 +182,7 @@ Local dev (no build tooling needed anywhere in this repo):
 # Demos — self-contained HTML, open directly or serve the folder:
 python3 -m http.server 8080 --directory templates/landing

-# Back office (needs Postgres + a Google service account):
+# Back office (needs Postgres):
 docker compose -f backoffice/docker-compose.yml up
 ```
 Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy;
@@ -209,18 +195,13 @@ see `deploy/clients/README.md` and each group's own compose file for the exact r
 - The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients
  can't send custom headers — treat that token as effectively public-linkable and rotate it if a
  feed URL leaks.
- Sheet cell values are defended against formula injection (`_cell()` in `app.py` prefixes
-  values starting with `=+-@` with a `'`).
 - Credentials for client-owned accounts are recorded two ways: `clients.vault_ref` points to a
  Vaultwarden item for anything the operator manually stashes there; the `credentials` table
  holds secrets the *system itself* generates (currently: the Easy!Appointments provider login
-  created during onboarding), gated by `X-CRM-Token` even to read and deliberately excluded from
-  the Sheets mirror. Stored as plaintext in Postgres today — same trust boundary as the rest of
-  the CRM (Caddy basic-auth + host security); revisit with column-level encryption (pgcrypto) if
-  the dashboard is ever exposed more broadly (see `TODO.md`).
- The DB→Sheets mirror is clear-then-write, not atomic — a reader can theoretically catch a
-  cleared tab mid-sync. Accepted as low-risk (human overview, not a system of record) — see
-  `TODO.md`.
+  created during onboarding), gated by `X-CRM-Token` even to read. Stored as plaintext in
+  Postgres today — same trust boundary as the rest of the CRM (Caddy basic-auth + host security);
+  revisit with column-level encryption (pgcrypto) if the dashboard is ever exposed more broadly
+  (see `TODO.md`).

 ## 7. Legal / positioning constraint
 The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays