diff --git a/Documentation.md b/Documentation.md index aec8bbc..fc81fb8 100644 --- a/Documentation.md +++ b/Documentation.md @@ -23,15 +23,15 @@ every client regardless of tier. │ Postgres (smb-db) │ │ clients · leads · projects · bookings · │ │ invoices · activity_log · credentials — │ - │ SOURCE OF TRUTH (credentials never mirror) │ - └───────────────▲───────────────┬────────────┘ - │ SQL │ one-way mirror - JSON API │ ▼ - ┌────────────────────┴───────┐ ┌───────────────────┐ - │ smb-crm (Flask, backoffice)│ │ Google Sheets │ - │ onboard.mivanchenko.de/crm │ │ (human overview, │ - │ dashboard + CRUD + iCal │ │ Looker Studio) │ - └────────────────────▲─────────┘ └───────────────────┘ + │ SOLE SOURCE OF TRUTH │ + └───────────────▲──────────────────────────────┘ + │ SQL + JSON API │ + ┌────────────────────┴───────┐ + │ smb-crm (Flask, backoffice)│ + │ onboard.mivanchenko.de/crm │ + │ dashboard + CRUD + iCal │ + └────────────────────▲─────────┘ │ HTTP (CRM_API_TOKEN) │ ┌────────────────────┴─────────────────────────────┐ @@ -49,10 +49,9 @@ every client regardless of tier. └───────────────────────────┘ └───────────────────────────┘ ``` -**Postgres is the source of truth.** Google Sheets used to be authoritative; the back-office -rewrite (2026-06-25) flipped that — the DB now owns writes, and Sheets is a best-effort, -one-way projection kept for the human-readable overview and the Looker Studio dashboard. See -`backoffice/app/db.py` (schema/contract), `backoffice/app/sheets.py` (mirror client). +**Postgres is the sole source of truth.** Google Sheets used to be authoritative, then (from +2026-06-25) became a best-effort downstream mirror; that mirror was removed entirely (#13) — +there is no second data path anymore. See `backoffice/app/db.py` (schema/contract). Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a shared external `proxy` Docker network. Compose groups (§4) are independent projects that only @@ -61,7 +60,7 @@ share that network — see `deploy/clients/README.md` for the full picture. ## 3. Data model Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into -`backoffice/db/init.sql` (Postgres) and Google Sheets tabs (`sheets/SCHEMA.md`): +`backoffice/db/init.sql` (Postgres): | Entity | PK | Purpose | |---|---|---| @@ -71,11 +70,10 @@ Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 in | `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. | | `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). | | `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. | -| `credentials` | `cred_id` (`CR-`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. **The one entity excluded from the Sheets mirror** (`mirror: False` in `TABLES`) so secrets never leave Postgres; reading it also requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone). Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. | +| `credentials` | `cred_id` (`CR-`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. Reading it requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone), so secrets never leave Postgres. Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. | `db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps, -numbers, booleans) so the CRUD API, the n8n ingest path and the one-time Sheets importer can never -drift on types. +numbers, booleans) so the CRUD API and the n8n ingest path can never drift on types. ## 4. Components @@ -88,7 +86,6 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al `renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted. - `PATCH /api//` — partial update, same auth. - `DELETE /api//` — delete, same auth. -- `POST /api/sync` — force a full DB→Sheets resync of every tab (same auth). - `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of the header token). Supports `?client_id=` to scope to one client. @@ -98,15 +95,10 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al tab masks the `secret` column by default with a per-row 👁 reveal toggle and a 📋 copy-to- clipboard button). -Every write is audit-logged to `activity_log` and enqueues an async, best-effort mirror of that -entity (and of `activity_log` itself) into the linked Google Sheet — mirror failures never fail -the DB write (`mirror_async` / `_mirror_worker` in `app.py`). - -`import_from_sheets.py` is a one-time, idempotent bootstrap (upsert by PK) used only to seed -Postgres from the pre-existing Sheet; after that the Sheet is purely downstream. +Every write is audit-logged to `activity_log`. Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from -`backoffice/.env.example` + a service-account JSON mounted at `./secrets/gcp-sa.json`). +`backoffice/.env.example`). ### 4.2 `templates/landing/` — landing pages & demos Static, self-contained HTML (no build step). `templates/landing/index.html` is the public @@ -160,19 +152,13 @@ keeps the 14 most recent dumps in `/home/mivanchenko/backups/smb-crm/`. | `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. | | `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients` → `POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA username → `POST /api/credentials` with the EA username **and password**. Fully automates "sign a client" end to end, including capturing the generated password so it isn't lost (it used to be discarded after the EA API call — see `TODO.md`). | | `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. | -| `renewal-reminder.json` | daily 08:00 schedule | Read `Clients` from Sheets → find renewals due soon → Telegram notify → append a row to `Activity Log`. **Note:** still reads from the Sheets mirror rather than the DB directly — safe today because the mirror is kept current, but a re-point to the DB would remove that indirection. | +| `renewal-reminder.json` | daily 08:00 schedule | `GET /api/clients` → find renewals due soon → Telegram notify → `POST /api/activity_log`. | n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a separate, already-running compose stack on the homelab; only the exported workflow definitions live here. -### 4.6 `sheets/SCHEMA.md` — Sheets mirror spec -Describes the Google Sheets workbook (`SMB-Online — CRM`) that Postgres mirrors into: one tab per -entity, headers matching column names (n8n / the mirror map by header). Also specifies the -**Looker Studio** dashboard built on top of it (pipeline by status, MRR, renewals due in 30 days, -recent leads) — the dashboard itself lives in Google, not in this repo. - -### 4.7 `playbooks/` — operator procedure +### 4.6 `playbooks/` — operator procedure - `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out (email/DM/phone scripts in German). - `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the @@ -187,7 +173,7 @@ its own `.env.example` to copy from: | File | Fills | |---|---| -| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN`, `SHEET_ID` (+ a service-account JSON mounted at `secrets/gcp-sa.json`, not example-tracked) | +| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN` | | `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` | | `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) | @@ -196,7 +182,7 @@ Local dev (no build tooling needed anywhere in this repo): # Demos — self-contained HTML, open directly or serve the folder: python3 -m http.server 8080 --directory templates/landing -# Back office (needs Postgres + a Google service account): +# Back office (needs Postgres): docker compose -f backoffice/docker-compose.yml up ``` Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy; @@ -209,18 +195,13 @@ see `deploy/clients/README.md` and each group's own compose file for the exact r - The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients can't send custom headers — treat that token as effectively public-linkable and rotate it if a feed URL leaks. -- Sheet cell values are defended against formula injection (`_cell()` in `app.py` prefixes - values starting with `=+-@` with a `'`). - Credentials for client-owned accounts are recorded two ways: `clients.vault_ref` points to a Vaultwarden item for anything the operator manually stashes there; the `credentials` table holds secrets the *system itself* generates (currently: the Easy!Appointments provider login - created during onboarding), gated by `X-CRM-Token` even to read and deliberately excluded from - the Sheets mirror. Stored as plaintext in Postgres today — same trust boundary as the rest of - the CRM (Caddy basic-auth + host security); revisit with column-level encryption (pgcrypto) if - the dashboard is ever exposed more broadly (see `TODO.md`). -- The DB→Sheets mirror is clear-then-write, not atomic — a reader can theoretically catch a - cleared tab mid-sync. Accepted as low-risk (human overview, not a system of record) — see - `TODO.md`. + created during onboarding), gated by `X-CRM-Token` even to read. Stored as plaintext in + Postgres today — same trust boundary as the rest of the CRM (Caddy basic-auth + host security); + revisit with column-level encryption (pgcrypto) if the dashboard is ever exposed more broadly + (see `TODO.md`). ## 7. Legal / positioning constraint The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays diff --git a/README.md b/README.md index 8f8592c..2fd5c76 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,8 @@ A productized service that gets local small businesses online: **landing page + online booking + lead capture + small automations**. Built to be **thoroughly tracked and logged** — -every client, lead, booking and invoice lives in a **Postgres CRM** (source of truth), driven by -**n8n** on the homelab, with a one-way mirror into Google Sheets for a human-readable overview / -Looker Studio dashboard. +every client, lead, booking and invoice lives in a **Postgres CRM** (sole source of truth), +driven by **n8n** on the homelab. > Full documentation (architecture, data model, every component): **`Documentation.md`**. @@ -29,7 +28,6 @@ deploy/ backup/ Daily Postgres backup script (cron on the homelab) n8n/ Exported n8n workflows (lead-intake, onboarding, booking-sync, renewal-reminder) playbooks/ Operator playbooks (outreach, lead-to-customer, tier-a, tier-b) -sheets/SCHEMA.md Google Sheets mirror spec (tabs + columns) — downstream of Postgres templates/landing/ Landing-page demos + client preview pages demo-tier-a/ "Schnittpunkt" barbershop demo demo-tier-b/ "PhysioVital" physio practice demo @@ -65,8 +63,7 @@ Tracked against actual repo state — see `Documentation.md` for what backs each - [x] **Phase 1** — Demo landing pages - [x] **Phase 2** — Accounts & setup: Google account + workbook, n8n live, Google creds bound, Vaultwarden (`clients.vault_ref`), domain (wildcard `*.mivanchenko.de`) -- [x] **Phase 3** — CRM (now Postgres-first, Sheets mirrored) + n8n lead-intake & renewal - workflows — Looker Studio dashboard itself lives in Google and isn't repo-verifiable +- [x] **Phase 3** — CRM (Postgres-only, no Sheets mirror) + n8n lead-intake & renewal workflows - [x] **Phase 4** — Demo forms + booking wired live to n8n webhooks (lead-intake, booking-sync) - [ ] **Phase 5** — First real client + reception v1 — onboarding automation and per-client deploy tooling exist and are exercised (`deploy/clients/`); not verifiable from the repo diff --git a/TODO.md b/TODO.md index f89c414..03ea486 100644 --- a/TODO.md +++ b/TODO.md @@ -12,7 +12,6 @@ **CRM / back office** - **#2** Decouple tier from business type → hosting/plan model; realign onboarding form + workflow. `enhancement` `crm` - **#3** Extend back-office CRUD beyond Leads & Clients (projects, bookings, invoices, activity). `enhancement` `crm` -- **#5** Make the DB→Sheets mirror atomic (no brief empty-sheet window). `tech-debt` `crm` **Booking** - **#7** Proper Orders flow + Orders table/tab (pizzeria demo posts into Leads as a stop-gap). `enhancement` `booking` @@ -41,8 +40,12 @@ exposed more broadly than Caddy basic-auth + host security. ## Done -- [x] 2026-07-15 — **Credentials in the CRM**: `credentials` entity (never mirrored to Sheets, - `X-CRM-Token`-gated even to read), dashboard Credentials tab with masked value + reveal/copy; +- [x] 2026-08-04 — **Cut Google Sheets entirely** (#13): Postgres is now the sole source of + truth — deleted the DB→Sheets mirror, the Sheets client, and the one-time importer; + `renewal-reminder.json` reads from the DB API instead of the Sheets mirror. Moots #5 + (atomic-mirror tech-debt, closed as no longer applicable). +- [x] 2026-07-15 — **Credentials in the CRM**: `credentials` entity (`X-CRM-Token`-gated even to + read), dashboard Credentials tab with masked value + reveal/copy; `n8n/onboarding.json` now persists the auto-generated EA provider password. Shipped live. - [x] 2026-07-15 — **Gitea Actions deploy pipeline** (backoffice): self-hosted `homelab-runner` (`act-runner` systemd service), `workflow_dispatch` `deploy-backoffice.yml` that syncs diff --git a/backoffice/.env.example b/backoffice/.env.example index 7e321a3..7722eba 100644 --- a/backoffice/.env.example +++ b/backoffice/.env.example @@ -5,7 +5,6 @@ BOOKING_TOKEN_SECRET=change-me-long-random-too # Owner-login session cookie signing key (#19). Dedicated secret -- rotating # it just logs owners out, without touching CRM_API_TOKEN/BOOKING_TOKEN_SECRET. SESSION_SECRET_KEY=change-me-long-random-session-too -SHEET_ID=1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8 # Booking confirmation email (#18). Left blank, sending is skipped (logged, # not fatal) -- mail relay setup is a separate infra/triage item. SMTP_HOST= diff --git a/backoffice/app/app.py b/backoffice/app/app.py index 1cd5a9a..5b03da6 100644 --- a/backoffice/app/app.py +++ b/backoffice/app/app.py @@ -1,16 +1,12 @@ """smb-crm back-office service. -Postgres is the source of truth. This serves the operator dashboard + a small -JSON API (read now; add/edit/delete and the DB->Sheets mirror layered on next). +Postgres is the source of truth for the operator dashboard + a small JSON API. Browser access is gated by Caddy basic-auth on onboard.mivanchenko.de; the machine-to-machine ingest path (n8n) is gated by the CRM_API_TOKEN header. """ import os import re import time -import queue -import threading -import traceback from datetime import datetime, date, timezone from decimal import Decimal @@ -20,7 +16,6 @@ from waitress import serve import booking_db as bdb import db import owner_mail -from sheets import Sheets from booking_api import bp as booking_bp from public_booking import bp as public_booking_bp from manage_booking import bp as manage_booking_bp @@ -47,69 +42,6 @@ CRM_TOKEN = os.environ.get("CRM_API_TOKEN", "") # the basic-auth header, so the feed is gated by this query token instead). ICS_TOKEN = os.environ.get("ICS_TOKEN", "") -# DB -> Sheets one-way mirror. Postgres is the source of truth; the Sheet is a -# best-effort projection. A mirror failure never fails the DB write. -SH = None -try: - SH = Sheets(os.environ["GOOGLE_SA_JSON"], os.environ["SHEET_ID"]) -except Exception: # noqa: BLE001 - traceback.print_exc() - - -def _cell(v): - if v is None: - return "" - if isinstance(v, bool): - return "TRUE" if v else "FALSE" - if isinstance(v, datetime): - return v.strftime("%Y-%m-%d %H:%M") - if isinstance(v, date): - return v.strftime("%Y-%m-%d") - if isinstance(v, Decimal): - f = float(v) - return str(int(f)) if f == int(f) else str(f) - s = str(v) - return ("'" + s) if s[:1] in "=+-@" else s # neutralise formula injection - - -def mirror_entity(entity): - spec = db.TABLES[entity] - if SH is None or spec.get("mirror") is False: - return 0 - cols = spec["cols"] - order = LIST_ORDER.get(entity, spec["pk"]) - with db.connect() as conn, conn.cursor() as cur: - cur.execute(f"SELECT {', '.join(cols)} FROM {entity} ORDER BY {order}") - rows = cur.fetchall() - grid = [[_cell(r[c]) for c in cols] for r in rows] - SH.overwrite(spec["tab"], cols, grid) - return len(grid) - - -# Serialize all mirror writes through one worker so concurrent mutations can't -# race on the shared Sheets client / token. -_mirror_q = queue.Queue() - - -def _mirror_worker(): - while True: - entity = _mirror_q.get() - try: - mirror_entity(entity) - except Exception: # noqa: BLE001 - print(f"[mirror] {entity} sync failed:", flush=True) - traceback.print_exc() - finally: - _mirror_q.task_done() - - -threading.Thread(target=_mirror_worker, daemon=True).start() - - -def mirror_async(entity): - if SH is not None and db.TABLES[entity].get("mirror") is not False: - _mirror_q.put(entity) - # Static dashboard, with the CRM token injected so the (basic-auth-gated) # operator page can call the token-protected mutation endpoints. with open(os.path.join(os.path.dirname(__file__), "static", "index.html")) as _f: @@ -235,8 +167,6 @@ def add_entity(entity): [row.get(c) for c in cols]) log_activity(cur, row.get("client_id"), f"add {entity}", f"{pk}={row.get(pk)}") conn.commit() - mirror_async(entity) - mirror_async("activity_log") return jsonify({"added": row.get(pk)}), 201 @@ -262,8 +192,6 @@ def edit_entity(entity, ident): log_activity(cur, ident if entity == "clients" else None, f"edit {entity}", f"{pk}={ident}: {', '.join(setcols)}") conn.commit() - mirror_async(entity) - mirror_async("activity_log") return jsonify({"updated": ident, "fields": setcols}) @@ -281,8 +209,6 @@ def delete_entity(entity, ident): log_activity(cur, ident if entity == "clients" else None, f"delete {entity}", f"{pk}={ident}") conn.commit() - mirror_async(entity) - mirror_async("activity_log") return jsonify({"deleted": ident}) @@ -313,26 +239,9 @@ def trigger_owner_password_reset(user_id): log_activity(cur, user["client_id"], "owner password reset", f"user_id={user_id} (operator-triggered)") conn.commit() - mirror_async("activity_log") return jsonify({"sent": user_id}) -@app.post("/api/sync") -def sync_all(): - """Full DB -> Sheets resync of every tab (manual / alignment).""" - if not authed(): - return jsonify({"error": "forbidden"}), 403 - if SH is None: - return jsonify({"error": "sheets unavailable"}), 503 - out = {} - for entity in db.TABLES: - try: - out[entity] = mirror_entity(entity) - except Exception as e: # noqa: BLE001 - out[entity] = f"error: {e}" - return jsonify({"synced": out}) - - def _ics_dt(v): if isinstance(v, datetime): u = v if v.tzinfo else v.replace(tzinfo=timezone.utc) diff --git a/backoffice/app/db.py b/backoffice/app/db.py index 5dd345f..1a86063 100644 --- a/backoffice/app/db.py +++ b/backoffice/app/db.py @@ -1,8 +1,8 @@ """Postgres access + the table/column contract shared across the app. The TABLES map is the single definition of which entities exist, their columns, -and their primary key. Read/CRUD endpoints, the importer and the Sheets mirror -all derive from it so they can never drift apart. +and their primary key. Read/CRUD endpoints all derive from it so they can +never drift apart. """ import os import re @@ -13,10 +13,9 @@ from psycopg.rows import dict_row DATABASE_URL = os.environ["DATABASE_URL"] -# entity -> (sheet tab, primary key, ordered columns) +# entity -> (primary key, ordered columns) TABLES = { "clients": { - "tab": "Clients", "pk": "client_id", "cols": ["client_id", "business_name", "owner_name", "email", "phone", "niche", "tier", "status", "domain", "stack_notes", "vault_ref", @@ -28,7 +27,6 @@ TABLES = { "bools": [], }, "leads": { - "tab": "Leads", "pk": "lead_id", "cols": ["lead_id", "received_at", "client_id", "source", "name", "contact", "service_interest", "message", "status", "notified"], @@ -38,7 +36,6 @@ TABLES = { "bools": ["notified"], }, "projects": { - "tab": "Projects", "pk": "project_id", "cols": ["project_id", "client_id", "deliverable", "tier", "checklist", "go_live_date", "status"], @@ -48,7 +45,6 @@ TABLES = { "bools": [], }, "bookings": { - "tab": "Bookings", "pk": "booking_id", "cols": ["booking_id", "created_at", "client_id", "customer_name", "customer_contact", "service", "start_time", "end_time", @@ -59,7 +55,6 @@ TABLES = { "bools": [], }, "invoices": { - "tab": "Invoices", "pk": "invoice_id", "cols": ["invoice_id", "client_id", "issued_date", "due_date", "amount_eur", "period", "status", "paid_date"], @@ -69,7 +64,6 @@ TABLES = { "bools": [], }, "activity_log": { - "tab": "Activity Log", "pk": "id", "cols": ["ts", "workflow", "client_id", "action", "detail", "result"], "dates": [], @@ -78,14 +72,13 @@ TABLES = { "bools": [], }, "credentials": { - # No "tab": never mirrored to Sheets (see "mirror" below) — secrets stay in Postgres only. + # Secrets stay in Postgres only. "pk": "cred_id", "cols": ["cred_id", "client_id", "label", "username", "secret", "notes", "created_at"], "dates": [], "timestamps": ["created_at"], "numbers": [], "bools": [], - "mirror": False, }, } @@ -94,7 +87,7 @@ def connect(): return psycopg.connect(DATABASE_URL, row_factory=dict_row) -# ---- coercion: Sheet strings / JSON values -> typed Python for Postgres ---- +# ---- coercion: JSON request values -> typed Python for Postgres ---- def parse_date(v): if v in (None, ""): diff --git a/backoffice/app/import_from_sheets.py b/backoffice/app/import_from_sheets.py deleted file mode 100644 index 52f581a..0000000 --- a/backoffice/app/import_from_sheets.py +++ /dev/null @@ -1,57 +0,0 @@ -"""One-time (idempotent) seed of Postgres from the existing Google Sheets. - -This is the ONLY place the Sheet is treated as authoritative — to bootstrap the -DB. After this, Postgres is the source of truth and Sheets is a downstream -mirror. Safe to re-run: keyed tables upsert by primary key; activity_log is -replaced wholesale. - -Run inside the container: docker compose exec smb-crm python import_from_sheets.py -""" -import os -import sys - -import db -from sheets import Sheets - - -def upsert(cur, entity, records): - spec = db.TABLES[entity] - cols = spec["cols"] - pk = spec["pk"] - n = 0 - for rec in records: - row = db.coerce_row(entity, rec) - if entity != "activity_log" and not row.get(pk): - continue # skip rows without a primary key - placeholders = ", ".join(["%s"] * len(cols)) - collist = ", ".join(cols) - if entity == "activity_log": - cur.execute(f"INSERT INTO {entity} ({collist}) VALUES ({placeholders})", - [row[c] for c in cols]) - else: - updates = ", ".join(f"{c} = EXCLUDED.{c}" for c in cols if c != pk) - cur.execute( - f"INSERT INTO {entity} ({collist}) VALUES ({placeholders}) " - f"ON CONFLICT ({pk}) DO UPDATE SET {updates}", - [row[c] for c in cols]) - n += 1 - return n - - -def main(): - sh = Sheets(os.environ["GOOGLE_SA_JSON"], os.environ["SHEET_ID"]) - with db.connect() as conn: - with conn.cursor() as cur: - cur.execute("TRUNCATE activity_log RESTART IDENTITY") - for entity, spec in db.TABLES.items(): - if spec.get("mirror") is False: - continue # e.g. credentials — never lived in the Sheet, nothing to import - records = sh.read_records(spec["tab"]) - n = upsert(cur, entity, records) - print(f" {entity:<13} <- {spec['tab']:<13} {n} row(s)") - conn.commit() - print("import complete.") - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/backoffice/app/mailer.py b/backoffice/app/mailer.py index 64ea1aa..a6c95ca 100644 --- a/backoffice/app/mailer.py +++ b/backoffice/app/mailer.py @@ -1,8 +1,8 @@ """Best-effort SMTP email sending (#18). -Mirrors app.py's DB -> Sheets mirror pattern: a single background worker -thread drains a queue, and a send failure is logged, never raised back to the -caller -- booking creation must succeed even if the mail relay is down. +A single background worker thread drains a queue, and a send failure is +logged, never raised back to the caller -- booking creation must succeed even +if the mail relay is down. """ import os import queue diff --git a/backoffice/app/requirements.txt b/backoffice/app/requirements.txt index 4d9c685..41d157e 100644 --- a/backoffice/app/requirements.txt +++ b/backoffice/app/requirements.txt @@ -8,7 +8,6 @@ Flask==3.0.3 psycopg[binary]==3.2.1 waitress==3.0.0 PyJWT[crypto]==2.9.0 -requests==2.32.3 # python:3.12-slim has no system IANA tz database; zoneinfo (used by # availability.py for Europe/Berlin) falls back to this package for it. tzdata==2024.1 diff --git a/backoffice/app/sheets.py b/backoffice/app/sheets.py deleted file mode 100644 index 2ddc3e9..0000000 --- a/backoffice/app/sheets.py +++ /dev/null @@ -1,98 +0,0 @@ -"""Minimal Google Sheets v4 client using a service-account JWT. - -Used for (a) the one-time import of existing Sheet rows into Postgres and -(b) the one-way DB -> Sheets mirror sync. Postgres stays the source of truth; -nothing here treats the Sheet as authoritative except the explicit import. -""" -import json -import time -import threading - -import jwt -import requests - -SHEETS_API = "https://sheets.googleapis.com/v4/spreadsheets" -TOKEN_URL = "https://oauth2.googleapis.com/token" -SCOPE = "https://www.googleapis.com/auth/spreadsheets" - -# gid map for the live workbook (tab name -> gid), used when clearing/sizing. -TAB_GIDS = { - "Clients": 470934735, - "Leads": 571719114, - "Bookings": 946084008, - "Projects": 619112786, - "Invoices": 413377229, - "Activity Log": 170940481, -} - - -class Sheets: - def __init__(self, sa_json_path, sheet_id): - with open(sa_json_path) as f: - self.sa = json.load(f) - self.sheet_id = sheet_id - self._tok = None - self._exp = 0 - self._lock = threading.Lock() - - def _token(self): - with self._lock: - now = int(time.time()) - if self._tok and now < self._exp - 60: - return self._tok - claim = { - "iss": self.sa["client_email"], - "scope": SCOPE, - "aud": TOKEN_URL, - "iat": now, - "exp": now + 3600, - } - assertion = jwt.encode(claim, self.sa["private_key"], algorithm="RS256") - r = requests.post(TOKEN_URL, data={ - "grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer", - "assertion": assertion, - }, timeout=30) - r.raise_for_status() - self._tok = r.json()["access_token"] - self._exp = now + 3600 - return self._tok - - def _headers(self): - return {"Authorization": "Bearer " + self._token()} - - def read(self, a1_range): - """Return raw 2D list of cell values for an A1 range (e.g. "Leads!A1:Z").""" - url = f"{SHEETS_API}/{self.sheet_id}/values/{requests.utils.quote(a1_range)}" - r = requests.get(url, headers=self._headers(), timeout=30) - r.raise_for_status() - return r.json().get("values", []) - - def read_records(self, tab): - """Read a whole tab as a list of dicts keyed by the header row.""" - rows = self.read(f"{tab}!A1:Z") - if not rows: - return [] - header = rows[0] - out = [] - for raw in rows[1:]: - if not any(c.strip() for c in raw): - continue - rec = {header[i]: (raw[i] if i < len(raw) else "") for i in range(len(header))} - out.append(rec) - return out - - def overwrite(self, tab, header, rows): - """Replace a tab's contents with header + rows (the DB->Sheets mirror). - - Clears the existing value range, then writes the new grid starting at A1. - Postgres is the source of truth; this projects it onto the Sheet. - """ - # clear current values (keeps formatting / the tab itself) - clr = f"{SHEETS_API}/{self.sheet_id}/values/{requests.utils.quote(tab + '!A1:Z')}:clear" - requests.post(clr, headers=self._headers(), timeout=30).raise_for_status() - body = {"values": [header] + rows} - url = (f"{SHEETS_API}/{self.sheet_id}/values/" - f"{requests.utils.quote(tab + '!A1')}?valueInputOption=RAW") - r = requests.put(url, headers=self._headers(), json=body, timeout=60) - r.raise_for_status() - return len(rows) diff --git a/backoffice/app/static/index.html b/backoffice/app/static/index.html index df5c48c..d500228 100644 --- a/backoffice/app/static/index.html +++ b/backoffice/app/static/index.html @@ -74,7 +74,7 @@

🗂️ smb-crm · Back Office

📝 Onboarding-Formular - Quelle: Postgres (Sheets = Spiegel) · + Quelle: Postgres ·
diff --git a/backoffice/app/tests/test_mailer.py b/backoffice/app/tests/test_mailer.py index ec43b92..dbc9ab4 100644 --- a/backoffice/app/tests/test_mailer.py +++ b/backoffice/app/tests/test_mailer.py @@ -1,7 +1,6 @@ """Unit tests for mailer.py's SMTP call shape (#18). Calls _send_now directly rather than going through the background-thread queue, so assertions are -synchronous -- the queue itself is just plumbing, already covered indirectly -by app.py's identical Sheets-mirror pattern. +synchronous -- the queue itself is just plumbing. """ from email.message import EmailMessage diff --git a/backoffice/db/init.sql b/backoffice/db/init.sql index 2533b45..faee5fd 100644 --- a/backoffice/db/init.sql +++ b/backoffice/db/init.sql @@ -1,6 +1,4 @@ --- smb-crm — source-of-truth schema (Postgres). --- Google Sheets is a one-way downstream mirror fed by the DB->Sheets sync. --- Column sets mirror the Sheet tabs so the mirror stays 1:1. +-- smb-crm — sole source-of-truth schema (Postgres). CREATE TABLE IF NOT EXISTS clients ( client_id text PRIMARY KEY, @@ -87,8 +85,8 @@ CREATE TABLE IF NOT EXISTS invoices ( ); -- Per-client login credentials (e.g. the auto-generated Easy!Appointments --- provider login). Deliberately NOT mirrored to Sheets — see db.py TABLES --- ("mirror": False) — so secrets never leave Postgres. +-- provider login). Gated by X-CRM-Token even to read, so secrets never leave +-- Postgres. CREATE TABLE IF NOT EXISTS credentials ( cred_id text PRIMARY KEY, client_id text, diff --git a/backoffice/docker-compose.yml b/backoffice/docker-compose.yml index 0c1758d..c76fcd5 100644 --- a/backoffice/docker-compose.yml +++ b/backoffice/docker-compose.yml @@ -27,16 +27,12 @@ services: BOOKING_TOKEN_SECRET: ${BOOKING_TOKEN_SECRET} SESSION_SECRET_KEY: ${SESSION_SECRET_KEY} ICS_TOKEN: ${ICS_TOKEN} - SHEET_ID: ${SHEET_ID} - GOOGLE_SA_JSON: /run/secrets/gcp-sa.json SMTP_HOST: ${SMTP_HOST} SMTP_PORT: ${SMTP_PORT} SMTP_USERNAME: ${SMTP_USERNAME} SMTP_PASSWORD: ${SMTP_PASSWORD} MAIL_FALLBACK_FROM: ${MAIL_FALLBACK_FROM} PUBLIC_BASE_URL: ${PUBLIC_BASE_URL} - volumes: - - ./secrets/gcp-sa.json:/run/secrets/gcp-sa.json:ro depends_on: smb-db: condition: service_healthy diff --git a/n8n/renewal-reminder.json b/n8n/renewal-reminder.json index 352f07e..f81ad14 100644 --- a/n8n/renewal-reminder.json +++ b/n8n/renewal-reminder.json @@ -1,5 +1,5 @@ { - "name": "SMB \u00b7 Renewal Reminder", + "name": "SMB · Renewal Reminder", "nodes": [ { "parameters": { @@ -23,43 +23,24 @@ }, { "parameters": { - "authentication": "serviceAccount", - "resource": "sheet", - "operation": "read", - "documentId": { - "__rl": true, - "value": "1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8", - "mode": "id" - }, - "sheetName": { - "__rl": true, - "value": "470934735", - "mode": "list", - "cachedResultName": "Clients" - }, + "url": "http://smb-crm:8080/api/clients", "options": {} }, "id": "f06a60af-dff1-415d-aca6-fef0367f3553", - "name": "Read Clients", - "type": "n8n-nodes-base.googleSheets", - "typeVersion": 4.5, + "name": "Read Clients (DB)", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.2, "position": [ 460, 300 ], - "credentials": { - "googleApi": { - "id": "45TWPyl1wVk2Vxdm", - "name": "SMB Google (service account)" - } - }, "retryOnFail": true, "maxTries": 4, "waitBetweenTries": 3000 }, { "parameters": { - "jsCode": "\nconst rows = $input.all().map(i=>i.json);\nconst today=new Date();\nconst t0=Date.UTC(today.getUTCFullYear(),today.getUTCMonth(),today.getUTCDate());\nfunction parseDate(v){\n if(v==null||v==='') return null;\n const s=String(v).trim();\n if(/^\\d+(\\.\\d+)?$/.test(s)){ return Date.UTC(1899,11,30)+Number(s)*86400000; }\n const m=s.match(/^(\\d{4})-(\\d{2})-(\\d{2})/);\n if(m) return Date.UTC(+m[1],+m[2]-1,+m[3]);\n const d=Date.parse(s); return isNaN(d)?null:d;\n}\nconst offsets=[14,7,3,1,0];\nconst due=[];\nfor(const r of rows){\n const status=String(r.status||'').toLowerCase();\n if(status!=='active') continue;\n const rd=parseDate(r.renewal_date);\n if(rd==null) continue;\n const days=Math.round((rd-t0)/86400000);\n if(days<0 || offsets.includes(days)){\n due.push({client_id:r.client_id, business:r.business_name, renewal_date:r.renewal_date,\n days, fee:r.monthly_fee_eur, cycle:r.billing_cycle});\n }\n}\nif(due.length===0) return [];\ndue.sort((a,b)=>a.days-b.days);\nconst line=d=>`\u2022 ${d.business||d.client_id} \u2014 ${d.days<0?('\u00dcBERF\u00c4LLIG seit '+(-d.days)+'T'):(d.days===0?'heute':('in '+d.days+'T'))} (${d.renewal_date}${d.fee?(', '+d.fee+'\u20ac/'+(d.cycle||'')):''})`;\nconst summary='\ud83d\udd14 Anstehende Verl\u00e4ngerungen ('+due.length+')\\n'+due.map(line).join('\\n');\nreturn [{json:{count:due.length, summary, due}}];\n" + "jsCode": "\nconst rows = $input.first().json.rows || [];\nconst today=new Date();\nconst t0=Date.UTC(today.getUTCFullYear(),today.getUTCMonth(),today.getUTCDate());\nfunction parseDate(v){\n if(v==null||v==='') return null;\n const s=String(v).trim();\n const m=s.match(/^(\\d{4})-(\\d{2})-(\\d{2})/);\n if(m) return Date.UTC(+m[1],+m[2]-1,+m[3]);\n const d=Date.parse(s); return isNaN(d)?null:d;\n}\nconst offsets=[14,7,3,1,0];\nconst due=[];\nfor(const r of rows){\n const status=String(r.status||'').toLowerCase();\n if(status!=='active') continue;\n const rd=parseDate(r.renewal_date);\n if(rd==null) continue;\n const days=Math.round((rd-t0)/86400000);\n if(days<0 || offsets.includes(days)){\n due.push({client_id:r.client_id, business:r.business_name, renewal_date:r.renewal_date,\n days, fee:r.monthly_fee_eur, cycle:r.billing_cycle});\n }\n}\nif(due.length===0) return [];\ndue.sort((a,b)=>a.days-b.days);\nconst line=d=>`• ${d.business||d.client_id} — ${d.days<0?('ÜBERFÄLLIG seit '+(-d.days)+'T'):(d.days===0?'heute':('in '+d.days+'T'))} (${d.renewal_date}${d.fee?(', '+d.fee+'€/'+(d.cycle||'')):''})`;\nconst summary='🔔 Anstehende Verlängerungen ('+due.length+')\\n'+due.map(line).join('\\n');\nreturn [{json:{count:due.length, summary, due}}];\n" }, "id": "9f3e4ca9-0d87-4d1c-9f5e-abdf5f9f77dd", "name": "Find due renewals", @@ -110,42 +91,30 @@ }, { "parameters": { - "authentication": "serviceAccount", - "resource": "sheet", - "operation": "append", - "documentId": { - "__rl": true, - "value": "1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8", - "mode": "id" - }, - "sheetName": { - "__rl": true, - "value": "170940481", - "mode": "list", - "cachedResultName": "Activity Log" - }, - "columns": { - "mappingMode": "autoMapInputData", - "value": {}, - "matchingColumns": [], - "schema": [] + "method": "POST", + "url": "http://smb-crm:8080/api/activity_log", + "sendHeaders": true, + "headerParameters": { + "parameters": [ + { + "name": "X-CRM-Token", + "value": "__CRM_TOKEN__" + } + ] }, + "sendBody": true, + "specifyBody": "json", + "jsonBody": "={{ JSON.stringify($json) }}", "options": {} }, "id": "55978962-489b-4740-a242-7470f08087b6", - "name": "Append to Activity Log", - "type": "n8n-nodes-base.googleSheets", - "typeVersion": 4.5, + "name": "Append to Activity Log (DB)", + "type": "n8n-nodes-base.httpRequest", + "typeVersion": 4.2, "position": [ 1340, 300 ], - "credentials": { - "googleApi": { - "id": "45TWPyl1wVk2Vxdm", - "name": "SMB Google (service account)" - } - }, "retryOnFail": true, "maxTries": 4, "waitBetweenTries": 3000 @@ -156,14 +125,14 @@ "main": [ [ { - "node": "Read Clients", + "node": "Read Clients (DB)", "type": "main", "index": 0 } ] ] }, - "Read Clients": { + "Read Clients (DB)": { "main": [ [ { @@ -200,7 +169,7 @@ "main": [ [ { - "node": "Append to Activity Log", + "node": "Append to Activity Log (DB)", "type": "main", "index": 0 } @@ -213,4 +182,4 @@ "callerPolicy": "workflowsFromSameOwner", "availableInMCP": false } -} \ No newline at end of file +} diff --git a/playbooks/tier-a-google.md b/playbooks/tier-a-google.md index b6aa277..c9529f4 100644 --- a/playbooks/tier-a-google.md +++ b/playbooks/tier-a-google.md @@ -19,7 +19,7 @@ ongoing management. Target build time: ~half a day per client. 6. **Reception** — Gmail for the business; set a signature + canned replies. 7. **Google Business Profile** — claim/verify so they show up on Maps & Search. 8. **CRM** — add a `Clients` row (tier = A, services, monthly_fee, renewal_date, vault_ref). -9. **Handover** — share logins via Vaultwarden; record only the `vault_ref` in the sheet. +9. **Handover** — share logins via Vaultwarden; record only the `vault_ref` in the CRM. ## What to charge (suggestion) - One-off setup: €250–600 depending on pages/content. diff --git a/playbooks/tier-b-selfhosted.md b/playbooks/tier-b-selfhosted.md index 211ebbe..5c3ea13 100644 --- a/playbooks/tier-b-selfhosted.md +++ b/playbooks/tier-b-selfhosted.md @@ -16,7 +16,7 @@ German-hosted infrastructure. You charge a privacy/hosting premium. Target build 4. **Lead form** — self-hosted form endpoint → n8n webhook → CRM. No third-party form service. 5. **Reception** — business mailbox; (later) the **AI receptionist** widget (Phase 6, Claude Haiku 4.5) wired to the booking + FAQ. -6. **Credentials** — store all client logins in **Vaultwarden**; sheet holds only `vault_ref`. +6. **Credentials** — store all client logins in **Vaultwarden**; the CRM holds only `vault_ref`. 7. **CRM** — add a `Clients` row (tier = B, services, monthly_fee, renewal_date, vault_ref). 8. **Backups** — ensure the Tier-B services + CRM are in the daily backup job. diff --git a/sheets/SCHEMA.md b/sheets/SCHEMA.md deleted file mode 100644 index e83d903..0000000 --- a/sheets/SCHEMA.md +++ /dev/null @@ -1,63 +0,0 @@ -# Master CRM — Google Sheets workbook spec - -One Google Sheets workbook = the whole CRM + log. I create/maintain this via the Google Drive MCP -once the fresh project Google account exists (Phase 2). n8n reads/writes it via the Sheets node. - -Workbook name: **`SMB-Online — CRM`**. One tab per concern below. Row 1 = headers (exact column -names matter — n8n maps by header). - -## Tab: `Clients` -The client list/table — the heart of the tracking requirement. - -| Column | Type | Notes | -|---|---|---| -| `client_id` | text | e.g. `C-0001` (stable key) | -| `business_name` | text | | -| `owner_name` | text | | -| `email` | text | | -| `phone` | text | | -| `niche` | text | barber, physio, café, … | -| `tier` | A \| B | delivery tier | -| `status` | enum | `lead` → `onboarding` → `active` → `paused` → `churned` | -| `domain` | text | live site URL | -| `stack_notes` | text | hosting, booking tool, etc. | -| `vault_ref` | text | **pointer** to Vaultwarden item — never a password | -| `services` | text | comma list: site, booking, reception, … | -| `billing_cycle` | enum | monthly / yearly / one-off | -| `monthly_fee_eur` | number | | -| `start_date` | date | | -| `renewal_date` | date | drives the renewal-reminder workflow | -| `created_at` | datetime | | -| `notes` | text | | - -## Tab: `Leads` -Every inbound lead across all client sites (and your own). - -`lead_id` · `received_at` · `client_id` · `source` (which site/form) · `name` · `contact` · -`service_interest` · `message` · `status` (new / contacted / won / lost) · `notified` (bool) - -## Tab: `Bookings` -Every appointment booked across client sites. - -`booking_id` · `created_at` · `client_id` · `customer_name` · `customer_contact` · `service` · -`start_time` · `end_time` · `source` (gcal / calcom) · `status` (confirmed / cancelled / no-show) - -## Tab: `Projects` -Per-client deliverable tracking. - -`project_id` · `client_id` · `deliverable` · `tier` · `checklist` (e.g. `domain✓ page✓ booking☐`) · -`go_live_date` · `status` (todo / in-progress / live) - -## Tab: `Invoices` -`invoice_id` · `client_id` · `issued_date` · `due_date` · `amount_eur` · `period` · -`status` (draft / sent / paid / overdue) · `paid_date` - -## Tab: `Activity Log` -Append-only audit trail — **every automated action writes one row here.** - -`ts` · `workflow` (lead-intake / booking-sync / renewal / onboarding / invoice) · `client_id` · -`action` · `detail` · `result` (ok / error) - -## Dashboard -Not a tab — a **Looker Studio** report wired to this workbook: pipeline by `status`, MRR -(sum of `monthly_fee_eur` where `status=active`), renewals due in next 30 days, recent leads.