Remove Google Sheets mirror entirely (#13)
Test backoffice (smb-crm) / test (push) Has been cancelled

Postgres is now the sole source of truth: delete sheets.py and
import_from_sheets.py, strip mirror_entity/mirror_async/_mirror_worker and
POST /api/sync from app.py, drop the tab/mirror keys from db.py's TABLES.
Re-point n8n/renewal-reminder.json at the CRM's own HTTP API (GET
/api/clients, POST /api/activity_log) instead of the Sheets nodes, and drop
SHEET_ID/GOOGLE_SA_JSON from deploy env/compose and requests from
requirements.txt (PyJWT stays — still used by booking_api.py). Updates
docs/README/playbooks accordingly and closes the old #5 (atomic mirror) as
moot.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-04 11:29:57 +02:00
parent 16500c4392
commit 319218ce21
18 changed files with 76 additions and 451 deletions
+25 -44
View File
@@ -23,15 +23,15 @@ every client regardless of tier.
│ Postgres (smb-db) │
│ clients · leads · projects · bookings · │
│ invoices · activity_log · credentials — │
│ SOURCE OF TRUTH (credentials never mirror)
└───────────────▲───────────────────────────┘
│ SQL │ one-way mirror
JSON API │
┌────────────────────┴───────┐ ┌───────────────────┐
│ smb-crm (Flask, backoffice)│ │ Google Sheets │
│ onboard.mivanchenko.de/crm │ │ (human overview, │
│ dashboard + CRUD + iCal │ │ Looker Studio) │
└────────────────────▲─────────┘ └───────────────────┘
SOLE SOURCE OF TRUTH
└───────────────▲──────────────────────────────┘
│ SQL
JSON API │
┌────────────────────┴───────┐
│ smb-crm (Flask, backoffice)│
│ onboard.mivanchenko.de/crm │
│ dashboard + CRUD + iCal │
└────────────────────▲─────────┘
│ HTTP (CRM_API_TOKEN)
┌────────────────────┴─────────────────────────────┐
@@ -49,10 +49,9 @@ every client regardless of tier.
└───────────────────────────┘ └───────────────────────────┘
```
**Postgres is the source of truth.** Google Sheets used to be authoritative; the back-office
rewrite (2026-06-25) flipped that — the DB now owns writes, and Sheets is a best-effort,
one-way projection kept for the human-readable overview and the Looker Studio dashboard. See
`backoffice/app/db.py` (schema/contract), `backoffice/app/sheets.py` (mirror client).
**Postgres is the sole source of truth.** Google Sheets used to be authoritative, then (from
2026-06-25) became a best-effort downstream mirror; that mirror was removed entirely (#13) —
there is no second data path anymore. See `backoffice/app/db.py` (schema/contract).
Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a
shared external `proxy` Docker network. Compose groups (§4) are independent projects that only
@@ -61,7 +60,7 @@ share that network — see `deploy/clients/README.md` for the full picture.
## 3. Data model
Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into
`backoffice/db/init.sql` (Postgres) and Google Sheets tabs (`sheets/SCHEMA.md`):
`backoffice/db/init.sql` (Postgres):
| Entity | PK | Purpose |
|---|---|---|
@@ -71,11 +70,10 @@ Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 in
| `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. |
| `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). |
| `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. |
| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. **The one entity excluded from the Sheets mirror** (`mirror: False` in `TABLES`) so secrets never leave Postgres; reading it also requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone). Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |
| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. Reading it requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone), so secrets never leave Postgres. Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |
`db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps,
numbers, booleans) so the CRUD API, the n8n ingest path and the one-time Sheets importer can never
drift on types.
numbers, booleans) so the CRUD API and the n8n ingest path can never drift on types.
## 4. Components
@@ -88,7 +86,6 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al
`renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted.
- `PATCH /api/<entity>/<id>` — partial update, same auth.
- `DELETE /api/<entity>/<id>` — delete, same auth.
- `POST /api/sync` — force a full DB→Sheets resync of every tab (same auth).
- `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't
send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of
the header token). Supports `?client_id=` to scope to one client.
@@ -98,15 +95,10 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al
tab masks the `secret` column by default with a per-row 👁 reveal toggle and a 📋 copy-to-
clipboard button).
Every write is audit-logged to `activity_log` and enqueues an async, best-effort mirror of that
entity (and of `activity_log` itself) into the linked Google Sheet — mirror failures never fail
the DB write (`mirror_async` / `_mirror_worker` in `app.py`).
`import_from_sheets.py` is a one-time, idempotent bootstrap (upsert by PK) used only to seed
Postgres from the pre-existing Sheet; after that the Sheet is purely downstream.
Every write is audit-logged to `activity_log`.
Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from
`backoffice/.env.example` + a service-account JSON mounted at `./secrets/gcp-sa.json`).
`backoffice/.env.example`).
### 4.2 `templates/landing/` — landing pages & demos
Static, self-contained HTML (no build step). `templates/landing/index.html` is the public
@@ -160,19 +152,13 @@ keeps the 14 most recent dumps in `/home/mivanchenko/backups/smb-crm/`.
| `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. |
| `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients``POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA username → `POST /api/credentials` with the EA username **and password**. Fully automates "sign a client" end to end, including capturing the generated password so it isn't lost (it used to be discarded after the EA API call — see `TODO.md`). |
| `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. |
| `renewal-reminder.json` | daily 08:00 schedule | Read `Clients` from Sheets → find renewals due soon → Telegram notify → append a row to `Activity Log`. **Note:** still reads from the Sheets mirror rather than the DB directly — safe today because the mirror is kept current, but a re-point to the DB would remove that indirection. |
| `renewal-reminder.json` | daily 08:00 schedule | `GET /api/clients` → find renewals due soon → Telegram notify → `POST /api/activity_log`. |
n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a
separate, already-running compose stack on the homelab; only the exported workflow definitions
live here.
### 4.6 `sheets/SCHEMA.md` — Sheets mirror spec
Describes the Google Sheets workbook (`SMB-Online — CRM`) that Postgres mirrors into: one tab per
entity, headers matching column names (n8n / the mirror map by header). Also specifies the
**Looker Studio** dashboard built on top of it (pipeline by status, MRR, renewals due in 30 days,
recent leads) — the dashboard itself lives in Google, not in this repo.
### 4.7 `playbooks/` — operator procedure
### 4.6 `playbooks/` — operator procedure
- `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out
(email/DM/phone scripts in German).
- `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the
@@ -187,7 +173,7 @@ its own `.env.example` to copy from:
| File | Fills |
|---|---|
| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN`, `SHEET_ID` (+ a service-account JSON mounted at `secrets/gcp-sa.json`, not example-tracked) |
| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN` |
| `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` |
| `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) |
@@ -196,7 +182,7 @@ Local dev (no build tooling needed anywhere in this repo):
# Demos — self-contained HTML, open directly or serve the folder:
python3 -m http.server 8080 --directory templates/landing
# Back office (needs Postgres + a Google service account):
# Back office (needs Postgres):
docker compose -f backoffice/docker-compose.yml up
```
Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy;
@@ -209,18 +195,13 @@ see `deploy/clients/README.md` and each group's own compose file for the exact r
- The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients
can't send custom headers — treat that token as effectively public-linkable and rotate it if a
feed URL leaks.
- Sheet cell values are defended against formula injection (`_cell()` in `app.py` prefixes
values starting with `=+-@` with a `'`).
- Credentials for client-owned accounts are recorded two ways: `clients.vault_ref` points to a
Vaultwarden item for anything the operator manually stashes there; the `credentials` table
holds secrets the *system itself* generates (currently: the Easy!Appointments provider login
created during onboarding), gated by `X-CRM-Token` even to read and deliberately excluded from
the Sheets mirror. Stored as plaintext in Postgres today — same trust boundary as the rest of
the CRM (Caddy basic-auth + host security); revisit with column-level encryption (pgcrypto) if
the dashboard is ever exposed more broadly (see `TODO.md`).
- The DB→Sheets mirror is clear-then-write, not atomic — a reader can theoretically catch a
cleared tab mid-sync. Accepted as low-risk (human overview, not a system of record) — see
`TODO.md`.
created during onboarding), gated by `X-CRM-Token` even to read. Stored as plaintext in
Postgres today — same trust boundary as the rest of the CRM (Caddy basic-auth + host security);
revisit with column-level encryption (pgcrypto) if the dashboard is ever exposed more broadly
(see `TODO.md`).
## 7. Legal / positioning constraint
The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays
+3 -6
View File
@@ -2,9 +2,8 @@
A productized service that gets local small businesses online: **landing page + online
booking + lead capture + small automations**. Built to be **thoroughly tracked and logged**
every client, lead, booking and invoice lives in a **Postgres CRM** (source of truth), driven by
**n8n** on the homelab, with a one-way mirror into Google Sheets for a human-readable overview /
Looker Studio dashboard.
every client, lead, booking and invoice lives in a **Postgres CRM** (sole source of truth),
driven by **n8n** on the homelab.
> Full documentation (architecture, data model, every component): **`Documentation.md`**.
@@ -29,7 +28,6 @@ deploy/
backup/ Daily Postgres backup script (cron on the homelab)
n8n/ Exported n8n workflows (lead-intake, onboarding, booking-sync, renewal-reminder)
playbooks/ Operator playbooks (outreach, lead-to-customer, tier-a, tier-b)
sheets/SCHEMA.md Google Sheets mirror spec (tabs + columns) — downstream of Postgres
templates/landing/ Landing-page demos + client preview pages
demo-tier-a/ "Schnittpunkt" barbershop demo
demo-tier-b/ "PhysioVital" physio practice demo
@@ -65,8 +63,7 @@ Tracked against actual repo state — see `Documentation.md` for what backs each
- [x] **Phase 1** — Demo landing pages
- [x] **Phase 2** — Accounts & setup: Google account + workbook, n8n live, Google creds bound,
Vaultwarden (`clients.vault_ref`), domain (wildcard `*.mivanchenko.de`)
- [x] **Phase 3** — CRM (now Postgres-first, Sheets mirrored) + n8n lead-intake & renewal
workflows — Looker Studio dashboard itself lives in Google and isn't repo-verifiable
- [x] **Phase 3** — CRM (Postgres-only, no Sheets mirror) + n8n lead-intake & renewal workflows
- [x] **Phase 4** — Demo forms + booking wired live to n8n webhooks (lead-intake, booking-sync)
- [ ] **Phase 5** — First real client + reception v1 — onboarding automation and per-client
deploy tooling exist and are exercised (`deploy/clients/`); not verifiable from the repo
+6 -3
View File
@@ -12,7 +12,6 @@
**CRM / back office**
- **#2** Decouple tier from business type → hosting/plan model; realign onboarding form + workflow. `enhancement` `crm`
- **#3** Extend back-office CRUD beyond Leads & Clients (projects, bookings, invoices, activity). `enhancement` `crm`
- **#5** Make the DB→Sheets mirror atomic (no brief empty-sheet window). `tech-debt` `crm`
**Booking**
- **#7** Proper Orders flow + Orders table/tab (pizzeria demo posts into Leads as a stop-gap). `enhancement` `booking`
@@ -41,8 +40,12 @@
exposed more broadly than Caddy basic-auth + host security.
## Done
- [x] 2026-07-15**Credentials in the CRM**: `credentials` entity (never mirrored to Sheets,
`X-CRM-Token`-gated even to read), dashboard Credentials tab with masked value + reveal/copy;
- [x] 2026-08-04**Cut Google Sheets entirely** (#13): Postgres is now the sole source of
truth — deleted the DB→Sheets mirror, the Sheets client, and the one-time importer;
`renewal-reminder.json` reads from the DB API instead of the Sheets mirror. Moots #5
(atomic-mirror tech-debt, closed as no longer applicable).
- [x] 2026-07-15 — **Credentials in the CRM**: `credentials` entity (`X-CRM-Token`-gated even to
read), dashboard Credentials tab with masked value + reveal/copy;
`n8n/onboarding.json` now persists the auto-generated EA provider password. Shipped live.
- [x] 2026-07-15 — **Gitea Actions deploy pipeline** (backoffice): self-hosted `homelab-runner`
(`act-runner` systemd service), `workflow_dispatch` `deploy-backoffice.yml` that syncs
-1
View File
@@ -5,7 +5,6 @@ BOOKING_TOKEN_SECRET=change-me-long-random-too
# Owner-login session cookie signing key (#19). Dedicated secret -- rotating
# it just logs owners out, without touching CRM_API_TOKEN/BOOKING_TOKEN_SECRET.
SESSION_SECRET_KEY=change-me-long-random-session-too
SHEET_ID=1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8
# Booking confirmation email (#18). Left blank, sending is skipped (logged,
# not fatal) -- mail relay setup is a separate infra/triage item.
SMTP_HOST=
+1 -92
View File
@@ -1,16 +1,12 @@
"""smb-crm back-office service.
Postgres is the source of truth. This serves the operator dashboard + a small
JSON API (read now; add/edit/delete and the DB->Sheets mirror layered on next).
Postgres is the source of truth for the operator dashboard + a small JSON API.
Browser access is gated by Caddy basic-auth on onboard.mivanchenko.de; the
machine-to-machine ingest path (n8n) is gated by the CRM_API_TOKEN header.
"""
import os
import re
import time
import queue
import threading
import traceback
from datetime import datetime, date, timezone
from decimal import Decimal
@@ -20,7 +16,6 @@ from waitress import serve
import booking_db as bdb
import db
import owner_mail
from sheets import Sheets
from booking_api import bp as booking_bp
from public_booking import bp as public_booking_bp
from manage_booking import bp as manage_booking_bp
@@ -47,69 +42,6 @@ CRM_TOKEN = os.environ.get("CRM_API_TOKEN", "")
# the basic-auth header, so the feed is gated by this query token instead).
ICS_TOKEN = os.environ.get("ICS_TOKEN", "")
# DB -> Sheets one-way mirror. Postgres is the source of truth; the Sheet is a
# best-effort projection. A mirror failure never fails the DB write.
SH = None
try:
SH = Sheets(os.environ["GOOGLE_SA_JSON"], os.environ["SHEET_ID"])
except Exception: # noqa: BLE001
traceback.print_exc()
def _cell(v):
if v is None:
return ""
if isinstance(v, bool):
return "TRUE" if v else "FALSE"
if isinstance(v, datetime):
return v.strftime("%Y-%m-%d %H:%M")
if isinstance(v, date):
return v.strftime("%Y-%m-%d")
if isinstance(v, Decimal):
f = float(v)
return str(int(f)) if f == int(f) else str(f)
s = str(v)
return ("'" + s) if s[:1] in "=+-@" else s # neutralise formula injection
def mirror_entity(entity):
spec = db.TABLES[entity]
if SH is None or spec.get("mirror") is False:
return 0
cols = spec["cols"]
order = LIST_ORDER.get(entity, spec["pk"])
with db.connect() as conn, conn.cursor() as cur:
cur.execute(f"SELECT {', '.join(cols)} FROM {entity} ORDER BY {order}")
rows = cur.fetchall()
grid = [[_cell(r[c]) for c in cols] for r in rows]
SH.overwrite(spec["tab"], cols, grid)
return len(grid)
# Serialize all mirror writes through one worker so concurrent mutations can't
# race on the shared Sheets client / token.
_mirror_q = queue.Queue()
def _mirror_worker():
while True:
entity = _mirror_q.get()
try:
mirror_entity(entity)
except Exception: # noqa: BLE001
print(f"[mirror] {entity} sync failed:", flush=True)
traceback.print_exc()
finally:
_mirror_q.task_done()
threading.Thread(target=_mirror_worker, daemon=True).start()
def mirror_async(entity):
if SH is not None and db.TABLES[entity].get("mirror") is not False:
_mirror_q.put(entity)
# Static dashboard, with the CRM token injected so the (basic-auth-gated)
# operator page can call the token-protected mutation endpoints.
with open(os.path.join(os.path.dirname(__file__), "static", "index.html")) as _f:
@@ -235,8 +167,6 @@ def add_entity(entity):
[row.get(c) for c in cols])
log_activity(cur, row.get("client_id"), f"add {entity}", f"{pk}={row.get(pk)}")
conn.commit()
mirror_async(entity)
mirror_async("activity_log")
return jsonify({"added": row.get(pk)}), 201
@@ -262,8 +192,6 @@ def edit_entity(entity, ident):
log_activity(cur, ident if entity == "clients" else None,
f"edit {entity}", f"{pk}={ident}: {', '.join(setcols)}")
conn.commit()
mirror_async(entity)
mirror_async("activity_log")
return jsonify({"updated": ident, "fields": setcols})
@@ -281,8 +209,6 @@ def delete_entity(entity, ident):
log_activity(cur, ident if entity == "clients" else None,
f"delete {entity}", f"{pk}={ident}")
conn.commit()
mirror_async(entity)
mirror_async("activity_log")
return jsonify({"deleted": ident})
@@ -313,26 +239,9 @@ def trigger_owner_password_reset(user_id):
log_activity(cur, user["client_id"], "owner password reset",
f"user_id={user_id} (operator-triggered)")
conn.commit()
mirror_async("activity_log")
return jsonify({"sent": user_id})
@app.post("/api/sync")
def sync_all():
"""Full DB -> Sheets resync of every tab (manual / alignment)."""
if not authed():
return jsonify({"error": "forbidden"}), 403
if SH is None:
return jsonify({"error": "sheets unavailable"}), 503
out = {}
for entity in db.TABLES:
try:
out[entity] = mirror_entity(entity)
except Exception as e: # noqa: BLE001
out[entity] = f"error: {e}"
return jsonify({"synced": out})
def _ics_dt(v):
if isinstance(v, datetime):
u = v if v.tzinfo else v.replace(tzinfo=timezone.utc)
+5 -12
View File
@@ -1,8 +1,8 @@
"""Postgres access + the table/column contract shared across the app.
The TABLES map is the single definition of which entities exist, their columns,
and their primary key. Read/CRUD endpoints, the importer and the Sheets mirror
all derive from it so they can never drift apart.
and their primary key. Read/CRUD endpoints all derive from it so they can
never drift apart.
"""
import os
import re
@@ -13,10 +13,9 @@ from psycopg.rows import dict_row
DATABASE_URL = os.environ["DATABASE_URL"]
# entity -> (sheet tab, primary key, ordered columns)
# entity -> (primary key, ordered columns)
TABLES = {
"clients": {
"tab": "Clients",
"pk": "client_id",
"cols": ["client_id", "business_name", "owner_name", "email", "phone",
"niche", "tier", "status", "domain", "stack_notes", "vault_ref",
@@ -28,7 +27,6 @@ TABLES = {
"bools": [],
},
"leads": {
"tab": "Leads",
"pk": "lead_id",
"cols": ["lead_id", "received_at", "client_id", "source", "name",
"contact", "service_interest", "message", "status", "notified"],
@@ -38,7 +36,6 @@ TABLES = {
"bools": ["notified"],
},
"projects": {
"tab": "Projects",
"pk": "project_id",
"cols": ["project_id", "client_id", "deliverable", "tier", "checklist",
"go_live_date", "status"],
@@ -48,7 +45,6 @@ TABLES = {
"bools": [],
},
"bookings": {
"tab": "Bookings",
"pk": "booking_id",
"cols": ["booking_id", "created_at", "client_id", "customer_name",
"customer_contact", "service", "start_time", "end_time",
@@ -59,7 +55,6 @@ TABLES = {
"bools": [],
},
"invoices": {
"tab": "Invoices",
"pk": "invoice_id",
"cols": ["invoice_id", "client_id", "issued_date", "due_date",
"amount_eur", "period", "status", "paid_date"],
@@ -69,7 +64,6 @@ TABLES = {
"bools": [],
},
"activity_log": {
"tab": "Activity Log",
"pk": "id",
"cols": ["ts", "workflow", "client_id", "action", "detail", "result"],
"dates": [],
@@ -78,14 +72,13 @@ TABLES = {
"bools": [],
},
"credentials": {
# No "tab": never mirrored to Sheets (see "mirror" below) — secrets stay in Postgres only.
# Secrets stay in Postgres only.
"pk": "cred_id",
"cols": ["cred_id", "client_id", "label", "username", "secret", "notes", "created_at"],
"dates": [],
"timestamps": ["created_at"],
"numbers": [],
"bools": [],
"mirror": False,
},
}
@@ -94,7 +87,7 @@ def connect():
return psycopg.connect(DATABASE_URL, row_factory=dict_row)
# ---- coercion: Sheet strings / JSON values -> typed Python for Postgres ----
# ---- coercion: JSON request values -> typed Python for Postgres ----
def parse_date(v):
if v in (None, ""):
-57
View File
@@ -1,57 +0,0 @@
"""One-time (idempotent) seed of Postgres from the existing Google Sheets.
This is the ONLY place the Sheet is treated as authoritative — to bootstrap the
DB. After this, Postgres is the source of truth and Sheets is a downstream
mirror. Safe to re-run: keyed tables upsert by primary key; activity_log is
replaced wholesale.
Run inside the container: docker compose exec smb-crm python import_from_sheets.py
"""
import os
import sys
import db
from sheets import Sheets
def upsert(cur, entity, records):
spec = db.TABLES[entity]
cols = spec["cols"]
pk = spec["pk"]
n = 0
for rec in records:
row = db.coerce_row(entity, rec)
if entity != "activity_log" and not row.get(pk):
continue # skip rows without a primary key
placeholders = ", ".join(["%s"] * len(cols))
collist = ", ".join(cols)
if entity == "activity_log":
cur.execute(f"INSERT INTO {entity} ({collist}) VALUES ({placeholders})",
[row[c] for c in cols])
else:
updates = ", ".join(f"{c} = EXCLUDED.{c}" for c in cols if c != pk)
cur.execute(
f"INSERT INTO {entity} ({collist}) VALUES ({placeholders}) "
f"ON CONFLICT ({pk}) DO UPDATE SET {updates}",
[row[c] for c in cols])
n += 1
return n
def main():
sh = Sheets(os.environ["GOOGLE_SA_JSON"], os.environ["SHEET_ID"])
with db.connect() as conn:
with conn.cursor() as cur:
cur.execute("TRUNCATE activity_log RESTART IDENTITY")
for entity, spec in db.TABLES.items():
if spec.get("mirror") is False:
continue # e.g. credentials — never lived in the Sheet, nothing to import
records = sh.read_records(spec["tab"])
n = upsert(cur, entity, records)
print(f" {entity:<13} <- {spec['tab']:<13} {n} row(s)")
conn.commit()
print("import complete.")
if __name__ == "__main__":
sys.exit(main())
+3 -3
View File
@@ -1,8 +1,8 @@
"""Best-effort SMTP email sending (#18).
Mirrors app.py's DB -> Sheets mirror pattern: a single background worker
thread drains a queue, and a send failure is logged, never raised back to the
caller -- booking creation must succeed even if the mail relay is down.
A single background worker thread drains a queue, and a send failure is
logged, never raised back to the caller -- booking creation must succeed even
if the mail relay is down.
"""
import os
import queue
-1
View File
@@ -8,7 +8,6 @@ Flask==3.0.3
psycopg[binary]==3.2.1
waitress==3.0.0
PyJWT[crypto]==2.9.0
requests==2.32.3
# python:3.12-slim has no system IANA tz database; zoneinfo (used by
# availability.py for Europe/Berlin) falls back to this package for it.
tzdata==2024.1
-98
View File
@@ -1,98 +0,0 @@
"""Minimal Google Sheets v4 client using a service-account JWT.
Used for (a) the one-time import of existing Sheet rows into Postgres and
(b) the one-way DB -> Sheets mirror sync. Postgres stays the source of truth;
nothing here treats the Sheet as authoritative except the explicit import.
"""
import json
import time
import threading
import jwt
import requests
SHEETS_API = "https://sheets.googleapis.com/v4/spreadsheets"
TOKEN_URL = "https://oauth2.googleapis.com/token"
SCOPE = "https://www.googleapis.com/auth/spreadsheets"
# gid map for the live workbook (tab name -> gid), used when clearing/sizing.
TAB_GIDS = {
"Clients": 470934735,
"Leads": 571719114,
"Bookings": 946084008,
"Projects": 619112786,
"Invoices": 413377229,
"Activity Log": 170940481,
}
class Sheets:
def __init__(self, sa_json_path, sheet_id):
with open(sa_json_path) as f:
self.sa = json.load(f)
self.sheet_id = sheet_id
self._tok = None
self._exp = 0
self._lock = threading.Lock()
def _token(self):
with self._lock:
now = int(time.time())
if self._tok and now < self._exp - 60:
return self._tok
claim = {
"iss": self.sa["client_email"],
"scope": SCOPE,
"aud": TOKEN_URL,
"iat": now,
"exp": now + 3600,
}
assertion = jwt.encode(claim, self.sa["private_key"], algorithm="RS256")
r = requests.post(TOKEN_URL, data={
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": assertion,
}, timeout=30)
r.raise_for_status()
self._tok = r.json()["access_token"]
self._exp = now + 3600
return self._tok
def _headers(self):
return {"Authorization": "Bearer " + self._token()}
def read(self, a1_range):
"""Return raw 2D list of cell values for an A1 range (e.g. "Leads!A1:Z")."""
url = f"{SHEETS_API}/{self.sheet_id}/values/{requests.utils.quote(a1_range)}"
r = requests.get(url, headers=self._headers(), timeout=30)
r.raise_for_status()
return r.json().get("values", [])
def read_records(self, tab):
"""Read a whole tab as a list of dicts keyed by the header row."""
rows = self.read(f"{tab}!A1:Z")
if not rows:
return []
header = rows[0]
out = []
for raw in rows[1:]:
if not any(c.strip() for c in raw):
continue
rec = {header[i]: (raw[i] if i < len(raw) else "") for i in range(len(header))}
out.append(rec)
return out
def overwrite(self, tab, header, rows):
"""Replace a tab's contents with header + rows (the DB->Sheets mirror).
Clears the existing value range, then writes the new grid starting at A1.
Postgres is the source of truth; this projects it onto the Sheet.
"""
# clear current values (keeps formatting / the tab itself)
clr = f"{SHEETS_API}/{self.sheet_id}/values/{requests.utils.quote(tab + '!A1:Z')}:clear"
requests.post(clr, headers=self._headers(), timeout=30).raise_for_status()
body = {"values": [header] + rows}
url = (f"{SHEETS_API}/{self.sheet_id}/values/"
f"{requests.utils.quote(tab + '!A1')}?valueInputOption=RAW")
r = requests.put(url, headers=self._headers(), json=body, timeout=60)
r.raise_for_status()
return len(rows)
+1 -1
View File
@@ -74,7 +74,7 @@
<h1>🗂️ smb-crm · Back Office</h1>
<div class="hgroup">
<a class="btn ghost" href="/">📝 Onboarding-Formular</a>
<span class="src">Quelle: Postgres (Sheets = Spiegel) · <span id="now"></span></span>
<span class="src">Quelle: Postgres · <span id="now"></span></span>
</div>
</header>
<div class="wrap">
+1 -2
View File
@@ -1,7 +1,6 @@
"""Unit tests for mailer.py's SMTP call shape (#18). Calls _send_now directly
rather than going through the background-thread queue, so assertions are
synchronous -- the queue itself is just plumbing, already covered indirectly
by app.py's identical Sheets-mirror pattern.
synchronous -- the queue itself is just plumbing.
"""
from email.message import EmailMessage
+3 -5
View File
@@ -1,6 +1,4 @@
-- smb-crm — source-of-truth schema (Postgres).
-- Google Sheets is a one-way downstream mirror fed by the DB->Sheets sync.
-- Column sets mirror the Sheet tabs so the mirror stays 1:1.
-- smb-crm — sole source-of-truth schema (Postgres).
CREATE TABLE IF NOT EXISTS clients (
client_id text PRIMARY KEY,
@@ -87,8 +85,8 @@ CREATE TABLE IF NOT EXISTS invoices (
);
-- Per-client login credentials (e.g. the auto-generated Easy!Appointments
-- provider login). Deliberately NOT mirrored to Sheets — see db.py TABLES
-- ("mirror": False) — so secrets never leave Postgres.
-- provider login). Gated by X-CRM-Token even to read, so secrets never leave
-- Postgres.
CREATE TABLE IF NOT EXISTS credentials (
cred_id text PRIMARY KEY,
client_id text,
-4
View File
@@ -27,16 +27,12 @@ services:
BOOKING_TOKEN_SECRET: ${BOOKING_TOKEN_SECRET}
SESSION_SECRET_KEY: ${SESSION_SECRET_KEY}
ICS_TOKEN: ${ICS_TOKEN}
SHEET_ID: ${SHEET_ID}
GOOGLE_SA_JSON: /run/secrets/gcp-sa.json
SMTP_HOST: ${SMTP_HOST}
SMTP_PORT: ${SMTP_PORT}
SMTP_USERNAME: ${SMTP_USERNAME}
SMTP_PASSWORD: ${SMTP_PASSWORD}
MAIL_FALLBACK_FROM: ${MAIL_FALLBACK_FROM}
PUBLIC_BASE_URL: ${PUBLIC_BASE_URL}
volumes:
- ./secrets/gcp-sa.json:/run/secrets/gcp-sa.json:ro
depends_on:
smb-db:
condition: service_healthy
+26 -57
View File
@@ -1,5 +1,5 @@
{
"name": "SMB \u00b7 Renewal Reminder",
"name": "SMB · Renewal Reminder",
"nodes": [
{
"parameters": {
@@ -23,43 +23,24 @@
},
{
"parameters": {
"authentication": "serviceAccount",
"resource": "sheet",
"operation": "read",
"documentId": {
"__rl": true,
"value": "1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8",
"mode": "id"
},
"sheetName": {
"__rl": true,
"value": "470934735",
"mode": "list",
"cachedResultName": "Clients"
},
"url": "http://smb-crm:8080/api/clients",
"options": {}
},
"id": "f06a60af-dff1-415d-aca6-fef0367f3553",
"name": "Read Clients",
"type": "n8n-nodes-base.googleSheets",
"typeVersion": 4.5,
"name": "Read Clients (DB)",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
460,
300
],
"credentials": {
"googleApi": {
"id": "45TWPyl1wVk2Vxdm",
"name": "SMB Google (service account)"
}
},
"retryOnFail": true,
"maxTries": 4,
"waitBetweenTries": 3000
},
{
"parameters": {
"jsCode": "\nconst rows = $input.all().map(i=>i.json);\nconst today=new Date();\nconst t0=Date.UTC(today.getUTCFullYear(),today.getUTCMonth(),today.getUTCDate());\nfunction parseDate(v){\n if(v==null||v==='') return null;\n const s=String(v).trim();\n if(/^\\d+(\\.\\d+)?$/.test(s)){ return Date.UTC(1899,11,30)+Number(s)*86400000; }\n const m=s.match(/^(\\d{4})-(\\d{2})-(\\d{2})/);\n if(m) return Date.UTC(+m[1],+m[2]-1,+m[3]);\n const d=Date.parse(s); return isNaN(d)?null:d;\n}\nconst offsets=[14,7,3,1,0];\nconst due=[];\nfor(const r of rows){\n const status=String(r.status||'').toLowerCase();\n if(status!=='active') continue;\n const rd=parseDate(r.renewal_date);\n if(rd==null) continue;\n const days=Math.round((rd-t0)/86400000);\n if(days<0 || offsets.includes(days)){\n due.push({client_id:r.client_id, business:r.business_name, renewal_date:r.renewal_date,\n days, fee:r.monthly_fee_eur, cycle:r.billing_cycle});\n }\n}\nif(due.length===0) return [];\ndue.sort((a,b)=>a.days-b.days);\nconst line=d=>`\u2022 ${d.business||d.client_id} \u2014 ${d.days<0?('\u00dcBERF\u00c4LLIG seit '+(-d.days)+'T'):(d.days===0?'heute':('in '+d.days+'T'))} (${d.renewal_date}${d.fee?(', '+d.fee+'\u20ac/'+(d.cycle||'')):''})`;\nconst summary='\ud83d\udd14 Anstehende Verl\u00e4ngerungen ('+due.length+')\\n'+due.map(line).join('\\n');\nreturn [{json:{count:due.length, summary, due}}];\n"
"jsCode": "\nconst rows = $input.first().json.rows || [];\nconst today=new Date();\nconst t0=Date.UTC(today.getUTCFullYear(),today.getUTCMonth(),today.getUTCDate());\nfunction parseDate(v){\n if(v==null||v==='') return null;\n const s=String(v).trim();\n const m=s.match(/^(\\d{4})-(\\d{2})-(\\d{2})/);\n if(m) return Date.UTC(+m[1],+m[2]-1,+m[3]);\n const d=Date.parse(s); return isNaN(d)?null:d;\n}\nconst offsets=[14,7,3,1,0];\nconst due=[];\nfor(const r of rows){\n const status=String(r.status||'').toLowerCase();\n if(status!=='active') continue;\n const rd=parseDate(r.renewal_date);\n if(rd==null) continue;\n const days=Math.round((rd-t0)/86400000);\n if(days<0 || offsets.includes(days)){\n due.push({client_id:r.client_id, business:r.business_name, renewal_date:r.renewal_date,\n days, fee:r.monthly_fee_eur, cycle:r.billing_cycle});\n }\n}\nif(due.length===0) return [];\ndue.sort((a,b)=>a.days-b.days);\nconst line=d=>` ${d.business||d.client_id} ${d.days<0?('ÜBERFÄLLIG seit '+(-d.days)+'T'):(d.days===0?'heute':('in '+d.days+'T'))} (${d.renewal_date}${d.fee?(', '+d.fee+'/'+(d.cycle||'')):''})`;\nconst summary='🔔 Anstehende Verlängerungen ('+due.length+')\\n'+due.map(line).join('\\n');\nreturn [{json:{count:due.length, summary, due}}];\n"
},
"id": "9f3e4ca9-0d87-4d1c-9f5e-abdf5f9f77dd",
"name": "Find due renewals",
@@ -110,42 +91,30 @@
},
{
"parameters": {
"authentication": "serviceAccount",
"resource": "sheet",
"operation": "append",
"documentId": {
"__rl": true,
"value": "1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8",
"mode": "id"
},
"sheetName": {
"__rl": true,
"value": "170940481",
"mode": "list",
"cachedResultName": "Activity Log"
},
"columns": {
"mappingMode": "autoMapInputData",
"value": {},
"matchingColumns": [],
"schema": []
"method": "POST",
"url": "http://smb-crm:8080/api/activity_log",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{
"name": "X-CRM-Token",
"value": "__CRM_TOKEN__"
}
]
},
"sendBody": true,
"specifyBody": "json",
"jsonBody": "={{ JSON.stringify($json) }}",
"options": {}
},
"id": "55978962-489b-4740-a242-7470f08087b6",
"name": "Append to Activity Log",
"type": "n8n-nodes-base.googleSheets",
"typeVersion": 4.5,
"name": "Append to Activity Log (DB)",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"position": [
1340,
300
],
"credentials": {
"googleApi": {
"id": "45TWPyl1wVk2Vxdm",
"name": "SMB Google (service account)"
}
},
"retryOnFail": true,
"maxTries": 4,
"waitBetweenTries": 3000
@@ -156,14 +125,14 @@
"main": [
[
{
"node": "Read Clients",
"node": "Read Clients (DB)",
"type": "main",
"index": 0
}
]
]
},
"Read Clients": {
"Read Clients (DB)": {
"main": [
[
{
@@ -200,7 +169,7 @@
"main": [
[
{
"node": "Append to Activity Log",
"node": "Append to Activity Log (DB)",
"type": "main",
"index": 0
}
@@ -213,4 +182,4 @@
"callerPolicy": "workflowsFromSameOwner",
"availableInMCP": false
}
}
}
+1 -1
View File
@@ -19,7 +19,7 @@ ongoing management. Target build time: ~half a day per client.
6. **Reception** — Gmail for the business; set a signature + canned replies.
7. **Google Business Profile** — claim/verify so they show up on Maps & Search.
8. **CRM** — add a `Clients` row (tier = A, services, monthly_fee, renewal_date, vault_ref).
9. **Handover** — share logins via Vaultwarden; record only the `vault_ref` in the sheet.
9. **Handover** — share logins via Vaultwarden; record only the `vault_ref` in the CRM.
## What to charge (suggestion)
- One-off setup: €250600 depending on pages/content.
+1 -1
View File
@@ -16,7 +16,7 @@ German-hosted infrastructure. You charge a privacy/hosting premium. Target build
4. **Lead form** — self-hosted form endpoint → n8n webhook → CRM. No third-party form service.
5. **Reception** — business mailbox; (later) the **AI receptionist** widget (Phase 6, Claude
Haiku 4.5) wired to the booking + FAQ.
6. **Credentials** — store all client logins in **Vaultwarden**; sheet holds only `vault_ref`.
6. **Credentials** — store all client logins in **Vaultwarden**; the CRM holds only `vault_ref`.
7. **CRM** — add a `Clients` row (tier = B, services, monthly_fee, renewal_date, vault_ref).
8. **Backups** — ensure the Tier-B services + CRM are in the daily backup job.
-63
View File
@@ -1,63 +0,0 @@
# Master CRM — Google Sheets workbook spec
One Google Sheets workbook = the whole CRM + log. I create/maintain this via the Google Drive MCP
once the fresh project Google account exists (Phase 2). n8n reads/writes it via the Sheets node.
Workbook name: **`SMB-Online — CRM`**. One tab per concern below. Row 1 = headers (exact column
names matter — n8n maps by header).
## Tab: `Clients`
The client list/table — the heart of the tracking requirement.
| Column | Type | Notes |
|---|---|---|
| `client_id` | text | e.g. `C-0001` (stable key) |
| `business_name` | text | |
| `owner_name` | text | |
| `email` | text | |
| `phone` | text | |
| `niche` | text | barber, physio, café, … |
| `tier` | A \| B | delivery tier |
| `status` | enum | `lead``onboarding``active``paused``churned` |
| `domain` | text | live site URL |
| `stack_notes` | text | hosting, booking tool, etc. |
| `vault_ref` | text | **pointer** to Vaultwarden item — never a password |
| `services` | text | comma list: site, booking, reception, … |
| `billing_cycle` | enum | monthly / yearly / one-off |
| `monthly_fee_eur` | number | |
| `start_date` | date | |
| `renewal_date` | date | drives the renewal-reminder workflow |
| `created_at` | datetime | |
| `notes` | text | |
## Tab: `Leads`
Every inbound lead across all client sites (and your own).
`lead_id` · `received_at` · `client_id` · `source` (which site/form) · `name` · `contact` ·
`service_interest` · `message` · `status` (new / contacted / won / lost) · `notified` (bool)
## Tab: `Bookings`
Every appointment booked across client sites.
`booking_id` · `created_at` · `client_id` · `customer_name` · `customer_contact` · `service` ·
`start_time` · `end_time` · `source` (gcal / calcom) · `status` (confirmed / cancelled / no-show)
## Tab: `Projects`
Per-client deliverable tracking.
`project_id` · `client_id` · `deliverable` · `tier` · `checklist` (e.g. `domain✓ page✓ booking☐`) ·
`go_live_date` · `status` (todo / in-progress / live)
## Tab: `Invoices`
`invoice_id` · `client_id` · `issued_date` · `due_date` · `amount_eur` · `period` ·
`status` (draft / sent / paid / overdue) · `paid_date`
## Tab: `Activity Log`
Append-only audit trail — **every automated action writes one row here.**
`ts` · `workflow` (lead-intake / booking-sync / renewal / onboarding / invoice) · `client_id` ·
`action` · `detail` · `result` (ok / error)
## Dashboard
Not a tab — a **Looker Studio** report wired to this workbook: pipeline by `status`, MRR
(sum of `monthly_fee_eur` where `status=active`), renewals due in next 30 days, recent leads.