Remove Google Sheets mirror entirely (#13)
Test backoffice (smb-crm) / test (push) Has been cancelled
Test backoffice (smb-crm) / test (push) Has been cancelled
Postgres is now the sole source of truth: delete sheets.py and import_from_sheets.py, strip mirror_entity/mirror_async/_mirror_worker and POST /api/sync from app.py, drop the tab/mirror keys from db.py's TABLES. Re-point n8n/renewal-reminder.json at the CRM's own HTTP API (GET /api/clients, POST /api/activity_log) instead of the Sheets nodes, and drop SHEET_ID/GOOGLE_SA_JSON from deploy env/compose and requests from requirements.txt (PyJWT stays — still used by booking_api.py). Updates docs/README/playbooks accordingly and closes the old #5 (atomic mirror) as moot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+25
-44
@@ -23,15 +23,15 @@ every client regardless of tier.
|
|||||||
│ Postgres (smb-db) │
|
│ Postgres (smb-db) │
|
||||||
│ clients · leads · projects · bookings · │
|
│ clients · leads · projects · bookings · │
|
||||||
│ invoices · activity_log · credentials — │
|
│ invoices · activity_log · credentials — │
|
||||||
│ SOURCE OF TRUTH (credentials never mirror) │
|
│ SOLE SOURCE OF TRUTH │
|
||||||
└───────────────▲───────────────┬────────────┘
|
└───────────────▲──────────────────────────────┘
|
||||||
│ SQL │ one-way mirror
|
│ SQL
|
||||||
JSON API │ ▼
|
JSON API │
|
||||||
┌────────────────────┴───────┐ ┌───────────────────┐
|
┌────────────────────┴───────┐
|
||||||
│ smb-crm (Flask, backoffice)│ │ Google Sheets │
|
│ smb-crm (Flask, backoffice)│
|
||||||
│ onboard.mivanchenko.de/crm │ │ (human overview, │
|
│ onboard.mivanchenko.de/crm │
|
||||||
│ dashboard + CRUD + iCal │ │ Looker Studio) │
|
│ dashboard + CRUD + iCal │
|
||||||
└────────────────────▲─────────┘ └───────────────────┘
|
└────────────────────▲─────────┘
|
||||||
│ HTTP (CRM_API_TOKEN)
|
│ HTTP (CRM_API_TOKEN)
|
||||||
│
|
│
|
||||||
┌────────────────────┴─────────────────────────────┐
|
┌────────────────────┴─────────────────────────────┐
|
||||||
@@ -49,10 +49,9 @@ every client regardless of tier.
|
|||||||
└───────────────────────────┘ └───────────────────────────┘
|
└───────────────────────────┘ └───────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
**Postgres is the source of truth.** Google Sheets used to be authoritative; the back-office
|
**Postgres is the sole source of truth.** Google Sheets used to be authoritative, then (from
|
||||||
rewrite (2026-06-25) flipped that — the DB now owns writes, and Sheets is a best-effort,
|
2026-06-25) became a best-effort downstream mirror; that mirror was removed entirely (#13) —
|
||||||
one-way projection kept for the human-readable overview and the Looker Studio dashboard. See
|
there is no second data path anymore. See `backoffice/app/db.py` (schema/contract).
|
||||||
`backoffice/app/db.py` (schema/contract), `backoffice/app/sheets.py` (mirror client).
|
|
||||||
|
|
||||||
Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a
|
Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a
|
||||||
shared external `proxy` Docker network. Compose groups (§4) are independent projects that only
|
shared external `proxy` Docker network. Compose groups (§4) are independent projects that only
|
||||||
@@ -61,7 +60,7 @@ share that network — see `deploy/clients/README.md` for the full picture.
|
|||||||
## 3. Data model
|
## 3. Data model
|
||||||
|
|
||||||
Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into
|
Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into
|
||||||
`backoffice/db/init.sql` (Postgres) and Google Sheets tabs (`sheets/SCHEMA.md`):
|
`backoffice/db/init.sql` (Postgres):
|
||||||
|
|
||||||
| Entity | PK | Purpose |
|
| Entity | PK | Purpose |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
@@ -71,11 +70,10 @@ Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 in
|
|||||||
| `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. |
|
| `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. |
|
||||||
| `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). |
|
| `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). |
|
||||||
| `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. |
|
| `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. |
|
||||||
| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. **The one entity excluded from the Sheets mirror** (`mirror: False` in `TABLES`) so secrets never leave Postgres; reading it also requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone). Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |
|
| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. Reading it requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone), so secrets never leave Postgres. Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |
|
||||||
|
|
||||||
`db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps,
|
`db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps,
|
||||||
numbers, booleans) so the CRUD API, the n8n ingest path and the one-time Sheets importer can never
|
numbers, booleans) so the CRUD API and the n8n ingest path can never drift on types.
|
||||||
drift on types.
|
|
||||||
|
|
||||||
## 4. Components
|
## 4. Components
|
||||||
|
|
||||||
@@ -88,7 +86,6 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al
|
|||||||
`renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted.
|
`renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted.
|
||||||
- `PATCH /api/<entity>/<id>` — partial update, same auth.
|
- `PATCH /api/<entity>/<id>` — partial update, same auth.
|
||||||
- `DELETE /api/<entity>/<id>` — delete, same auth.
|
- `DELETE /api/<entity>/<id>` — delete, same auth.
|
||||||
- `POST /api/sync` — force a full DB→Sheets resync of every tab (same auth).
|
|
||||||
- `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't
|
- `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't
|
||||||
send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of
|
send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of
|
||||||
the header token). Supports `?client_id=` to scope to one client.
|
the header token). Supports `?client_id=` to scope to one client.
|
||||||
@@ -98,15 +95,10 @@ Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-al
|
|||||||
tab masks the `secret` column by default with a per-row 👁 reveal toggle and a 📋 copy-to-
|
tab masks the `secret` column by default with a per-row 👁 reveal toggle and a 📋 copy-to-
|
||||||
clipboard button).
|
clipboard button).
|
||||||
|
|
||||||
Every write is audit-logged to `activity_log` and enqueues an async, best-effort mirror of that
|
Every write is audit-logged to `activity_log`.
|
||||||
entity (and of `activity_log` itself) into the linked Google Sheet — mirror failures never fail
|
|
||||||
the DB write (`mirror_async` / `_mirror_worker` in `app.py`).
|
|
||||||
|
|
||||||
`import_from_sheets.py` is a one-time, idempotent bootstrap (upsert by PK) used only to seed
|
|
||||||
Postgres from the pre-existing Sheet; after that the Sheet is purely downstream.
|
|
||||||
|
|
||||||
Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from
|
Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from
|
||||||
`backoffice/.env.example` + a service-account JSON mounted at `./secrets/gcp-sa.json`).
|
`backoffice/.env.example`).
|
||||||
|
|
||||||
### 4.2 `templates/landing/` — landing pages & demos
|
### 4.2 `templates/landing/` — landing pages & demos
|
||||||
Static, self-contained HTML (no build step). `templates/landing/index.html` is the public
|
Static, self-contained HTML (no build step). `templates/landing/index.html` is the public
|
||||||
@@ -160,19 +152,13 @@ keeps the 14 most recent dumps in `/home/mivanchenko/backups/smb-crm/`.
|
|||||||
| `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. |
|
| `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. |
|
||||||
| `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients` → `POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA username → `POST /api/credentials` with the EA username **and password**. Fully automates "sign a client" end to end, including capturing the generated password so it isn't lost (it used to be discarded after the EA API call — see `TODO.md`). |
|
| `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients` → `POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA username → `POST /api/credentials` with the EA username **and password**. Fully automates "sign a client" end to end, including capturing the generated password so it isn't lost (it used to be discarded after the EA API call — see `TODO.md`). |
|
||||||
| `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. |
|
| `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. |
|
||||||
| `renewal-reminder.json` | daily 08:00 schedule | Read `Clients` from Sheets → find renewals due soon → Telegram notify → append a row to `Activity Log`. **Note:** still reads from the Sheets mirror rather than the DB directly — safe today because the mirror is kept current, but a re-point to the DB would remove that indirection. |
|
| `renewal-reminder.json` | daily 08:00 schedule | `GET /api/clients` → find renewals due soon → Telegram notify → `POST /api/activity_log`. |
|
||||||
|
|
||||||
n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a
|
n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a
|
||||||
separate, already-running compose stack on the homelab; only the exported workflow definitions
|
separate, already-running compose stack on the homelab; only the exported workflow definitions
|
||||||
live here.
|
live here.
|
||||||
|
|
||||||
### 4.6 `sheets/SCHEMA.md` — Sheets mirror spec
|
### 4.6 `playbooks/` — operator procedure
|
||||||
Describes the Google Sheets workbook (`SMB-Online — CRM`) that Postgres mirrors into: one tab per
|
|
||||||
entity, headers matching column names (n8n / the mirror map by header). Also specifies the
|
|
||||||
**Looker Studio** dashboard built on top of it (pipeline by status, MRR, renewals due in 30 days,
|
|
||||||
recent leads) — the dashboard itself lives in Google, not in this repo.
|
|
||||||
|
|
||||||
### 4.7 `playbooks/` — operator procedure
|
|
||||||
- `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out
|
- `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out
|
||||||
(email/DM/phone scripts in German).
|
(email/DM/phone scripts in German).
|
||||||
- `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the
|
- `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the
|
||||||
@@ -187,7 +173,7 @@ its own `.env.example` to copy from:
|
|||||||
|
|
||||||
| File | Fills |
|
| File | Fills |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN`, `SHEET_ID` (+ a service-account JSON mounted at `secrets/gcp-sa.json`, not example-tracked) |
|
| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN` |
|
||||||
| `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` |
|
| `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` |
|
||||||
| `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) |
|
| `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) |
|
||||||
|
|
||||||
@@ -196,7 +182,7 @@ Local dev (no build tooling needed anywhere in this repo):
|
|||||||
# Demos — self-contained HTML, open directly or serve the folder:
|
# Demos — self-contained HTML, open directly or serve the folder:
|
||||||
python3 -m http.server 8080 --directory templates/landing
|
python3 -m http.server 8080 --directory templates/landing
|
||||||
|
|
||||||
# Back office (needs Postgres + a Google service account):
|
# Back office (needs Postgres):
|
||||||
docker compose -f backoffice/docker-compose.yml up
|
docker compose -f backoffice/docker-compose.yml up
|
||||||
```
|
```
|
||||||
Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy;
|
Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy;
|
||||||
@@ -209,18 +195,13 @@ see `deploy/clients/README.md` and each group's own compose file for the exact r
|
|||||||
- The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients
|
- The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients
|
||||||
can't send custom headers — treat that token as effectively public-linkable and rotate it if a
|
can't send custom headers — treat that token as effectively public-linkable and rotate it if a
|
||||||
feed URL leaks.
|
feed URL leaks.
|
||||||
- Sheet cell values are defended against formula injection (`_cell()` in `app.py` prefixes
|
|
||||||
values starting with `=+-@` with a `'`).
|
|
||||||
- Credentials for client-owned accounts are recorded two ways: `clients.vault_ref` points to a
|
- Credentials for client-owned accounts are recorded two ways: `clients.vault_ref` points to a
|
||||||
Vaultwarden item for anything the operator manually stashes there; the `credentials` table
|
Vaultwarden item for anything the operator manually stashes there; the `credentials` table
|
||||||
holds secrets the *system itself* generates (currently: the Easy!Appointments provider login
|
holds secrets the *system itself* generates (currently: the Easy!Appointments provider login
|
||||||
created during onboarding), gated by `X-CRM-Token` even to read and deliberately excluded from
|
created during onboarding), gated by `X-CRM-Token` even to read. Stored as plaintext in
|
||||||
the Sheets mirror. Stored as plaintext in Postgres today — same trust boundary as the rest of
|
Postgres today — same trust boundary as the rest of the CRM (Caddy basic-auth + host security);
|
||||||
the CRM (Caddy basic-auth + host security); revisit with column-level encryption (pgcrypto) if
|
revisit with column-level encryption (pgcrypto) if the dashboard is ever exposed more broadly
|
||||||
the dashboard is ever exposed more broadly (see `TODO.md`).
|
(see `TODO.md`).
|
||||||
- The DB→Sheets mirror is clear-then-write, not atomic — a reader can theoretically catch a
|
|
||||||
cleared tab mid-sync. Accepted as low-risk (human overview, not a system of record) — see
|
|
||||||
`TODO.md`.
|
|
||||||
|
|
||||||
## 7. Legal / positioning constraint
|
## 7. Legal / positioning constraint
|
||||||
The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays
|
The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays
|
||||||
|
|||||||
@@ -2,9 +2,8 @@
|
|||||||
|
|
||||||
A productized service that gets local small businesses online: **landing page + online
|
A productized service that gets local small businesses online: **landing page + online
|
||||||
booking + lead capture + small automations**. Built to be **thoroughly tracked and logged** —
|
booking + lead capture + small automations**. Built to be **thoroughly tracked and logged** —
|
||||||
every client, lead, booking and invoice lives in a **Postgres CRM** (source of truth), driven by
|
every client, lead, booking and invoice lives in a **Postgres CRM** (sole source of truth),
|
||||||
**n8n** on the homelab, with a one-way mirror into Google Sheets for a human-readable overview /
|
driven by **n8n** on the homelab.
|
||||||
Looker Studio dashboard.
|
|
||||||
|
|
||||||
> Full documentation (architecture, data model, every component): **`Documentation.md`**.
|
> Full documentation (architecture, data model, every component): **`Documentation.md`**.
|
||||||
|
|
||||||
@@ -29,7 +28,6 @@ deploy/
|
|||||||
backup/ Daily Postgres backup script (cron on the homelab)
|
backup/ Daily Postgres backup script (cron on the homelab)
|
||||||
n8n/ Exported n8n workflows (lead-intake, onboarding, booking-sync, renewal-reminder)
|
n8n/ Exported n8n workflows (lead-intake, onboarding, booking-sync, renewal-reminder)
|
||||||
playbooks/ Operator playbooks (outreach, lead-to-customer, tier-a, tier-b)
|
playbooks/ Operator playbooks (outreach, lead-to-customer, tier-a, tier-b)
|
||||||
sheets/SCHEMA.md Google Sheets mirror spec (tabs + columns) — downstream of Postgres
|
|
||||||
templates/landing/ Landing-page demos + client preview pages
|
templates/landing/ Landing-page demos + client preview pages
|
||||||
demo-tier-a/ "Schnittpunkt" barbershop demo
|
demo-tier-a/ "Schnittpunkt" barbershop demo
|
||||||
demo-tier-b/ "PhysioVital" physio practice demo
|
demo-tier-b/ "PhysioVital" physio practice demo
|
||||||
@@ -65,8 +63,7 @@ Tracked against actual repo state — see `Documentation.md` for what backs each
|
|||||||
- [x] **Phase 1** — Demo landing pages
|
- [x] **Phase 1** — Demo landing pages
|
||||||
- [x] **Phase 2** — Accounts & setup: Google account + workbook, n8n live, Google creds bound,
|
- [x] **Phase 2** — Accounts & setup: Google account + workbook, n8n live, Google creds bound,
|
||||||
Vaultwarden (`clients.vault_ref`), domain (wildcard `*.mivanchenko.de`)
|
Vaultwarden (`clients.vault_ref`), domain (wildcard `*.mivanchenko.de`)
|
||||||
- [x] **Phase 3** — CRM (now Postgres-first, Sheets mirrored) + n8n lead-intake & renewal
|
- [x] **Phase 3** — CRM (Postgres-only, no Sheets mirror) + n8n lead-intake & renewal workflows
|
||||||
workflows — Looker Studio dashboard itself lives in Google and isn't repo-verifiable
|
|
||||||
- [x] **Phase 4** — Demo forms + booking wired live to n8n webhooks (lead-intake, booking-sync)
|
- [x] **Phase 4** — Demo forms + booking wired live to n8n webhooks (lead-intake, booking-sync)
|
||||||
- [ ] **Phase 5** — First real client + reception v1 — onboarding automation and per-client
|
- [ ] **Phase 5** — First real client + reception v1 — onboarding automation and per-client
|
||||||
deploy tooling exist and are exercised (`deploy/clients/`); not verifiable from the repo
|
deploy tooling exist and are exercised (`deploy/clients/`); not verifiable from the repo
|
||||||
|
|||||||
@@ -12,7 +12,6 @@
|
|||||||
**CRM / back office**
|
**CRM / back office**
|
||||||
- **#2** Decouple tier from business type → hosting/plan model; realign onboarding form + workflow. `enhancement` `crm`
|
- **#2** Decouple tier from business type → hosting/plan model; realign onboarding form + workflow. `enhancement` `crm`
|
||||||
- **#3** Extend back-office CRUD beyond Leads & Clients (projects, bookings, invoices, activity). `enhancement` `crm`
|
- **#3** Extend back-office CRUD beyond Leads & Clients (projects, bookings, invoices, activity). `enhancement` `crm`
|
||||||
- **#5** Make the DB→Sheets mirror atomic (no brief empty-sheet window). `tech-debt` `crm`
|
|
||||||
|
|
||||||
**Booking**
|
**Booking**
|
||||||
- **#7** Proper Orders flow + Orders table/tab (pizzeria demo posts into Leads as a stop-gap). `enhancement` `booking`
|
- **#7** Proper Orders flow + Orders table/tab (pizzeria demo posts into Leads as a stop-gap). `enhancement` `booking`
|
||||||
@@ -41,8 +40,12 @@
|
|||||||
exposed more broadly than Caddy basic-auth + host security.
|
exposed more broadly than Caddy basic-auth + host security.
|
||||||
|
|
||||||
## Done
|
## Done
|
||||||
- [x] 2026-07-15 — **Credentials in the CRM**: `credentials` entity (never mirrored to Sheets,
|
- [x] 2026-08-04 — **Cut Google Sheets entirely** (#13): Postgres is now the sole source of
|
||||||
`X-CRM-Token`-gated even to read), dashboard Credentials tab with masked value + reveal/copy;
|
truth — deleted the DB→Sheets mirror, the Sheets client, and the one-time importer;
|
||||||
|
`renewal-reminder.json` reads from the DB API instead of the Sheets mirror. Moots #5
|
||||||
|
(atomic-mirror tech-debt, closed as no longer applicable).
|
||||||
|
- [x] 2026-07-15 — **Credentials in the CRM**: `credentials` entity (`X-CRM-Token`-gated even to
|
||||||
|
read), dashboard Credentials tab with masked value + reveal/copy;
|
||||||
`n8n/onboarding.json` now persists the auto-generated EA provider password. Shipped live.
|
`n8n/onboarding.json` now persists the auto-generated EA provider password. Shipped live.
|
||||||
- [x] 2026-07-15 — **Gitea Actions deploy pipeline** (backoffice): self-hosted `homelab-runner`
|
- [x] 2026-07-15 — **Gitea Actions deploy pipeline** (backoffice): self-hosted `homelab-runner`
|
||||||
(`act-runner` systemd service), `workflow_dispatch` `deploy-backoffice.yml` that syncs
|
(`act-runner` systemd service), `workflow_dispatch` `deploy-backoffice.yml` that syncs
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ BOOKING_TOKEN_SECRET=change-me-long-random-too
|
|||||||
# Owner-login session cookie signing key (#19). Dedicated secret -- rotating
|
# Owner-login session cookie signing key (#19). Dedicated secret -- rotating
|
||||||
# it just logs owners out, without touching CRM_API_TOKEN/BOOKING_TOKEN_SECRET.
|
# it just logs owners out, without touching CRM_API_TOKEN/BOOKING_TOKEN_SECRET.
|
||||||
SESSION_SECRET_KEY=change-me-long-random-session-too
|
SESSION_SECRET_KEY=change-me-long-random-session-too
|
||||||
SHEET_ID=1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8
|
|
||||||
# Booking confirmation email (#18). Left blank, sending is skipped (logged,
|
# Booking confirmation email (#18). Left blank, sending is skipped (logged,
|
||||||
# not fatal) -- mail relay setup is a separate infra/triage item.
|
# not fatal) -- mail relay setup is a separate infra/triage item.
|
||||||
SMTP_HOST=
|
SMTP_HOST=
|
||||||
|
|||||||
+1
-92
@@ -1,16 +1,12 @@
|
|||||||
"""smb-crm back-office service.
|
"""smb-crm back-office service.
|
||||||
|
|
||||||
Postgres is the source of truth. This serves the operator dashboard + a small
|
Postgres is the source of truth for the operator dashboard + a small JSON API.
|
||||||
JSON API (read now; add/edit/delete and the DB->Sheets mirror layered on next).
|
|
||||||
Browser access is gated by Caddy basic-auth on onboard.mivanchenko.de; the
|
Browser access is gated by Caddy basic-auth on onboard.mivanchenko.de; the
|
||||||
machine-to-machine ingest path (n8n) is gated by the CRM_API_TOKEN header.
|
machine-to-machine ingest path (n8n) is gated by the CRM_API_TOKEN header.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import time
|
import time
|
||||||
import queue
|
|
||||||
import threading
|
|
||||||
import traceback
|
|
||||||
from datetime import datetime, date, timezone
|
from datetime import datetime, date, timezone
|
||||||
from decimal import Decimal
|
from decimal import Decimal
|
||||||
|
|
||||||
@@ -20,7 +16,6 @@ from waitress import serve
|
|||||||
import booking_db as bdb
|
import booking_db as bdb
|
||||||
import db
|
import db
|
||||||
import owner_mail
|
import owner_mail
|
||||||
from sheets import Sheets
|
|
||||||
from booking_api import bp as booking_bp
|
from booking_api import bp as booking_bp
|
||||||
from public_booking import bp as public_booking_bp
|
from public_booking import bp as public_booking_bp
|
||||||
from manage_booking import bp as manage_booking_bp
|
from manage_booking import bp as manage_booking_bp
|
||||||
@@ -47,69 +42,6 @@ CRM_TOKEN = os.environ.get("CRM_API_TOKEN", "")
|
|||||||
# the basic-auth header, so the feed is gated by this query token instead).
|
# the basic-auth header, so the feed is gated by this query token instead).
|
||||||
ICS_TOKEN = os.environ.get("ICS_TOKEN", "")
|
ICS_TOKEN = os.environ.get("ICS_TOKEN", "")
|
||||||
|
|
||||||
# DB -> Sheets one-way mirror. Postgres is the source of truth; the Sheet is a
|
|
||||||
# best-effort projection. A mirror failure never fails the DB write.
|
|
||||||
SH = None
|
|
||||||
try:
|
|
||||||
SH = Sheets(os.environ["GOOGLE_SA_JSON"], os.environ["SHEET_ID"])
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
traceback.print_exc()
|
|
||||||
|
|
||||||
|
|
||||||
def _cell(v):
|
|
||||||
if v is None:
|
|
||||||
return ""
|
|
||||||
if isinstance(v, bool):
|
|
||||||
return "TRUE" if v else "FALSE"
|
|
||||||
if isinstance(v, datetime):
|
|
||||||
return v.strftime("%Y-%m-%d %H:%M")
|
|
||||||
if isinstance(v, date):
|
|
||||||
return v.strftime("%Y-%m-%d")
|
|
||||||
if isinstance(v, Decimal):
|
|
||||||
f = float(v)
|
|
||||||
return str(int(f)) if f == int(f) else str(f)
|
|
||||||
s = str(v)
|
|
||||||
return ("'" + s) if s[:1] in "=+-@" else s # neutralise formula injection
|
|
||||||
|
|
||||||
|
|
||||||
def mirror_entity(entity):
|
|
||||||
spec = db.TABLES[entity]
|
|
||||||
if SH is None or spec.get("mirror") is False:
|
|
||||||
return 0
|
|
||||||
cols = spec["cols"]
|
|
||||||
order = LIST_ORDER.get(entity, spec["pk"])
|
|
||||||
with db.connect() as conn, conn.cursor() as cur:
|
|
||||||
cur.execute(f"SELECT {', '.join(cols)} FROM {entity} ORDER BY {order}")
|
|
||||||
rows = cur.fetchall()
|
|
||||||
grid = [[_cell(r[c]) for c in cols] for r in rows]
|
|
||||||
SH.overwrite(spec["tab"], cols, grid)
|
|
||||||
return len(grid)
|
|
||||||
|
|
||||||
|
|
||||||
# Serialize all mirror writes through one worker so concurrent mutations can't
|
|
||||||
# race on the shared Sheets client / token.
|
|
||||||
_mirror_q = queue.Queue()
|
|
||||||
|
|
||||||
|
|
||||||
def _mirror_worker():
|
|
||||||
while True:
|
|
||||||
entity = _mirror_q.get()
|
|
||||||
try:
|
|
||||||
mirror_entity(entity)
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
print(f"[mirror] {entity} sync failed:", flush=True)
|
|
||||||
traceback.print_exc()
|
|
||||||
finally:
|
|
||||||
_mirror_q.task_done()
|
|
||||||
|
|
||||||
|
|
||||||
threading.Thread(target=_mirror_worker, daemon=True).start()
|
|
||||||
|
|
||||||
|
|
||||||
def mirror_async(entity):
|
|
||||||
if SH is not None and db.TABLES[entity].get("mirror") is not False:
|
|
||||||
_mirror_q.put(entity)
|
|
||||||
|
|
||||||
# Static dashboard, with the CRM token injected so the (basic-auth-gated)
|
# Static dashboard, with the CRM token injected so the (basic-auth-gated)
|
||||||
# operator page can call the token-protected mutation endpoints.
|
# operator page can call the token-protected mutation endpoints.
|
||||||
with open(os.path.join(os.path.dirname(__file__), "static", "index.html")) as _f:
|
with open(os.path.join(os.path.dirname(__file__), "static", "index.html")) as _f:
|
||||||
@@ -235,8 +167,6 @@ def add_entity(entity):
|
|||||||
[row.get(c) for c in cols])
|
[row.get(c) for c in cols])
|
||||||
log_activity(cur, row.get("client_id"), f"add {entity}", f"{pk}={row.get(pk)}")
|
log_activity(cur, row.get("client_id"), f"add {entity}", f"{pk}={row.get(pk)}")
|
||||||
conn.commit()
|
conn.commit()
|
||||||
mirror_async(entity)
|
|
||||||
mirror_async("activity_log")
|
|
||||||
return jsonify({"added": row.get(pk)}), 201
|
return jsonify({"added": row.get(pk)}), 201
|
||||||
|
|
||||||
|
|
||||||
@@ -262,8 +192,6 @@ def edit_entity(entity, ident):
|
|||||||
log_activity(cur, ident if entity == "clients" else None,
|
log_activity(cur, ident if entity == "clients" else None,
|
||||||
f"edit {entity}", f"{pk}={ident}: {', '.join(setcols)}")
|
f"edit {entity}", f"{pk}={ident}: {', '.join(setcols)}")
|
||||||
conn.commit()
|
conn.commit()
|
||||||
mirror_async(entity)
|
|
||||||
mirror_async("activity_log")
|
|
||||||
return jsonify({"updated": ident, "fields": setcols})
|
return jsonify({"updated": ident, "fields": setcols})
|
||||||
|
|
||||||
|
|
||||||
@@ -281,8 +209,6 @@ def delete_entity(entity, ident):
|
|||||||
log_activity(cur, ident if entity == "clients" else None,
|
log_activity(cur, ident if entity == "clients" else None,
|
||||||
f"delete {entity}", f"{pk}={ident}")
|
f"delete {entity}", f"{pk}={ident}")
|
||||||
conn.commit()
|
conn.commit()
|
||||||
mirror_async(entity)
|
|
||||||
mirror_async("activity_log")
|
|
||||||
return jsonify({"deleted": ident})
|
return jsonify({"deleted": ident})
|
||||||
|
|
||||||
|
|
||||||
@@ -313,26 +239,9 @@ def trigger_owner_password_reset(user_id):
|
|||||||
log_activity(cur, user["client_id"], "owner password reset",
|
log_activity(cur, user["client_id"], "owner password reset",
|
||||||
f"user_id={user_id} (operator-triggered)")
|
f"user_id={user_id} (operator-triggered)")
|
||||||
conn.commit()
|
conn.commit()
|
||||||
mirror_async("activity_log")
|
|
||||||
return jsonify({"sent": user_id})
|
return jsonify({"sent": user_id})
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/sync")
|
|
||||||
def sync_all():
|
|
||||||
"""Full DB -> Sheets resync of every tab (manual / alignment)."""
|
|
||||||
if not authed():
|
|
||||||
return jsonify({"error": "forbidden"}), 403
|
|
||||||
if SH is None:
|
|
||||||
return jsonify({"error": "sheets unavailable"}), 503
|
|
||||||
out = {}
|
|
||||||
for entity in db.TABLES:
|
|
||||||
try:
|
|
||||||
out[entity] = mirror_entity(entity)
|
|
||||||
except Exception as e: # noqa: BLE001
|
|
||||||
out[entity] = f"error: {e}"
|
|
||||||
return jsonify({"synced": out})
|
|
||||||
|
|
||||||
|
|
||||||
def _ics_dt(v):
|
def _ics_dt(v):
|
||||||
if isinstance(v, datetime):
|
if isinstance(v, datetime):
|
||||||
u = v if v.tzinfo else v.replace(tzinfo=timezone.utc)
|
u = v if v.tzinfo else v.replace(tzinfo=timezone.utc)
|
||||||
|
|||||||
+5
-12
@@ -1,8 +1,8 @@
|
|||||||
"""Postgres access + the table/column contract shared across the app.
|
"""Postgres access + the table/column contract shared across the app.
|
||||||
|
|
||||||
The TABLES map is the single definition of which entities exist, their columns,
|
The TABLES map is the single definition of which entities exist, their columns,
|
||||||
and their primary key. Read/CRUD endpoints, the importer and the Sheets mirror
|
and their primary key. Read/CRUD endpoints all derive from it so they can
|
||||||
all derive from it so they can never drift apart.
|
never drift apart.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
@@ -13,10 +13,9 @@ from psycopg.rows import dict_row
|
|||||||
|
|
||||||
DATABASE_URL = os.environ["DATABASE_URL"]
|
DATABASE_URL = os.environ["DATABASE_URL"]
|
||||||
|
|
||||||
# entity -> (sheet tab, primary key, ordered columns)
|
# entity -> (primary key, ordered columns)
|
||||||
TABLES = {
|
TABLES = {
|
||||||
"clients": {
|
"clients": {
|
||||||
"tab": "Clients",
|
|
||||||
"pk": "client_id",
|
"pk": "client_id",
|
||||||
"cols": ["client_id", "business_name", "owner_name", "email", "phone",
|
"cols": ["client_id", "business_name", "owner_name", "email", "phone",
|
||||||
"niche", "tier", "status", "domain", "stack_notes", "vault_ref",
|
"niche", "tier", "status", "domain", "stack_notes", "vault_ref",
|
||||||
@@ -28,7 +27,6 @@ TABLES = {
|
|||||||
"bools": [],
|
"bools": [],
|
||||||
},
|
},
|
||||||
"leads": {
|
"leads": {
|
||||||
"tab": "Leads",
|
|
||||||
"pk": "lead_id",
|
"pk": "lead_id",
|
||||||
"cols": ["lead_id", "received_at", "client_id", "source", "name",
|
"cols": ["lead_id", "received_at", "client_id", "source", "name",
|
||||||
"contact", "service_interest", "message", "status", "notified"],
|
"contact", "service_interest", "message", "status", "notified"],
|
||||||
@@ -38,7 +36,6 @@ TABLES = {
|
|||||||
"bools": ["notified"],
|
"bools": ["notified"],
|
||||||
},
|
},
|
||||||
"projects": {
|
"projects": {
|
||||||
"tab": "Projects",
|
|
||||||
"pk": "project_id",
|
"pk": "project_id",
|
||||||
"cols": ["project_id", "client_id", "deliverable", "tier", "checklist",
|
"cols": ["project_id", "client_id", "deliverable", "tier", "checklist",
|
||||||
"go_live_date", "status"],
|
"go_live_date", "status"],
|
||||||
@@ -48,7 +45,6 @@ TABLES = {
|
|||||||
"bools": [],
|
"bools": [],
|
||||||
},
|
},
|
||||||
"bookings": {
|
"bookings": {
|
||||||
"tab": "Bookings",
|
|
||||||
"pk": "booking_id",
|
"pk": "booking_id",
|
||||||
"cols": ["booking_id", "created_at", "client_id", "customer_name",
|
"cols": ["booking_id", "created_at", "client_id", "customer_name",
|
||||||
"customer_contact", "service", "start_time", "end_time",
|
"customer_contact", "service", "start_time", "end_time",
|
||||||
@@ -59,7 +55,6 @@ TABLES = {
|
|||||||
"bools": [],
|
"bools": [],
|
||||||
},
|
},
|
||||||
"invoices": {
|
"invoices": {
|
||||||
"tab": "Invoices",
|
|
||||||
"pk": "invoice_id",
|
"pk": "invoice_id",
|
||||||
"cols": ["invoice_id", "client_id", "issued_date", "due_date",
|
"cols": ["invoice_id", "client_id", "issued_date", "due_date",
|
||||||
"amount_eur", "period", "status", "paid_date"],
|
"amount_eur", "period", "status", "paid_date"],
|
||||||
@@ -69,7 +64,6 @@ TABLES = {
|
|||||||
"bools": [],
|
"bools": [],
|
||||||
},
|
},
|
||||||
"activity_log": {
|
"activity_log": {
|
||||||
"tab": "Activity Log",
|
|
||||||
"pk": "id",
|
"pk": "id",
|
||||||
"cols": ["ts", "workflow", "client_id", "action", "detail", "result"],
|
"cols": ["ts", "workflow", "client_id", "action", "detail", "result"],
|
||||||
"dates": [],
|
"dates": [],
|
||||||
@@ -78,14 +72,13 @@ TABLES = {
|
|||||||
"bools": [],
|
"bools": [],
|
||||||
},
|
},
|
||||||
"credentials": {
|
"credentials": {
|
||||||
# No "tab": never mirrored to Sheets (see "mirror" below) — secrets stay in Postgres only.
|
# Secrets stay in Postgres only.
|
||||||
"pk": "cred_id",
|
"pk": "cred_id",
|
||||||
"cols": ["cred_id", "client_id", "label", "username", "secret", "notes", "created_at"],
|
"cols": ["cred_id", "client_id", "label", "username", "secret", "notes", "created_at"],
|
||||||
"dates": [],
|
"dates": [],
|
||||||
"timestamps": ["created_at"],
|
"timestamps": ["created_at"],
|
||||||
"numbers": [],
|
"numbers": [],
|
||||||
"bools": [],
|
"bools": [],
|
||||||
"mirror": False,
|
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -94,7 +87,7 @@ def connect():
|
|||||||
return psycopg.connect(DATABASE_URL, row_factory=dict_row)
|
return psycopg.connect(DATABASE_URL, row_factory=dict_row)
|
||||||
|
|
||||||
|
|
||||||
# ---- coercion: Sheet strings / JSON values -> typed Python for Postgres ----
|
# ---- coercion: JSON request values -> typed Python for Postgres ----
|
||||||
|
|
||||||
def parse_date(v):
|
def parse_date(v):
|
||||||
if v in (None, ""):
|
if v in (None, ""):
|
||||||
|
|||||||
@@ -1,57 +0,0 @@
|
|||||||
"""One-time (idempotent) seed of Postgres from the existing Google Sheets.
|
|
||||||
|
|
||||||
This is the ONLY place the Sheet is treated as authoritative — to bootstrap the
|
|
||||||
DB. After this, Postgres is the source of truth and Sheets is a downstream
|
|
||||||
mirror. Safe to re-run: keyed tables upsert by primary key; activity_log is
|
|
||||||
replaced wholesale.
|
|
||||||
|
|
||||||
Run inside the container: docker compose exec smb-crm python import_from_sheets.py
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
import db
|
|
||||||
from sheets import Sheets
|
|
||||||
|
|
||||||
|
|
||||||
def upsert(cur, entity, records):
|
|
||||||
spec = db.TABLES[entity]
|
|
||||||
cols = spec["cols"]
|
|
||||||
pk = spec["pk"]
|
|
||||||
n = 0
|
|
||||||
for rec in records:
|
|
||||||
row = db.coerce_row(entity, rec)
|
|
||||||
if entity != "activity_log" and not row.get(pk):
|
|
||||||
continue # skip rows without a primary key
|
|
||||||
placeholders = ", ".join(["%s"] * len(cols))
|
|
||||||
collist = ", ".join(cols)
|
|
||||||
if entity == "activity_log":
|
|
||||||
cur.execute(f"INSERT INTO {entity} ({collist}) VALUES ({placeholders})",
|
|
||||||
[row[c] for c in cols])
|
|
||||||
else:
|
|
||||||
updates = ", ".join(f"{c} = EXCLUDED.{c}" for c in cols if c != pk)
|
|
||||||
cur.execute(
|
|
||||||
f"INSERT INTO {entity} ({collist}) VALUES ({placeholders}) "
|
|
||||||
f"ON CONFLICT ({pk}) DO UPDATE SET {updates}",
|
|
||||||
[row[c] for c in cols])
|
|
||||||
n += 1
|
|
||||||
return n
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
sh = Sheets(os.environ["GOOGLE_SA_JSON"], os.environ["SHEET_ID"])
|
|
||||||
with db.connect() as conn:
|
|
||||||
with conn.cursor() as cur:
|
|
||||||
cur.execute("TRUNCATE activity_log RESTART IDENTITY")
|
|
||||||
for entity, spec in db.TABLES.items():
|
|
||||||
if spec.get("mirror") is False:
|
|
||||||
continue # e.g. credentials — never lived in the Sheet, nothing to import
|
|
||||||
records = sh.read_records(spec["tab"])
|
|
||||||
n = upsert(cur, entity, records)
|
|
||||||
print(f" {entity:<13} <- {spec['tab']:<13} {n} row(s)")
|
|
||||||
conn.commit()
|
|
||||||
print("import complete.")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
"""Best-effort SMTP email sending (#18).
|
"""Best-effort SMTP email sending (#18).
|
||||||
|
|
||||||
Mirrors app.py's DB -> Sheets mirror pattern: a single background worker
|
A single background worker thread drains a queue, and a send failure is
|
||||||
thread drains a queue, and a send failure is logged, never raised back to the
|
logged, never raised back to the caller -- booking creation must succeed even
|
||||||
caller -- booking creation must succeed even if the mail relay is down.
|
if the mail relay is down.
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
import queue
|
import queue
|
||||||
|
|||||||
@@ -8,7 +8,6 @@ Flask==3.0.3
|
|||||||
psycopg[binary]==3.2.1
|
psycopg[binary]==3.2.1
|
||||||
waitress==3.0.0
|
waitress==3.0.0
|
||||||
PyJWT[crypto]==2.9.0
|
PyJWT[crypto]==2.9.0
|
||||||
requests==2.32.3
|
|
||||||
# python:3.12-slim has no system IANA tz database; zoneinfo (used by
|
# python:3.12-slim has no system IANA tz database; zoneinfo (used by
|
||||||
# availability.py for Europe/Berlin) falls back to this package for it.
|
# availability.py for Europe/Berlin) falls back to this package for it.
|
||||||
tzdata==2024.1
|
tzdata==2024.1
|
||||||
|
|||||||
@@ -1,98 +0,0 @@
|
|||||||
"""Minimal Google Sheets v4 client using a service-account JWT.
|
|
||||||
|
|
||||||
Used for (a) the one-time import of existing Sheet rows into Postgres and
|
|
||||||
(b) the one-way DB -> Sheets mirror sync. Postgres stays the source of truth;
|
|
||||||
nothing here treats the Sheet as authoritative except the explicit import.
|
|
||||||
"""
|
|
||||||
import json
|
|
||||||
import time
|
|
||||||
import threading
|
|
||||||
|
|
||||||
import jwt
|
|
||||||
import requests
|
|
||||||
|
|
||||||
SHEETS_API = "https://sheets.googleapis.com/v4/spreadsheets"
|
|
||||||
TOKEN_URL = "https://oauth2.googleapis.com/token"
|
|
||||||
SCOPE = "https://www.googleapis.com/auth/spreadsheets"
|
|
||||||
|
|
||||||
# gid map for the live workbook (tab name -> gid), used when clearing/sizing.
|
|
||||||
TAB_GIDS = {
|
|
||||||
"Clients": 470934735,
|
|
||||||
"Leads": 571719114,
|
|
||||||
"Bookings": 946084008,
|
|
||||||
"Projects": 619112786,
|
|
||||||
"Invoices": 413377229,
|
|
||||||
"Activity Log": 170940481,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class Sheets:
|
|
||||||
def __init__(self, sa_json_path, sheet_id):
|
|
||||||
with open(sa_json_path) as f:
|
|
||||||
self.sa = json.load(f)
|
|
||||||
self.sheet_id = sheet_id
|
|
||||||
self._tok = None
|
|
||||||
self._exp = 0
|
|
||||||
self._lock = threading.Lock()
|
|
||||||
|
|
||||||
def _token(self):
|
|
||||||
with self._lock:
|
|
||||||
now = int(time.time())
|
|
||||||
if self._tok and now < self._exp - 60:
|
|
||||||
return self._tok
|
|
||||||
claim = {
|
|
||||||
"iss": self.sa["client_email"],
|
|
||||||
"scope": SCOPE,
|
|
||||||
"aud": TOKEN_URL,
|
|
||||||
"iat": now,
|
|
||||||
"exp": now + 3600,
|
|
||||||
}
|
|
||||||
assertion = jwt.encode(claim, self.sa["private_key"], algorithm="RS256")
|
|
||||||
r = requests.post(TOKEN_URL, data={
|
|
||||||
"grant_type": "urn:ietf:params:oauth:grant-type:jwt-bearer",
|
|
||||||
"assertion": assertion,
|
|
||||||
}, timeout=30)
|
|
||||||
r.raise_for_status()
|
|
||||||
self._tok = r.json()["access_token"]
|
|
||||||
self._exp = now + 3600
|
|
||||||
return self._tok
|
|
||||||
|
|
||||||
def _headers(self):
|
|
||||||
return {"Authorization": "Bearer " + self._token()}
|
|
||||||
|
|
||||||
def read(self, a1_range):
|
|
||||||
"""Return raw 2D list of cell values for an A1 range (e.g. "Leads!A1:Z")."""
|
|
||||||
url = f"{SHEETS_API}/{self.sheet_id}/values/{requests.utils.quote(a1_range)}"
|
|
||||||
r = requests.get(url, headers=self._headers(), timeout=30)
|
|
||||||
r.raise_for_status()
|
|
||||||
return r.json().get("values", [])
|
|
||||||
|
|
||||||
def read_records(self, tab):
|
|
||||||
"""Read a whole tab as a list of dicts keyed by the header row."""
|
|
||||||
rows = self.read(f"{tab}!A1:Z")
|
|
||||||
if not rows:
|
|
||||||
return []
|
|
||||||
header = rows[0]
|
|
||||||
out = []
|
|
||||||
for raw in rows[1:]:
|
|
||||||
if not any(c.strip() for c in raw):
|
|
||||||
continue
|
|
||||||
rec = {header[i]: (raw[i] if i < len(raw) else "") for i in range(len(header))}
|
|
||||||
out.append(rec)
|
|
||||||
return out
|
|
||||||
|
|
||||||
def overwrite(self, tab, header, rows):
|
|
||||||
"""Replace a tab's contents with header + rows (the DB->Sheets mirror).
|
|
||||||
|
|
||||||
Clears the existing value range, then writes the new grid starting at A1.
|
|
||||||
Postgres is the source of truth; this projects it onto the Sheet.
|
|
||||||
"""
|
|
||||||
# clear current values (keeps formatting / the tab itself)
|
|
||||||
clr = f"{SHEETS_API}/{self.sheet_id}/values/{requests.utils.quote(tab + '!A1:Z')}:clear"
|
|
||||||
requests.post(clr, headers=self._headers(), timeout=30).raise_for_status()
|
|
||||||
body = {"values": [header] + rows}
|
|
||||||
url = (f"{SHEETS_API}/{self.sheet_id}/values/"
|
|
||||||
f"{requests.utils.quote(tab + '!A1')}?valueInputOption=RAW")
|
|
||||||
r = requests.put(url, headers=self._headers(), json=body, timeout=60)
|
|
||||||
r.raise_for_status()
|
|
||||||
return len(rows)
|
|
||||||
@@ -74,7 +74,7 @@
|
|||||||
<h1>🗂️ smb-crm · Back Office</h1>
|
<h1>🗂️ smb-crm · Back Office</h1>
|
||||||
<div class="hgroup">
|
<div class="hgroup">
|
||||||
<a class="btn ghost" href="/">📝 Onboarding-Formular</a>
|
<a class="btn ghost" href="/">📝 Onboarding-Formular</a>
|
||||||
<span class="src">Quelle: Postgres (Sheets = Spiegel) · <span id="now"></span></span>
|
<span class="src">Quelle: Postgres · <span id="now"></span></span>
|
||||||
</div>
|
</div>
|
||||||
</header>
|
</header>
|
||||||
<div class="wrap">
|
<div class="wrap">
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
"""Unit tests for mailer.py's SMTP call shape (#18). Calls _send_now directly
|
"""Unit tests for mailer.py's SMTP call shape (#18). Calls _send_now directly
|
||||||
rather than going through the background-thread queue, so assertions are
|
rather than going through the background-thread queue, so assertions are
|
||||||
synchronous -- the queue itself is just plumbing, already covered indirectly
|
synchronous -- the queue itself is just plumbing.
|
||||||
by app.py's identical Sheets-mirror pattern.
|
|
||||||
"""
|
"""
|
||||||
from email.message import EmailMessage
|
from email.message import EmailMessage
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,4 @@
|
|||||||
-- smb-crm — source-of-truth schema (Postgres).
|
-- smb-crm — sole source-of-truth schema (Postgres).
|
||||||
-- Google Sheets is a one-way downstream mirror fed by the DB->Sheets sync.
|
|
||||||
-- Column sets mirror the Sheet tabs so the mirror stays 1:1.
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS clients (
|
CREATE TABLE IF NOT EXISTS clients (
|
||||||
client_id text PRIMARY KEY,
|
client_id text PRIMARY KEY,
|
||||||
@@ -87,8 +85,8 @@ CREATE TABLE IF NOT EXISTS invoices (
|
|||||||
);
|
);
|
||||||
|
|
||||||
-- Per-client login credentials (e.g. the auto-generated Easy!Appointments
|
-- Per-client login credentials (e.g. the auto-generated Easy!Appointments
|
||||||
-- provider login). Deliberately NOT mirrored to Sheets — see db.py TABLES
|
-- provider login). Gated by X-CRM-Token even to read, so secrets never leave
|
||||||
-- ("mirror": False) — so secrets never leave Postgres.
|
-- Postgres.
|
||||||
CREATE TABLE IF NOT EXISTS credentials (
|
CREATE TABLE IF NOT EXISTS credentials (
|
||||||
cred_id text PRIMARY KEY,
|
cred_id text PRIMARY KEY,
|
||||||
client_id text,
|
client_id text,
|
||||||
|
|||||||
@@ -27,16 +27,12 @@ services:
|
|||||||
BOOKING_TOKEN_SECRET: ${BOOKING_TOKEN_SECRET}
|
BOOKING_TOKEN_SECRET: ${BOOKING_TOKEN_SECRET}
|
||||||
SESSION_SECRET_KEY: ${SESSION_SECRET_KEY}
|
SESSION_SECRET_KEY: ${SESSION_SECRET_KEY}
|
||||||
ICS_TOKEN: ${ICS_TOKEN}
|
ICS_TOKEN: ${ICS_TOKEN}
|
||||||
SHEET_ID: ${SHEET_ID}
|
|
||||||
GOOGLE_SA_JSON: /run/secrets/gcp-sa.json
|
|
||||||
SMTP_HOST: ${SMTP_HOST}
|
SMTP_HOST: ${SMTP_HOST}
|
||||||
SMTP_PORT: ${SMTP_PORT}
|
SMTP_PORT: ${SMTP_PORT}
|
||||||
SMTP_USERNAME: ${SMTP_USERNAME}
|
SMTP_USERNAME: ${SMTP_USERNAME}
|
||||||
SMTP_PASSWORD: ${SMTP_PASSWORD}
|
SMTP_PASSWORD: ${SMTP_PASSWORD}
|
||||||
MAIL_FALLBACK_FROM: ${MAIL_FALLBACK_FROM}
|
MAIL_FALLBACK_FROM: ${MAIL_FALLBACK_FROM}
|
||||||
PUBLIC_BASE_URL: ${PUBLIC_BASE_URL}
|
PUBLIC_BASE_URL: ${PUBLIC_BASE_URL}
|
||||||
volumes:
|
|
||||||
- ./secrets/gcp-sa.json:/run/secrets/gcp-sa.json:ro
|
|
||||||
depends_on:
|
depends_on:
|
||||||
smb-db:
|
smb-db:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|||||||
+25
-56
@@ -1,5 +1,5 @@
|
|||||||
{
|
{
|
||||||
"name": "SMB \u00b7 Renewal Reminder",
|
"name": "SMB · Renewal Reminder",
|
||||||
"nodes": [
|
"nodes": [
|
||||||
{
|
{
|
||||||
"parameters": {
|
"parameters": {
|
||||||
@@ -23,43 +23,24 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"authentication": "serviceAccount",
|
"url": "http://smb-crm:8080/api/clients",
|
||||||
"resource": "sheet",
|
|
||||||
"operation": "read",
|
|
||||||
"documentId": {
|
|
||||||
"__rl": true,
|
|
||||||
"value": "1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8",
|
|
||||||
"mode": "id"
|
|
||||||
},
|
|
||||||
"sheetName": {
|
|
||||||
"__rl": true,
|
|
||||||
"value": "470934735",
|
|
||||||
"mode": "list",
|
|
||||||
"cachedResultName": "Clients"
|
|
||||||
},
|
|
||||||
"options": {}
|
"options": {}
|
||||||
},
|
},
|
||||||
"id": "f06a60af-dff1-415d-aca6-fef0367f3553",
|
"id": "f06a60af-dff1-415d-aca6-fef0367f3553",
|
||||||
"name": "Read Clients",
|
"name": "Read Clients (DB)",
|
||||||
"type": "n8n-nodes-base.googleSheets",
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
"typeVersion": 4.5,
|
"typeVersion": 4.2,
|
||||||
"position": [
|
"position": [
|
||||||
460,
|
460,
|
||||||
300
|
300
|
||||||
],
|
],
|
||||||
"credentials": {
|
|
||||||
"googleApi": {
|
|
||||||
"id": "45TWPyl1wVk2Vxdm",
|
|
||||||
"name": "SMB Google (service account)"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"retryOnFail": true,
|
"retryOnFail": true,
|
||||||
"maxTries": 4,
|
"maxTries": 4,
|
||||||
"waitBetweenTries": 3000
|
"waitBetweenTries": 3000
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"jsCode": "\nconst rows = $input.all().map(i=>i.json);\nconst today=new Date();\nconst t0=Date.UTC(today.getUTCFullYear(),today.getUTCMonth(),today.getUTCDate());\nfunction parseDate(v){\n if(v==null||v==='') return null;\n const s=String(v).trim();\n if(/^\\d+(\\.\\d+)?$/.test(s)){ return Date.UTC(1899,11,30)+Number(s)*86400000; }\n const m=s.match(/^(\\d{4})-(\\d{2})-(\\d{2})/);\n if(m) return Date.UTC(+m[1],+m[2]-1,+m[3]);\n const d=Date.parse(s); return isNaN(d)?null:d;\n}\nconst offsets=[14,7,3,1,0];\nconst due=[];\nfor(const r of rows){\n const status=String(r.status||'').toLowerCase();\n if(status!=='active') continue;\n const rd=parseDate(r.renewal_date);\n if(rd==null) continue;\n const days=Math.round((rd-t0)/86400000);\n if(days<0 || offsets.includes(days)){\n due.push({client_id:r.client_id, business:r.business_name, renewal_date:r.renewal_date,\n days, fee:r.monthly_fee_eur, cycle:r.billing_cycle});\n }\n}\nif(due.length===0) return [];\ndue.sort((a,b)=>a.days-b.days);\nconst line=d=>`\u2022 ${d.business||d.client_id} \u2014 ${d.days<0?('\u00dcBERF\u00c4LLIG seit '+(-d.days)+'T'):(d.days===0?'heute':('in '+d.days+'T'))} (${d.renewal_date}${d.fee?(', '+d.fee+'\u20ac/'+(d.cycle||'')):''})`;\nconst summary='\ud83d\udd14 Anstehende Verl\u00e4ngerungen ('+due.length+')\\n'+due.map(line).join('\\n');\nreturn [{json:{count:due.length, summary, due}}];\n"
|
"jsCode": "\nconst rows = $input.first().json.rows || [];\nconst today=new Date();\nconst t0=Date.UTC(today.getUTCFullYear(),today.getUTCMonth(),today.getUTCDate());\nfunction parseDate(v){\n if(v==null||v==='') return null;\n const s=String(v).trim();\n const m=s.match(/^(\\d{4})-(\\d{2})-(\\d{2})/);\n if(m) return Date.UTC(+m[1],+m[2]-1,+m[3]);\n const d=Date.parse(s); return isNaN(d)?null:d;\n}\nconst offsets=[14,7,3,1,0];\nconst due=[];\nfor(const r of rows){\n const status=String(r.status||'').toLowerCase();\n if(status!=='active') continue;\n const rd=parseDate(r.renewal_date);\n if(rd==null) continue;\n const days=Math.round((rd-t0)/86400000);\n if(days<0 || offsets.includes(days)){\n due.push({client_id:r.client_id, business:r.business_name, renewal_date:r.renewal_date,\n days, fee:r.monthly_fee_eur, cycle:r.billing_cycle});\n }\n}\nif(due.length===0) return [];\ndue.sort((a,b)=>a.days-b.days);\nconst line=d=>`• ${d.business||d.client_id} — ${d.days<0?('ÜBERFÄLLIG seit '+(-d.days)+'T'):(d.days===0?'heute':('in '+d.days+'T'))} (${d.renewal_date}${d.fee?(', '+d.fee+'€/'+(d.cycle||'')):''})`;\nconst summary='🔔 Anstehende Verlängerungen ('+due.length+')\\n'+due.map(line).join('\\n');\nreturn [{json:{count:due.length, summary, due}}];\n"
|
||||||
},
|
},
|
||||||
"id": "9f3e4ca9-0d87-4d1c-9f5e-abdf5f9f77dd",
|
"id": "9f3e4ca9-0d87-4d1c-9f5e-abdf5f9f77dd",
|
||||||
"name": "Find due renewals",
|
"name": "Find due renewals",
|
||||||
@@ -110,42 +91,30 @@
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"parameters": {
|
"parameters": {
|
||||||
"authentication": "serviceAccount",
|
"method": "POST",
|
||||||
"resource": "sheet",
|
"url": "http://smb-crm:8080/api/activity_log",
|
||||||
"operation": "append",
|
"sendHeaders": true,
|
||||||
"documentId": {
|
"headerParameters": {
|
||||||
"__rl": true,
|
"parameters": [
|
||||||
"value": "1raMSWRZw_JfHlWqOb3LbhaQ6LWx0VGblxIV4Z2pSzp8",
|
{
|
||||||
"mode": "id"
|
"name": "X-CRM-Token",
|
||||||
},
|
"value": "__CRM_TOKEN__"
|
||||||
"sheetName": {
|
}
|
||||||
"__rl": true,
|
]
|
||||||
"value": "170940481",
|
|
||||||
"mode": "list",
|
|
||||||
"cachedResultName": "Activity Log"
|
|
||||||
},
|
|
||||||
"columns": {
|
|
||||||
"mappingMode": "autoMapInputData",
|
|
||||||
"value": {},
|
|
||||||
"matchingColumns": [],
|
|
||||||
"schema": []
|
|
||||||
},
|
},
|
||||||
|
"sendBody": true,
|
||||||
|
"specifyBody": "json",
|
||||||
|
"jsonBody": "={{ JSON.stringify($json) }}",
|
||||||
"options": {}
|
"options": {}
|
||||||
},
|
},
|
||||||
"id": "55978962-489b-4740-a242-7470f08087b6",
|
"id": "55978962-489b-4740-a242-7470f08087b6",
|
||||||
"name": "Append to Activity Log",
|
"name": "Append to Activity Log (DB)",
|
||||||
"type": "n8n-nodes-base.googleSheets",
|
"type": "n8n-nodes-base.httpRequest",
|
||||||
"typeVersion": 4.5,
|
"typeVersion": 4.2,
|
||||||
"position": [
|
"position": [
|
||||||
1340,
|
1340,
|
||||||
300
|
300
|
||||||
],
|
],
|
||||||
"credentials": {
|
|
||||||
"googleApi": {
|
|
||||||
"id": "45TWPyl1wVk2Vxdm",
|
|
||||||
"name": "SMB Google (service account)"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"retryOnFail": true,
|
"retryOnFail": true,
|
||||||
"maxTries": 4,
|
"maxTries": 4,
|
||||||
"waitBetweenTries": 3000
|
"waitBetweenTries": 3000
|
||||||
@@ -156,14 +125,14 @@
|
|||||||
"main": [
|
"main": [
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
"node": "Read Clients",
|
"node": "Read Clients (DB)",
|
||||||
"type": "main",
|
"type": "main",
|
||||||
"index": 0
|
"index": 0
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"Read Clients": {
|
"Read Clients (DB)": {
|
||||||
"main": [
|
"main": [
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
@@ -200,7 +169,7 @@
|
|||||||
"main": [
|
"main": [
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
"node": "Append to Activity Log",
|
"node": "Append to Activity Log (DB)",
|
||||||
"type": "main",
|
"type": "main",
|
||||||
"index": 0
|
"index": 0
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ ongoing management. Target build time: ~half a day per client.
|
|||||||
6. **Reception** — Gmail for the business; set a signature + canned replies.
|
6. **Reception** — Gmail for the business; set a signature + canned replies.
|
||||||
7. **Google Business Profile** — claim/verify so they show up on Maps & Search.
|
7. **Google Business Profile** — claim/verify so they show up on Maps & Search.
|
||||||
8. **CRM** — add a `Clients` row (tier = A, services, monthly_fee, renewal_date, vault_ref).
|
8. **CRM** — add a `Clients` row (tier = A, services, monthly_fee, renewal_date, vault_ref).
|
||||||
9. **Handover** — share logins via Vaultwarden; record only the `vault_ref` in the sheet.
|
9. **Handover** — share logins via Vaultwarden; record only the `vault_ref` in the CRM.
|
||||||
|
|
||||||
## What to charge (suggestion)
|
## What to charge (suggestion)
|
||||||
- One-off setup: €250–600 depending on pages/content.
|
- One-off setup: €250–600 depending on pages/content.
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ German-hosted infrastructure. You charge a privacy/hosting premium. Target build
|
|||||||
4. **Lead form** — self-hosted form endpoint → n8n webhook → CRM. No third-party form service.
|
4. **Lead form** — self-hosted form endpoint → n8n webhook → CRM. No third-party form service.
|
||||||
5. **Reception** — business mailbox; (later) the **AI receptionist** widget (Phase 6, Claude
|
5. **Reception** — business mailbox; (later) the **AI receptionist** widget (Phase 6, Claude
|
||||||
Haiku 4.5) wired to the booking + FAQ.
|
Haiku 4.5) wired to the booking + FAQ.
|
||||||
6. **Credentials** — store all client logins in **Vaultwarden**; sheet holds only `vault_ref`.
|
6. **Credentials** — store all client logins in **Vaultwarden**; the CRM holds only `vault_ref`.
|
||||||
7. **CRM** — add a `Clients` row (tier = B, services, monthly_fee, renewal_date, vault_ref).
|
7. **CRM** — add a `Clients` row (tier = B, services, monthly_fee, renewal_date, vault_ref).
|
||||||
8. **Backups** — ensure the Tier-B services + CRM are in the daily backup job.
|
8. **Backups** — ensure the Tier-B services + CRM are in the daily backup job.
|
||||||
|
|
||||||
|
|||||||
@@ -1,63 +0,0 @@
|
|||||||
# Master CRM — Google Sheets workbook spec
|
|
||||||
|
|
||||||
One Google Sheets workbook = the whole CRM + log. I create/maintain this via the Google Drive MCP
|
|
||||||
once the fresh project Google account exists (Phase 2). n8n reads/writes it via the Sheets node.
|
|
||||||
|
|
||||||
Workbook name: **`SMB-Online — CRM`**. One tab per concern below. Row 1 = headers (exact column
|
|
||||||
names matter — n8n maps by header).
|
|
||||||
|
|
||||||
## Tab: `Clients`
|
|
||||||
The client list/table — the heart of the tracking requirement.
|
|
||||||
|
|
||||||
| Column | Type | Notes |
|
|
||||||
|---|---|---|
|
|
||||||
| `client_id` | text | e.g. `C-0001` (stable key) |
|
|
||||||
| `business_name` | text | |
|
|
||||||
| `owner_name` | text | |
|
|
||||||
| `email` | text | |
|
|
||||||
| `phone` | text | |
|
|
||||||
| `niche` | text | barber, physio, café, … |
|
|
||||||
| `tier` | A \| B | delivery tier |
|
|
||||||
| `status` | enum | `lead` → `onboarding` → `active` → `paused` → `churned` |
|
|
||||||
| `domain` | text | live site URL |
|
|
||||||
| `stack_notes` | text | hosting, booking tool, etc. |
|
|
||||||
| `vault_ref` | text | **pointer** to Vaultwarden item — never a password |
|
|
||||||
| `services` | text | comma list: site, booking, reception, … |
|
|
||||||
| `billing_cycle` | enum | monthly / yearly / one-off |
|
|
||||||
| `monthly_fee_eur` | number | |
|
|
||||||
| `start_date` | date | |
|
|
||||||
| `renewal_date` | date | drives the renewal-reminder workflow |
|
|
||||||
| `created_at` | datetime | |
|
|
||||||
| `notes` | text | |
|
|
||||||
|
|
||||||
## Tab: `Leads`
|
|
||||||
Every inbound lead across all client sites (and your own).
|
|
||||||
|
|
||||||
`lead_id` · `received_at` · `client_id` · `source` (which site/form) · `name` · `contact` ·
|
|
||||||
`service_interest` · `message` · `status` (new / contacted / won / lost) · `notified` (bool)
|
|
||||||
|
|
||||||
## Tab: `Bookings`
|
|
||||||
Every appointment booked across client sites.
|
|
||||||
|
|
||||||
`booking_id` · `created_at` · `client_id` · `customer_name` · `customer_contact` · `service` ·
|
|
||||||
`start_time` · `end_time` · `source` (gcal / calcom) · `status` (confirmed / cancelled / no-show)
|
|
||||||
|
|
||||||
## Tab: `Projects`
|
|
||||||
Per-client deliverable tracking.
|
|
||||||
|
|
||||||
`project_id` · `client_id` · `deliverable` · `tier` · `checklist` (e.g. `domain✓ page✓ booking☐`) ·
|
|
||||||
`go_live_date` · `status` (todo / in-progress / live)
|
|
||||||
|
|
||||||
## Tab: `Invoices`
|
|
||||||
`invoice_id` · `client_id` · `issued_date` · `due_date` · `amount_eur` · `period` ·
|
|
||||||
`status` (draft / sent / paid / overdue) · `paid_date`
|
|
||||||
|
|
||||||
## Tab: `Activity Log`
|
|
||||||
Append-only audit trail — **every automated action writes one row here.**
|
|
||||||
|
|
||||||
`ts` · `workflow` (lead-intake / booking-sync / renewal / onboarding / invoice) · `client_id` ·
|
|
||||||
`action` · `detail` · `result` (ok / error)
|
|
||||||
|
|
||||||
## Dashboard
|
|
||||||
Not a tab — a **Looker Studio** report wired to this workbook: pipeline by `status`, MRR
|
|
||||||
(sum of `monthly_fee_eur` where `status=active`), renewals due in next 30 days, recent leads.
|
|
||||||
Reference in New Issue
Block a user