diff --git a/Documentation.md b/Documentation.md new file mode 100644 index 0000000..c372b1e --- /dev/null +++ b/Documentation.md @@ -0,0 +1,227 @@ +# smb-online — Documentation + +Operator-facing documentation for the whole system: what exists, how the pieces talk to each +other, and where to look when something needs changing. For pitch/positioning copy see +`README.md`; for open work see `TODO.md`; for day-to-day operator procedure see `playbooks/`. + +## 1. What this is + +A productized service that gets local small businesses (SMBs) online: **landing page + online +booking + lead capture + light automation**, run by one operator on a homelab. Every lead, +client, booking and invoice is tracked centrally so nothing falls through the cracks. + +Originally scoped as two fixed delivery tiers (**Tier A** = free Google stack, **Tier B** = +self-hosted/privacy). That coupling is being unwound (see `TODO.md`): tier is now an internal +effort/price signal chosen per client, not a property baked into the public demos. In practice, +booking has already converged on one shared self-hosted service (Easy!Appointments, §4.3) for +every client regardless of tier. + +## 2. Architecture + +``` + ┌────────────────────────────────────────────┐ + │ Postgres (smb-db) │ + │ clients · leads · projects · bookings · │ + │ invoices · activity_log — SOURCE OF TRUTH │ + └───────────────▲───────────────┬────────────┘ + │ SQL │ one-way mirror + JSON API │ ▼ + ┌────────────────────┴───────┐ ┌───────────────────┐ + │ smb-crm (Flask, backoffice)│ │ Google Sheets │ + │ onboard.mivanchenko.de/crm │ │ (human overview, │ + │ dashboard + CRUD + iCal │ │ Looker Studio) │ + └────────────────────▲─────────┘ └───────────────────┘ + │ HTTP (CRM_API_TOKEN) + │ + ┌────────────────────┴─────────────────────────────┐ + │ n8n │ + │ lead-intake · onboarding · booking-sync · │ + │ renewal-reminder (n8n.mivanchenko.de) │ + └───────────▲───────────────────────▲──────────────┘ + │ webhook │ webhook + ┌──────────────┴───────────┐ ┌─────────┴────────────────┐ + │ Landing pages / demos │ │ Easy!Appointments │ + │ (lead form, callback │ │ booking.mivanchenko.de │ + │ widget) — demos.*, and │ │ (shared, one instance, │ + │ each client's own site │ │ one service+provider │ + │ (client- container)│ │ per client) │ + └───────────────────────────┘ └───────────────────────────┘ +``` + +**Postgres is the source of truth.** Google Sheets used to be authoritative; the back-office +rewrite (2026-06-25) flipped that — the DB now owns writes, and Sheets is a best-effort, +one-way projection kept for the human-readable overview and the Looker Studio dashboard. See +`backoffice/app/db.py` (schema/contract), `backoffice/app/sheets.py` (mirror client). + +Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a +shared external `proxy` Docker network. Compose groups (§4) are independent projects that only +share that network — see `deploy/clients/README.md` for the full picture. + +## 3. Data model + +Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into +`backoffice/db/init.sql` (Postgres) and Google Sheets tabs (`sheets/SCHEMA.md`): + +| Entity | PK | Purpose | +|---|---|---| +| `clients` | `client_id` (`C-####`) | The client roster — tier, status (`lead→onboarding→active→paused→churned`), billing, renewal date, `vault_ref` (pointer into Vaultwarden, never a password), `stack_notes` (free text — hosting/booking details, e.g. the EA embed URL). | +| `leads` | `lead_id` (`L-`) | Every inbound lead/callback request across all sites. `status`: `new→contacted→qualified→won/lost`. | +| `projects` | `project_id` (`P-`) | Per-client deliverable/checklist tracking, auto-created on onboarding. | +| `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. | +| `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). | +| `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. | + +`db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps, +numbers, booleans) so the CRUD API, the n8n ingest path and the one-time Sheets importer can never +drift on types. + +## 4. Components + +### 4.1 `backoffice/` — CRM API + operator dashboard +Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-alpine`). + +- `GET /api/` — list (no auth; page itself sits behind Caddy basic-auth). +- `POST /api/` — create. Requires header `X-CRM-Token: $CRM_API_TOKEN`. Auto-assigns + `client_id`/`lead_id`/`project_id`, `created_at`/`received_at`, default `status`, and computes + `renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted. +- `PATCH /api//` — partial update, same auth. +- `DELETE /api//` — delete, same auth. +- `POST /api/sync` — force a full DB→Sheets resync of every tab (same auth). +- `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't + send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of + the header token). Supports `?client_id=` to scope to one client. +- `GET /healthz` — DB connectivity check. +- `GET /` — the dashboard (`static/index.html`; currently **Leads** and **Clients** tabs only — + read + add (+Neu) + edit (✎) + delete (🗑), all token-gated, all audit-logged). + +Every write is audit-logged to `activity_log` and enqueues an async, best-effort mirror of that +entity (and of `activity_log` itself) into the linked Google Sheet — mirror failures never fail +the DB write (`mirror_async` / `_mirror_worker` in `app.py`). + +`import_from_sheets.py` is a one-time, idempotent bootstrap (upsert by PK) used only to seed +Postgres from the pre-existing Sheet; after that the Sheet is purely downstream. + +Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from +`backoffice/.env.example` + a service-account JSON mounted at `./secrets/gcp-sa.json`). + +### 4.2 `templates/landing/` — landing pages & demos +Static, self-contained HTML (no build step). `templates/landing/index.html` is the public +chooser (also carries the "Rückruf" callback-request popup, wired to the lead-intake webhook). + +- `demo-tier-a/`, `demo-tier-b/`, `demo-pizzeria/` — the three neutralized (stack-agnostic) + outreach demos: barbershop, physio practice, pizzeria w/ online ordering. +- `preview-*/` — one-off rebrands used as personalised outreach hooks (see + `playbooks/outreach.md`) or as seeds for a real client site (see `new-client.sh`). +- Lead/callback forms POST directly (client-side `fetch`) to + `https://n8n.mivanchenko.de/webhook/lead-intake`; booking widgets embed/link to the shared + Easy!Appointments instance. +- Publicly hosted at `demos.mivanchenko.de` (see §4.4). + +### 4.3 `deploy/booking/` — shared booking service +One shared **Easy!Appointments** instance (PHP + MariaDB) for *all* clients — a client is +modeled as an EA "service" + "provider" pair, not a separate stack. Routed at +`booking.mivanchenko.de`. Ships with two host overrides applied read-only into the container: +`frontend.css` (recolors the widget to match brand, compacts layout for iframe embedding) and +`booking_layout.js` (reports rendered height to the embedding page for auto-fit). Both must be +re-synced from upstream whenever the EA image is upgraded, since production +(`DEBUG_MODE=FALSE`) serves the `.min.*` bundles that these files override. + +A booking event fires a webhook → n8n `booking-sync` → `POST /api/bookings` on the CRM → Telegram +notification to the operator. + +### 4.4 `deploy/` — hosting topology +Everything below shares the external `proxy` Docker network (Caddy reaches each container by its +Compose service name): + +| Compose group | What | Domain | Cardinality | +|---|---|---|---| +| `backoffice/docker-compose.yml` | `smb-db` + `smb-crm` | `onboard.mivanchenko.de/crm` | one, shared | +| *(external, not in this repo)* | n8n + Postgres + Redis | `n8n.mivanchenko.de` | one, shared | +| `deploy/booking/` | Easy!Appointments + MariaDB | `booking.mivanchenko.de` | one, shared | +| `deploy/smb-demos/` | Apache serving `templates/landing/` | `demos.mivanchenko.de` | prospect previews | +| `deploy/clients//` | one `nginx:alpine` static site | client's own domain (wildcard `*.mivanchenko.de` or client-owned) | **one per signed client** | + +Only the client-facing sites multiply — everything else stays a single shared instance. +`deploy/clients/new-client.sh [source-site-dir]` scaffolds a new isolated client +folder from `deploy/clients/_template/` (fills `.env`, seeds `site/`) and prints the Caddy block ++ deploy commands. `docker compose down` in a client's folder removes exactly that client and +nothing else (clean offboarding). + +`deploy/backup/smb-db-backup.sh` — daily cron job on the homelab: `pg_dump`s `smb-db`, gzips, +keeps the 14 most recent dumps in `/home/mivanchenko/backups/smb-crm/`. + +### 4.5 `n8n/` — automation workflows (exported JSON) +| Workflow | Trigger | Does | +|---|---|---| +| `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. | +| `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients` → `POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA login. Fully automates "sign a client" end to end. | +| `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. | +| `renewal-reminder.json` | daily 08:00 schedule | Read `Clients` from Sheets → find renewals due soon → Telegram notify → append a row to `Activity Log`. **Note:** still reads from the Sheets mirror rather than the DB directly — safe today because the mirror is kept current, but a re-point to the DB would remove that indirection. | + +n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a +separate, already-running compose stack on the homelab; only the exported workflow definitions +live here. + +### 4.6 `sheets/SCHEMA.md` — Sheets mirror spec +Describes the Google Sheets workbook (`SMB-Online — CRM`) that Postgres mirrors into: one tab per +entity, headers matching column names (n8n / the mirror map by header). Also specifies the +**Looker Studio** dashboard built on top of it (pipeline by status, MRR, renewals due in 30 days, +recent leads) — the dashboard itself lives in Google, not in this repo. + +### 4.7 `playbooks/` — operator procedure +- `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out + (email/DM/phone scripts in German). +- `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the + onboarding form → build/ship the real site → tune booking → hand over → go live → track renewal. +- `tier-a-google.md` / `tier-b-selfhosted.md` — per-tier setup checklists and suggested pricing. + +## 5. Environments, secrets, and how to run things + +Secrets are never committed (`.env`, `*.secret`, `.secrets/`, `credentials/`, +`backoffice/secrets/`, `stacks/**/.env` are gitignored — see `.gitignore`). Each deployable has +its own `.env.example` to copy from: + +| File | Fills | +|---|---| +| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN`, `SHEET_ID` (+ a service-account JSON mounted at `secrets/gcp-sa.json`, not example-tracked) | +| `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` | +| `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) | + +Local dev (no build tooling needed anywhere in this repo): +```bash +# Demos — self-contained HTML, open directly or serve the folder: +python3 -m http.server 8080 --directory templates/landing + +# Back office (needs Postgres + a Google service account): +docker compose -f backoffice/docker-compose.yml up +``` +Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy; +see `deploy/clients/README.md` and each group's own compose file for the exact routing. + +## 6. Security notes +- Browser-facing surfaces (`onboard.mivanchenko.de`, dashboards) sit behind Caddy basic-auth. +- Machine-to-machine writes (n8n → CRM) are gated by the `CRM_API_TOKEN` header, checked in + `backoffice/app/app.py::authed()`. +- The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients + can't send custom headers — treat that token as effectively public-linkable and rotate it if a + feed URL leaks. +- Sheet cell values are defended against formula injection (`_cell()` in `app.py` prefixes + values starting with `=+-@` with a `'`). +- Credentials for client-owned accounts are never stored directly — `clients.vault_ref` stores a + pointer into Vaultwarden only. +- The DB→Sheets mirror is clear-then-write, not atomic — a reader can theoretically catch a + cleared tab mid-sync. Accepted as low-risk (human overview, not a system of record) — see + `TODO.md`. + +## 7. Legal / positioning constraint +The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays +out of NETWAYS's lane (no managed cloud hosting / monitoring / OpenStack-K8s hosting sold to +clients) — see the compliance note in `README.md`. This shapes real decisions in the code/infra: +e.g. Tier B is pitched as "I host a small website + booking for you," not managed +infrastructure (`playbooks/tier-b-selfhosted.md`). + +## 8. Where to look next +- Open work, known gaps, and rationale for recent pivots: `TODO.md`. +- Positioning/pitch copy and quickstart: `README.md`. +- Exact table/column contract: `backoffice/app/db.py`. +- Exact API behavior: `backoffice/app/app.py`. diff --git a/README.md b/README.md index 4522e5d..8f8592c 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,40 @@ # smb-online — "Get Your Small Business Online" -A productized service that gets local small businesses online: **landing page + online booking + -online reception + small automations**. Built to be **thoroughly tracked and logged** — every -client, lead, booking and invoice lives in a Google Sheets CRM, driven by **n8n** on the homelab. +A productized service that gets local small businesses online: **landing page + online +booking + lead capture + small automations**. Built to be **thoroughly tracked and logged** — +every client, lead, booking and invoice lives in a **Postgres CRM** (source of truth), driven by +**n8n** on the homelab, with a one-way mirror into Google Sheets for a human-readable overview / +Looker Studio dashboard. -> Full plan: `~/.claude/plans/pitch-me-some-ideas-breezy-swing.md` +> Full documentation (architecture, data model, every component): **`Documentation.md`**. -## Two delivery tiers +## Delivery tiers — an internal signal, not a fixed product -| | **Tier A — Google stack** | **Tier B — Self-hosted / privacy** | -|---|---|---| -| For | casual businesses, no data-residency concern | privacy-sensitive (health, legal, tax) | -| Site | static page / Google Sites | static page (Astro) on free host | -| Booking | Google Calendar Appointment Schedule | self-hosted Cal.com | -| Lead capture | Google Forms | self-hosted form → n8n | -| Reception | Gmail | self-hosted + AI receptionist | -| Infra cost | ≈ €0 | small (homelab + domain) | - -Both tiers feed the **same Google Sheets CRM** via n8n webhooks. +`tier` (A = free Google stack, B = self-hosted/privacy) started as two fixed product bundles but +is being decoupled from the public offering (see `TODO.md`): which stack a client actually gets +is a decision made **with the client**, case-by-case, and recorded on their `clients` row +(`tier`, `stack_notes`) — it's no longer baked into the demos. In practice, **booking has already +converged on one shared self-hosted service** (Easy!Appointments, see `deploy/booking/`) used for +every client regardless of tier. ## Repository layout ``` -templates/landing/ Reusable landing-page template + the two tier demos - demo-tier-a/ Tier A demo — "Schnittpunkt" barbershop (Google stack) - demo-tier-b/ Tier B demo — "PhysioVital" physio practice (self-hosted/privacy) -sheets/SCHEMA.md Google Sheets CRM workbook spec (tabs + columns) -n8n/ Exported n8n workflow JSON (lead-intake, booking-sync, renewals, …) -stacks/selfhosted/ docker-compose for Tier-B services (Cal.com, Vaultwarden, lead form) -playbooks/ Per-client setup checklists (tier-a-google.md, tier-b-selfhosted.md) +Documentation.md Full architecture / component reference — start here for "how it works" +backoffice/ Postgres CRM + Flask API + operator dashboard (the source of truth) +deploy/ + clients/ One isolated nginx stack per signed client + new-client.sh scaffolder + booking/ Shared Easy!Appointments booking service (all clients) + smb-demos/ Public demo/preview hosting (Apache) + backup/ Daily Postgres backup script (cron on the homelab) +n8n/ Exported n8n workflows (lead-intake, onboarding, booking-sync, renewal-reminder) +playbooks/ Operator playbooks (outreach, lead-to-customer, tier-a, tier-b) +sheets/SCHEMA.md Google Sheets mirror spec (tabs + columns) — downstream of Postgres +templates/landing/ Landing-page demos + client preview pages + demo-tier-a/ "Schnittpunkt" barbershop demo + demo-tier-b/ "PhysioVital" physio practice demo + demo-pizzeria/ "Bella Napoli" pizzeria demo w/ online ordering +templates/onboarding/ Internal "create client" form → n8n onboarding workflow ``` ## View the demos @@ -36,28 +42,36 @@ playbooks/ Per-client setup checklists (tier-a-google.md, tier-b- They are self-contained HTML — open directly: ```bash -xdg-open templates/landing/demo-tier-a/index.html # barbershop / Google stack -xdg-open templates/landing/demo-tier-b/index.html # physio practice / self-hosted +xdg-open templates/landing/demo-tier-a/index.html # barbershop +xdg-open templates/landing/demo-tier-b/index.html # physio practice +xdg-open templates/landing/demo-pizzeria/index.html # pizzeria / online ordering ``` -Or serve both with a chooser page: +Or serve them all with the chooser page: ```bash python3 -m http.server 8080 --directory templates/landing -# → http://localhost:8080/ (links to both demos) +# → http://localhost:8080/ (links to all three demos) ``` -Forms and booking are **stubbed** until Phase 4 wires them to the n8n webhooks. +Lead/callback forms POST live to the n8n `lead-intake` webhook; booking widgets link to the +shared Easy!Appointments instance. Nothing here is stubbed anymore — see `Documentation.md` §4.5 +for the full workflow wiring. ## Phases (status) -- [x] **Phase 1** — Two demo landing pages (this commit) -- [ ] **Phase 2** — Accounts & setup: fresh Google account → connect Drive MCP → master workbook; - secure n8n; bind Google creds; Vaultwarden; domain -- [ ] **Phase 3** — CRM workbook + n8n lead-intake & renewal workflows + Looker Studio dashboard -- [ ] **Phase 4** — Wire demo forms/booking to n8n webhooks (live end-to-end proof) -- [ ] **Phase 5** — First real client + reception v1 -- [ ] **Phase 6** — AI receptionist (Claude Haiku 4.5) +Tracked against actual repo state — see `Documentation.md` for what backs each checkmark. + +- [x] **Phase 1** — Demo landing pages +- [x] **Phase 2** — Accounts & setup: Google account + workbook, n8n live, Google creds bound, + Vaultwarden (`clients.vault_ref`), domain (wildcard `*.mivanchenko.de`) +- [x] **Phase 3** — CRM (now Postgres-first, Sheets mirrored) + n8n lead-intake & renewal + workflows — Looker Studio dashboard itself lives in Google and isn't repo-verifiable +- [x] **Phase 4** — Demo forms + booking wired live to n8n webhooks (lead-intake, booking-sync) +- [ ] **Phase 5** — First real client + reception v1 — onboarding automation and per-client + deploy tooling exist and are exercised (`deploy/clients/`); not verifiable from the repo + alone whether a paying client is live +- [ ] **Phase 6** — AI receptionist (Claude Haiku 4.5) — not started ## ⚠️ Before going live (legal — not legal advice) diff --git a/TODO.md b/TODO.md index 9b96041..025d9a1 100644 --- a/TODO.md +++ b/TODO.md @@ -35,7 +35,9 @@ is a one-way mirror — see `backoffice/` and the `smb-db-source-of-truth` memor Follow-ups (not yet done): - [ ] Extend CRUD UI beyond Leads/Clients (projects, bookings, invoices, activity) if useful. -- [ ] Daily backup of the `smb-db` Postgres volume. + Dashboard (`backoffice/app/static/index.html`) still only has Leads/Clients tabs. +- [x] Daily backup of the `smb-db` Postgres volume — DONE: `deploy/backup/smb-db-backup.sh` + (`pg_dump` + gzip, keeps 14 days, cron on the homelab). - [ ] Make the DB→Sheets overwrite atomic (currently clear-then-write; brief mid-write window a reader could catch the cleared sheet — harmless for a human overview). @@ -46,7 +48,11 @@ Follow-ups (not yet done): - [ ] Orders: the pizzeria demo posts orders into the `Leads` tab via the lead-intake webhook as a stop-gap. Build a proper **Orders** flow + sheet tab (items, total, mode, status) when the order-management product is real. -- [ ] Booking-sync workflow (Tier-B Cal.com webhook + Tier-A Google Calendar poll). +- [x] Booking-sync workflow — DONE, but not as originally scoped: the plan to split + Tier-B-Cal.com-webhook / Tier-A-GCal-poll was superseded by the tier-decoupling above. Instead, + one shared **Easy!Appointments** instance handles booking for every client (`deploy/booking/`), + wired via `n8n/booking-sync.json` (webhook → `POST /api/bookings` → Telegram), and + `n8n/onboarding.json` auto-provisions each new client's EA service+provider. ## Done - [x] 2026-06-25 — Latency fix (n8n executions 1–4 min → ~6 s; DNS + PostHog telemetry). @@ -56,3 +62,8 @@ Follow-ups (not yet done): - [x] 2026-06-25 — All three demo pages neutralised (stack-agnostic, non-stale dates). - [x] 2026-06-25 — DB-first back office: Postgres source of truth + Sheets mirror + CRUD + n8n ingest swap (lead-intake/onboarding → DB API). Live at onboard.mivanchenko.de/crm. +- [x] 2026-07-14 — Shared self-hosted booking: Easy!Appointments stack (`deploy/booking/`), + auto-provisioning wired into the onboarding workflow, `booking-sync` workflow syncing bookings + to the CRM. +- [x] 2026-07-14 — Per-client deploy tooling: `deploy/clients/` isolated-stack model + + `new-client.sh` scaffolder; back-office updates.