smb-online/Documentation.md

# smb-online — Documentation

Operator-facing documentation for the whole system: what exists, how the pieces talk to each
other, and where to look when something needs changing. For pitch/positioning copy see
`README.md`; for open work see `TODO.md`; for day-to-day operator procedure see `playbooks/`.

## 1. What this is

A productized service that gets local small businesses (SMBs) online: **landing page + online
booking + lead capture + light automation**, run by one operator on a homelab. Every lead,
client, booking and invoice is tracked centrally so nothing falls through the cracks.

Originally scoped as two fixed delivery tiers (**Tier A** = free Google stack, **Tier B** =
self-hosted/privacy). That coupling is being unwound (see `TODO.md`): tier is now an internal
effort/price signal chosen per client, not a property baked into the public demos. In practice,
booking has already converged on one shared self-hosted service (Easy!Appointments, §4.3) for
every client regardless of tier.

## 2. Architecture

```
                         ┌────────────────────────────────────────────┐
                         │                Postgres (smb-db)            │
                         │  clients · leads · projects · bookings ·    │
                         │  invoices · activity_log · credentials —    │
                         │  SOURCE OF TRUTH (credentials never mirror) │
                         └───────────────▲───────────────┬────────────┘
                                         │ SQL             │ one-way mirror
                          JSON API       │                 ▼
                    ┌────────────────────┴───────┐   ┌───────────────────┐
                    │   smb-crm (Flask, backoffice)│   │  Google Sheets    │
                    │   onboard.mivanchenko.de/crm │   │  (human overview,  │
                    │   dashboard + CRUD + iCal    │   │  Looker Studio)    │
                    └────────────────────▲─────────┘   └───────────────────┘
                                         │ HTTP (CRM_API_TOKEN)
                                         │
                    ┌────────────────────┴─────────────────────────────┐
                    │                     n8n                          │
                    │  lead-intake · onboarding · booking-sync ·       │
                    │  renewal-reminder     (n8n.mivanchenko.de)       │
                    └───────────▲───────────────────────▲──────────────┘
                                │ webhook                │ webhook
                 ┌──────────────┴───────────┐  ┌─────────┴────────────────┐
                 │ Landing pages / demos     │  │ Easy!Appointments        │
                 │ (lead form, callback      │  │ booking.mivanchenko.de   │
                 │  widget) — demos.*, and   │  │ (shared, one instance,   │
                 │  each client's own site   │  │  one service+provider    │
                 │  (client-<slug> container)│  │  per client)             │
                 └───────────────────────────┘  └───────────────────────────┘
```

**Postgres is the source of truth.** Google Sheets used to be authoritative; the back-office
rewrite (2026-06-25) flipped that — the DB now owns writes, and Sheets is a best-effort,
one-way projection kept for the human-readable overview and the Looker Studio dashboard. See
`backoffice/app/db.py` (schema/contract), `backoffice/app/sheets.py` (mirror client).

Everything routes through **Caddy** on the homelab (TLS + basic-auth + reverse proxy) and a
shared external `proxy` Docker network. Compose groups (§4) are independent projects that only
share that network — see `deploy/clients/README.md` for the full picture.

## 3. Data model

Six entities, defined once in `backoffice/app/db.py::TABLES` and mirrored 1:1 into
`backoffice/db/init.sql` (Postgres) and Google Sheets tabs (`sheets/SCHEMA.md`):

| Entity | PK | Purpose |
|---|---|---|
| `clients` | `client_id` (`C-####`) | The client roster — tier, status (`lead→onboarding→active→paused→churned`), billing, renewal date, `vault_ref` (pointer into Vaultwarden, never a password), `stack_notes` (free text — hosting/booking details, e.g. the EA embed URL). |
| `leads` | `lead_id` (`L-<epoch-ms>`) | Every inbound lead/callback request across all sites. `status`: `new→contacted→qualified→won/lost`. |
| `projects` | `project_id` (`P-<epoch-ms>`) | Per-client deliverable/checklist tracking, auto-created on onboarding. |
| `bookings` | `booking_id` | Every appointment, synced from Easy!Appointments via `booking-sync`. |
| `invoices` | `invoice_id` | Billing records (manual today — no automated invoicing workflow yet). |
| `activity_log` | `id` (serial) | Append-only audit trail; every mutation (API or workflow) writes one row. |
| `credentials` | `cred_id` (`CR-<epoch-ms>`) | Per-client login credentials (e.g. the auto-generated Easy!Appointments provider login) — `client_id`, `label`, `username`, `secret`, `notes`. **The one entity excluded from the Sheets mirror** (`mirror: False` in `TABLES`) so secrets never leave Postgres; reading it also requires `X-CRM-Token` (every other entity's list is read-only-open behind Caddy basic-auth alone). Surfaced in the dashboard's **Credentials** tab with a masked value, a reveal toggle, and a copy-to-clipboard button. |

`db.coerce_row()` is the single place that types/normalizes incoming values (dates, timestamps,
numbers, booleans) so the CRUD API, the n8n ingest path and the one-time Sheets importer can never
drift on types.

## 4. Components

### 4.1 `backoffice/` — CRM API + operator dashboard
Flask app (`app.py`) + `waitress`, backed by Postgres (`smb-db`, `postgres:16-alpine`).

- `GET /api/<entity>` — list (no auth; page itself sits behind Caddy basic-auth).
- `POST /api/<entity>` — create. Requires header `X-CRM-Token: $CRM_API_TOKEN`. Auto-assigns
  `client_id`/`lead_id`/`project_id`, `created_at`/`received_at`, default `status`, and computes
  `renewal_date` from `start_date` + `billing_cycle` (monthly/yearly) when omitted.
- `PATCH /api/<entity>/<id>` — partial update, same auth.
- `DELETE /api/<entity>/<id>` — delete, same auth.
- `POST /api/sync` — force a full DB→Sheets resync of every tab (same auth).
- `GET /api/bookings.ics` — read-only iCal feed for calendar apps (Apple/Google Calendar can't
  send custom headers, so this is gated by a separate `?token=$ICS_TOKEN` query param instead of
  the header token). Supports `?client_id=` to scope to one client.
- `GET /healthz` — DB connectivity check.
- `GET /` — the dashboard (`static/index.html`; **Leads**, **Clients** and **Credentials** tabs —
  read + add (+Neu) + edit (✎) + delete (🗑), all token-gated, all audit-logged. The Credentials
  tab masks the `secret` column by default with a per-row 👁 reveal toggle and a 📋 copy-to-
  clipboard button).

Every write is audit-logged to `activity_log` and enqueues an async, best-effort mirror of that
entity (and of `activity_log` itself) into the linked Google Sheet — mirror failures never fail
the DB write (`mirror_async` / `_mirror_worker` in `app.py`).

`import_from_sheets.py` is a one-time, idempotent bootstrap (upsert by PK) used only to seed
Postgres from the pre-existing Sheet; after that the Sheet is purely downstream.

Run locally: `docker compose -f backoffice/docker-compose.yml up` (needs `.env` from
`backoffice/.env.example` + a service-account JSON mounted at `./secrets/gcp-sa.json`).

### 4.2 `templates/landing/` — landing pages & demos
Static, self-contained HTML (no build step). `templates/landing/index.html` is the public
chooser (also carries the "Rückruf" callback-request popup, wired to the lead-intake webhook).

- `demo-tier-a/`, `demo-tier-b/`, `demo-pizzeria/` — the three neutralized (stack-agnostic)
  outreach demos: barbershop, physio practice, pizzeria w/ online ordering.
- `preview-*/` — one-off rebrands used as personalised outreach hooks (see
  `playbooks/outreach.md`) or as seeds for a real client site (see `new-client.sh`).
- Lead/callback forms POST directly (client-side `fetch`) to
  `https://n8n.mivanchenko.de/webhook/lead-intake`; booking widgets embed/link to the shared
  Easy!Appointments instance.
- Publicly hosted at `demos.mivanchenko.de` (see §4.4).

### 4.3 `deploy/booking/` — shared booking service
One shared **Easy!Appointments** instance (PHP + MariaDB) for *all* clients — a client is
modeled as an EA "service" + "provider" pair, not a separate stack. Routed at
`booking.mivanchenko.de`. Ships with two host overrides applied read-only into the container:
`frontend.css` (recolors the widget to match brand, compacts layout for iframe embedding) and
`booking_layout.js` (reports rendered height to the embedding page for auto-fit). Both must be
re-synced from upstream whenever the EA image is upgraded, since production
(`DEBUG_MODE=FALSE`) serves the `.min.*` bundles that these files override.

A booking event fires a webhook → n8n `booking-sync` → `POST /api/bookings` on the CRM → Telegram
notification to the operator.

### 4.4 `deploy/` — hosting topology
Everything below shares the external `proxy` Docker network (Caddy reaches each container by its
Compose service name):

| Compose group | What | Domain | Cardinality |
|---|---|---|---|
| `backoffice/docker-compose.yml` | `smb-db` + `smb-crm` | `onboard.mivanchenko.de/crm` | one, shared |
| *(external, not in this repo)* | n8n + Postgres + Redis | `n8n.mivanchenko.de` | one, shared |
| `deploy/booking/` | Easy!Appointments + MariaDB | `booking.mivanchenko.de` | one, shared |
| `deploy/smb-demos/` | Apache serving `templates/landing/` | `demos.mivanchenko.de` | prospect previews |
| `deploy/clients/<slug>/` | one `nginx:alpine` static site | client's own domain (wildcard `*.mivanchenko.de` or client-owned) | **one per signed client** |

Only the client-facing sites multiply — everything else stays a single shared instance.
`deploy/clients/new-client.sh <slug> <domain> [source-site-dir]` scaffolds a new isolated client
folder from `deploy/clients/_template/` (fills `.env`, seeds `site/`) and prints the Caddy block
+ deploy commands. `docker compose down` in a client's folder removes exactly that client and
nothing else (clean offboarding).

`deploy/backup/smb-db-backup.sh` — daily cron job on the homelab: `pg_dump`s `smb-db`, gzips,
keeps the 14 most recent dumps in `/home/mivanchenko/backups/smb-crm/`.

### 4.5 `n8n/` — automation workflows (exported JSON)
| Workflow | Trigger | Does |
|---|---|---|
| `lead-intake.json` | webhook | Normalize a lead payload → `POST /api/leads` → Telegram notify. Used by every demo/client lead form and the callback widget. |
| `onboarding.json` | webhook (`onboard.mivanchenko.de` form) | Compute client+project rows → `POST /api/clients` → `POST /api/projects` → Telegram notify → **provision Easy!Appointments** (create service, create provider with a generated login, build the booking embed URL) → `PATCH` the client's `stack_notes` with that embed URL + EA username → `POST /api/credentials` with the EA username **and password**. Fully automates "sign a client" end to end, including capturing the generated password so it isn't lost (it used to be discarded after the EA API call — see `TODO.md`). |
| `booking-sync.json` | webhook (EA) | Normalize a booking event → `POST /api/bookings` → Telegram notify. |
| `renewal-reminder.json` | daily 08:00 schedule | Read `Clients` from Sheets → find renewals due soon → Telegram notify → append a row to `Activity Log`. **Note:** still reads from the Sheets mirror rather than the DB directly — safe today because the mirror is kept current, but a re-point to the DB would remove that indirection. |

n8n itself (the workflow engine + its own Postgres/Redis) is **not** part of this repo — it's a
separate, already-running compose stack on the homelab; only the exported workflow definitions
live here.

### 4.6 `sheets/SCHEMA.md` — Sheets mirror spec
Describes the Google Sheets workbook (`SMB-Online — CRM`) that Postgres mirrors into: one tab per
entity, headers matching column names (n8n / the mirror map by header). Also specifies the
**Looker Studio** dashboard built on top of it (pipeline by status, MRR, renewals due in 30 days,
recent leads) — the dashboard itself lives in Google, not in this repo.

### 4.7 `playbooks/` — operator procedure
- `outreach.md` — how to find prospects, build a personalised preview in ~20 min, and reach out
  (email/DM/phone scripts in German).
- `lead-to-customer.md` — the full lifecycle once a lead exists: qualify → close → run the
  onboarding form → build/ship the real site → tune booking → hand over → go live → track renewal.
- `tier-a-google.md` / `tier-b-selfhosted.md` — per-tier setup checklists and suggested pricing.

## 5. Environments, secrets, and how to run things

Secrets are never committed (`.env`, `*.secret`, `.secrets/`, `credentials/`,
`backoffice/secrets/`, `stacks/**/.env` are gitignored — see `.gitignore`). Each deployable has
its own `.env.example` to copy from:

| File | Fills |
|---|---|
| `backoffice/.env.example` | `DB_PASSWORD`, `CRM_API_TOKEN`, `SHEET_ID` (+ a service-account JSON mounted at `secrets/gcp-sa.json`, not example-tracked) |
| `deploy/booking/.env.example` | `EA_DB_PASSWORD`, `EA_DB_ROOT_PASSWORD` |
| `deploy/clients/_template/.env.example` | `CLIENT_SLUG`, `CLIENT_DOMAIN` (per-client, generated by `new-client.sh`) |

Local dev (no build tooling needed anywhere in this repo):
```bash
# Demos — self-contained HTML, open directly or serve the folder:
python3 -m http.server 8080 --directory templates/landing

# Back office (needs Postgres + a Google service account):
docker compose -f backoffice/docker-compose.yml up
```
Production deploys are `docker compose up -d` per Compose group on the homelab, fronted by Caddy;
see `deploy/clients/README.md` and each group's own compose file for the exact routing.

## 6. Security notes
- Browser-facing surfaces (`onboard.mivanchenko.de`, dashboards) sit behind Caddy basic-auth.
- Machine-to-machine writes (n8n → CRM) are gated by the `CRM_API_TOKEN` header, checked in
  `backoffice/app/app.py::authed()`.
- The iCal feed is gated by a separate query-string token (`ICS_TOKEN`) since calendar clients
  can't send custom headers — treat that token as effectively public-linkable and rotate it if a
  feed URL leaks.
- Sheet cell values are defended against formula injection (`_cell()` in `app.py` prefixes
  values starting with `=+-@` with a `'`).
- Credentials for client-owned accounts are recorded two ways: `clients.vault_ref` points to a
  Vaultwarden item for anything the operator manually stashes there; the `credentials` table
  holds secrets the *system itself* generates (currently: the Easy!Appointments provider login
  created during onboarding), gated by `X-CRM-Token` even to read and deliberately excluded from
  the Sheets mirror. Stored as plaintext in Postgres today — same trust boundary as the rest of
  the CRM (Caddy basic-auth + host security); revisit with column-level encryption (pgcrypto) if
  the dashboard is ever exposed more broadly (see `TODO.md`).
- The DB→Sheets mirror is clear-then-write, not atomic — a reader can theoretically catch a
  cleared tab mid-sync. Accepted as low-risk (human overview, not a system of record) — see
  `TODO.md`.

## 7. Legal / positioning constraint
The operator is doing a Cloud Engineer Ausbildung at NETWAYS. This offering deliberately stays
out of NETWAYS's lane (no managed cloud hosting / monitoring / OpenStack-K8s hosting sold to
clients) — see the compliance note in `README.md`. This shapes real decisions in the code/infra:
e.g. Tier B is pitched as "I host a small website + booking for you," not managed
infrastructure (`playbooks/tier-b-selfhosted.md`).

## 8. Where to look next
- Open work, known gaps, and rationale for recent pivots: `TODO.md`.
- Positioning/pitch copy and quickstart: `README.md`.
- Exact table/column contract: `backoffice/app/db.py`.
- Exact API behavior: `backoffice/app/app.py`.