# irc.now Project Memory

## Project Overview
IRC bouncer-as-a-service platform. Rust workspace monorepo at ~/development/irc-now.

## Architecture
- **Workspace crates**: common, web-api, web-landing, bot-landing, bot, soju-operator, ergo-operator, txt, pics, platform-operator
- **Non-crate services**: chat/ (gamja + nginx, not Rust)
- **Rust**: Axum 0.8, kube.rs 3.0, sqlx, openid 0.23, askama 0.15
- **MSRV**: 1.88 (askama 0.15, kube 3.0, home 0.5.12 require it)
- **Containerfiles**: rust:1.88 -> ubi9-minimal multi-stage builds

## Cluster State (OCP 4.20)
See [cluster.md](cluster.md) for details.

### Namespaces
- `irc-josie-cloud`: web-landing, web-api, bot-landing, accounts-db, soju, soju-operator, ergo-operator, platform-operator, irc-now-net, txt, pics, minio, paste-db, pics-db, bot-db, chat
- `irc-keycloak`: keycloak-db (CNPG only, Keycloak deployment moved to `keycloak` namespace)
- `keycloak`: keycloak-operator, irc-now Keycloak CR, josie.cloud Keycloak CR

### Services Live
- `irc.now` -- landing page (web-landing)
- `irc.bot` -- bot service + landing page (bot, bot-landing)
- `my.irc.now` -- account portal (web-api)
- `txt.irc.now` -- pastebin (txt)
- `irc.pics` -- image host (pics)
- `chat.irc.now` -- web IRC client (gamja + nginx, WebSocket proxy to soju)
- `auth.irc.now` -- Keycloak 26.1 (realm: irc-now, operator-managed CR in `keycloak` ns)
- `stats.irc.now` -- Grafana dashboards (Keycloak SSO, admin/viewer roles)

### TLS Pattern
cert-manager Certificate CR -> Secret -> Route `tls.externalCertificate.name`
Requires Role/RoleBinding granting `router` SA in `openshift-ingress` access to the secret.
ClusterIssuer: `letsencrypt-prod` (dns-01 via Cloudflare)

## Build Pattern
OCP binary builds using `--from-archive` with minimal tar (~65K):
```bash
tar czf /tmp/build.tar.gz --exclude='target' --exclude='.git' \
  --exclude='irc-now-landing-page' --exclude='status' --exclude='design' \
  --exclude='./docs' --exclude='notes' Cargo.toml Cargo.lock crates/
oc start-build <name> --from-archive=/tmp/build.tar.gz -n irc-josie-cloud --follow
```
Use `--exclude='./docs'` (not `--exclude='docs'`) to avoid excluding `crates/*/static/docs/`.
BuildConfig needs `dockerfilePath` set (e.g. `crates/web-api/Containerfile`).

## Keycloak Theme
Custom login theme at `deploy/keycloak/theme/irc-now/login/`. Deployed via ConfigMap +
init container on the operator-managed Keycloak CR. Theme extends `keycloak` parent,
overrides template.ftl, login.ftl, login-reset-password.ftl, error.ftl, info.ftl.
To update: edit theme files, recreate ConfigMap, delete the Keycloak pod.
```bash
oc create configmap keycloak-theme-irc-now -n keycloak --from-file=... --dry-run=client -o yaml | oc apply -f -
oc delete pod irc-now-0 -n keycloak
```

## Key Secrets
- `irc-now-keycloak-admin` (keycloak): admin credentials (copied from irc-keycloak)
- `irc-now-keycloak-db` (keycloak): DB credentials (copied from irc-keycloak)
- `keycloak-admin` (irc-keycloak): admin credentials (original)
- `keycloak-db-app` (irc-keycloak): CNPG-managed DB creds
- `accounts-db-app` (irc-josie-cloud): CNPG-managed DB creds
- `oidc-account-portal` (irc-josie-cloud): OIDC client secret
- `oidc-paste` (irc-josie-cloud): OIDC client for paste service
- `oidc-pics` (irc-josie-cloud): OIDC client for pics service
- `stripe-keys` (irc-josie-cloud): secret-key, price-id, webhook-secret (placeholder values -- replace with real keys)
- `minio-credentials` (irc-josie-cloud): root-user, root-password
- `paste-db-app` (irc-josie-cloud): CNPG-managed DB creds
- `pics-db-app` (irc-josie-cloud): CNPG-managed DB creds
- `oidc-soju` (irc-josie-cloud): soju OAuth2 client (client-id=soju, client-secret)
- `oidc-ergo` (irc-josie-cloud): ergo OAuth2 client (client-id=ergo, client-secret)
- `oidc-bot` (irc-josie-cloud): bot OIDC client (client-id=bot, client-secret)
- `bot-db-app` (irc-josie-cloud): CNPG-managed DB creds
- `oidc-stats` (irc-josie-cloud): Grafana OIDC client (client-id=stats)
- `grafana-sa-token` (irc-josie-cloud): SA token for Prometheus access
- `irc-now-admin-credentials` (irc-josie-cloud): Grafana admin password (operator-managed)

## CNPG Notes
- StorageClass: `customer-workload-storage` (hostPath PVs)
- SELinux context: `container_file_t` required on hostPath dirs
- UID ranges: irc-josie-cloud=1001260000, irc-keycloak=1001300000

## Phase 1 Status
Complete. Plan: docs/plans/2026-03-03-irc-now-phase1-plan.md

## Phase 1.5 Status
Deployed. Auth guard, users table, dashboard/bouncer k8s wiring, Stripe checkout flow,
content pages (docs/clients/faq/what-is-irc), ZNC decommissioned.
Remaining: replace placeholder stripe-keys, configure Upptime (needs GH token perms),
remove znc.josie.lol DNS from Cloudflare.
- `stripe_util.rs` (renamed from `stripe.rs` to avoid shadowing the `stripe` crate)

## Phase 3 Status
Deployed and live. Two crates: txt (txt.irc.now) and pics (irc.pics).
Auth: OIDC via Keycloak, plan + content_expires extracted from JWT custom claims.
Paste: nanoid IDs, free=24h expiry, pro=90d (opt-out to unlimited), 5 paste limit for free.
Pics: MinIO (S3) storage, thumbnails (300px WebP), 50MB free / 1GB pro storage.
Pics expiry: free=90d, pro=90d (opt-out to unlimited), hourly cleanup (S3 + DB).
Keycloak migrated to operator-managed CR in `keycloak` namespace.
Custom irc-now login theme deployed via ConfigMap + init container.
- `rust-s3 = "0.35"` for S3 client
- `image = "0.25"` for thumbnail generation
- `base64 = "0.22"` for JWT payload decoding in txt/pics
- Axum route ordering: static routes (`/my`) must precede parameterized (`/{id}`)
- SameSite::Lax required on session cookies for cross-site OIDC redirects

## Content Expiry + Keycloak Sync
- `content_expires` column on users table (boolean, default true)
- Profile page: pro users see content settings toggle
- `update_keycloak_user_attributes()` in profile.rs syncs plan + content_expires to Keycloak user attributes
- Called from: profile save, auth callback (login), billing webhook (upgrade/downgrade)
- Keycloak user attributes -> protocol mappers -> ID token claims
- txt/pics: `extract_custom_claims()` base64-decodes JWT payload for plan + content_expires
- `bearer.id_token: Option<String>` on openid 0.23 `Token` contains raw JWT

## Chat Service (chat.irc.now)
gamja web IRC client served by nginx. Top-level `chat/` directory (not a Rust crate).
- Containerfile clones gamja from `git.sr.ht/~emersion/gamja` (codeberg mirror unreliable)
- nginx serves static files, proxies `/socket` to soju WebSocket (`josie:8080`)
- `proxy_set_header Host $host;` required so soju's WebSocket Origin check passes
- Theme injected via nginx `sub_filter` on `<body>` tag (AGPL compliance)
- Title renamed via `sub_filter 'gamja IRC client' 'chat.irc.now'`
- soju-operator supports `websocket: bool` on Listener type (`ws+insecure://`, `wss://`)
- OAuth2 SSO: gamja -> Keycloak (public client `chat`) -> SASL OAUTHBEARER -> soju
- soju `auth oauth2` with `enable-user-on-auth true`, credentials from `oidc-soju` secret
- Keycloak `chat` client: PKCE disabled (gamja doesn't support it), `exclude.session.state.from.auth.response` and `exclude.issuer.from.auth.response` set to avoid redirect_uri mismatch
- Build: `oc start-build chat --from-dir=chat/ -n irc-josie-cloud --follow`
- Route has `haproxy.router.openshift.io/timeout: 600s` for WebSocket keepalive

## Whois Hostname Cloaking (Deployed)
Design: docs/plans/2026-03-05-whois-hostname-cloaking-design.md
Plan: docs/plans/2026-03-05-whois-hostname-cloaking-plan.md
- CRD has `hostname_cloak: Option<String>`, operator emits `hostname-cloak` in soju.conf
- Custom soju image at `crates/soju-operator/soju/` (Containerfile + apply-hostname-cloak.sh)
- Containerfile clones from Codeberg master, applies hostname-cloak sed patch, builds with nosqlite
- SOJU_IMAGE = internal registry, imagePullPolicy = Always
- All bouncer CRs patched with `hostname_cloak: <name>.irc.now`
- web-api sets hostname_cloak on bouncer creation
- Cloaking applies to **downstream** connections (clients connecting to soju bouncer)
- Does NOT affect upstream network whois (hackint etc. show soju server IP)
- Next: rDNS or network-level cloaks needed for upstream whois

## Ergo Operator (net.irc.now)
- CRD: ErgoNetwork (`irc.josie.cloud/v1alpha1`), manages Ergo IRC server instances
- Image: `ghcr.io/ergochat/ergo:stable`
- Config mounted at `/config/ircd.yaml`, TLS at `/config/tls/`
- Ergo config: `listeners` must be nested under `server:` (not top-level)
- Previous serde_yaml serialization put `listeners` at top level, causing crash
- Fixed by manual YAML string generation in `configmap.rs`
- OAuth2 token introspection via Keycloak (`oidc-ergo` secret)
- IP cloaking configured per-network via `spec.cloaking.netname`
- Storage: 1Gi PVC (`customer-workload-storage`) for ircd.db
- External connect: `net.irc.now:443` TLS (OCP passthrough route -> Ergo 6697)
- Internal connect: `irc-now-net:6667` plaintext (soju default upstream)
- Auth: SASL PLAIN with Keycloak credentials, autocreate enabled

## Soju Version Notes
- Primary repo: Codeberg (codeberg.org/emersion/soju) -- GitHub/SourceHut are stale mirrors
- No v0.10.x tags; Go module proxy serves pseudo-versions for tagged releases only
- Codeberg master has schema v20 (20 PostgreSQL migrations)
- Custom image builds from Codeberg master via `git clone` in Containerfile
- `nosqlite` build tag required for CGO_ENABLED=0 builds (we use PostgreSQL)
- imagePullPolicy must be Always since we use :latest tag

## Phase 6 Status (Bot Service)
Deployed. Crate: `crates/bot/` (irc-now-bot). Domain: irc.bot.
Axum service with mlua Lua 5.4 sandbox per bot. OIDC auth, PostgreSQL storage.
- `mlua = "0.11"` with features lua54, async, send, vendored (statically links Lua)
- `tokio-rustls = "0.26"` + `rustls-native-certs = "0.8"` for TLS IRC connections
- `make_interval` needs `$1::int` cast (sqlx binds i64 as bigint)
- CNPG bot-db provisioned, Keycloak client `bot`, oidc-bot secret, BuildConfig all done

## Monitoring (Phase 2)
- OCP user workload monitoring enabled (`cluster-monitoring-config` in openshift-monitoring)
- ServiceMonitors for web-api, txt, pics, bot (30s scrape, port http, path /metrics)
- Grafana at stats.irc.now via grafana-operator, Keycloak SSO (client `stats`)
- Groups: `stats-admin` -> Admin, everyone else -> Viewer
- Admin folder: overview dashboard (all metrics). Public: basic stats only
- Datasource: Thanos Querier via `grafana-sa` token (cluster-monitoring-view)
- Manifests: `deploy/monitoring/`
- `metrics-exporter-prometheus` needs `default-features = false` to avoid aws-lc-rs/ring conflict
- OnceLock<PrometheusHandle> pattern (not Extension) for metrics handler

## Business Metrics (Phase 2)
Each Axum service has a `business_metrics` module with background gauge tasks:
- web-api: `irc_now_users_total`, `irc_now_users_by_plan{plan}`, `irc_now_active_users_24h` (60s),
  `irc_now_logins_total` (counter on auth callback),
  `irc_now_bouncer_networks_total`, `irc_now_bouncer_channels_total`,
  `irc_now_bouncer_messages_total`, `irc_now_bouncer_active_users` (300s, per-tenant soju DB)
- txt: `irc_now_pastes_total`, `irc_now_pastes_storage_bytes` (60s)
- pics: `irc_now_images_total`, `irc_now_images_storage_bytes` (60s)
- bot: `irc_now_bots_total`, `irc_now_bots_enabled`, `irc_now_bots_running` (60s)
- `last_login_at` column on users table (migration: `crates/web-api/migrations/001_add_last_login_at.sql`)
- Deploy prerequisite: apply migration to accounts-db before deploying new web-api

## Platform Operator (Deployed)
Manages WebService, ChatService, MinioInstance CRDs. Server-side apply for Deployment,
Service, Route, Certificate, PVC per CR. Reconciles on 300s interval.
- CRD group: `irc.now`, version: `v1alpha1`
- All CRD types use `#[serde(rename_all = "camelCase")]` for K8s convention
- Shortnames: `ws` (WebService), `chat` (ChatService), `minio` (MinioInstance)
- Crate: `crates/platform-operator/`
- CRD gen: `cargo run -p platform-operator --bin platform-crdgen`
- Deploy manifests: `crates/platform-operator/deploy/` (crd.yaml, rbac.yaml, operator.yaml, buildconfig.yaml)
- Instance CRs: `deploy/instances/` (web-landing, bot-landing, web-api, txt, pics, chat, minio)
- Service labels (`app: <name>`) added to Service metadata for ServiceMonitor matching
- paste -> txt rename: BC renamed, image now `txt:latest`, CR name is `txt`

## Obsidian Docs
Service notes at ~/Documents/Obsidian Vault/98 josiedot/services/:
- irc-now-landing.md, irc-now-portal.md, irc-now-keycloak.md, irc-bot-landing.md
- irc-now-bot.md, irc-now-chat.md, irc-now-txt.md, irc-now-pics.md, irc-now-stats.md
- irc-now-soju-operator.md, irc-now-ergo-operator.md, irc-now-platform-operator.md