# Session Summary — Catalog · VIP Hub · Hybrid 4.7 · OpenClaw + ZeroClaw (Both Servers) — COMPREHENSIVE

| Field | Value |
|--------|--------|
| **Canonical comprehensive file (this doc)** | `/Users/khaledahmedmohamed/designs-content/vip/setting/SESSION_SUMMARY_2026-07-14_to_15_Catalog_Hub_Hybrid47_OpenClaw_ZeroClaw.md` |
| **Merged source A (ai-developer / OpenClaw)** | `/Users/khaledahmedmohamed/openclaw-session-summary.md` |
| **Merged source B (contabo2 / ZeroClaw)** | `/Users/khaledahmedmohamed/designs-content/vip/setting/openclaw-session-summary.md` |
| **Session window** | 2026-07-14 → 2026-07-15 |
| **Document type** | Permanent local `.md` (not chat artifact) — **build-on merge** of three reports |
| **Servers** | **ai-developer** `213.199.36.17` · **contabo2** `149.102.150.185` · **hetzner** `46.62.228.173` (api-new only) |

> **Merge policy:** Content from both `openclaw-session-summary.md` files is **absorbed into this document**. Sources A/B remain on disk as historical notes; **this file is the single comprehensive index.**

---

## 0. Dual-server OpenClaw/ZeroClaw map (quick)

| Public URL | Server | SSH | Systemd unit | Port | Config / state |
|------------|--------|-----|--------------|------|----------------|
| https://openclaw.addict.best/ | **ai-developer** `213.199.36.17` | `ssh -i ~/.ssh/ai_developer_key root@213.199.36.17` | `openclaw-gateway.service` | `127.0.0.1:18789` | `~/.openclaw` (default) + `/opt/openclaw` |
| https://zeroclaw.addict.best/chat?session=main | **contabo2** `149.102.150.185` | `ssh -i ~/.ssh/contabo2_new1 root@149.102.150.185` | `zeroclaw-gateway.service` | `127.0.0.1:18789` | `OPENCLAW_STATE_DIR=/root/zeroclaw` · `OPENCLAW_CONFIG_PATH=/root/zeroclaw/openclaw.json` |

Same product family (OpenClaw Control UI), **two VPS, two configs, two unit names** — do not confuse them.

---

## 1. Goals of the full session (in order)

1. Restyle viral tools **HTML catalog** (Tokyo purple → VS Code mid-gray → bold high-contrast).
2. Merge **credentials / API keys** into `vip_hub_merged.html` (`Add this.md`, `api-new.md`, Prodia).
3. Install **Hybrid 4.7** social automation memory (Grok / Claude / AGY).
4. **SCP** `api-new.md` to three VPS.
5. Permanently fix **OpenClaw** on ai-developer (+ later Caddy modular recovery).
6. Permanently fix **ZeroClaw** on contabo2 (auth + systemd + models/providers).

---

## 2. Chronological action log

### A. Catalog HTML theme

| Step | Action | How |
|------|--------|-----|
| A1–A4 | Theme + bold text | CSS tokens → VS Code `#1E1E1E` stack; scoped guide vars; high-contrast bold type. |

**File:** `/Users/khaledahmedmohamed/designs-content/vip/MERGED_AI_Viral_Video_Manga_Anime_Complete_2026_v2_Catalog.html`

---

### B. VIP Hub APIs / credentials

| Step | Action | How |
|------|--------|-----|
| B1–B4 | Hub + vault | Chrome/Gemini from Add this; OpenRouter/MuAPI/Prodia from api-new; SCP vault to 3 hosts. |

**Files:** `vip_hub_merged.html`, `api-new.md`

---

### C. Hybrid 4.7 permanent agent memory

| Step | Action | How |
|------|--------|-----|
| C1–C5 | Always-on policy | JSON v4.7.1, pro-tips, AGENTS.md blocks, skill symlink, Claude project memory. |

---

### D. OpenClaw on **ai-developer** (`213.199.36.17`) — merged detail

#### D1. Failure modes observed (why it broke)

| Issue | Evidence | Effect |
|-------|----------|--------|
| Gateway down mid-update | `npm i openclaw@latest` / `openclaw update` running | Caddy → **503** `connection refused` on `127.0.0.1:18789` |
| Node engine mismatch | OpenClaw **2026.7.1** needs Node **≥24.15**; system **24.13.1** | CLI refused to start after update |
| Missing / wrong unit type | No reliable **system** unit; user unit incomplete | Not boot-safe |
| Port fights (historical) | System unit **and** user unit both on `18789` | Cyclic kill/restart |
| Caddyfile overwrite | Config truncated to only `contabo213.addict.best` | Other domains / OpenClaw routing broken until restored |
| Bad Caddy patterns | `www` redirects without DNS; forced WS `Upgrade`/`Connection` headers | LE failures; HTTP/2 WebSocket breakage |

#### D2. Actions taken & how

1. **Diagnostics:** `ss`/`ps`, Caddy logs (`no upstreams available`), health check to `127.0.0.1:18789`.
2. **Node:** Permanent **v24.18.0** under `/opt/node/current` (+ nvm path during first recovery).
3. **Package:** OpenClaw under `/opt/openclaw` (not interactive-only nvm).
4. **Systemd:** `/etc/systemd/system/openclaw-gateway.service` — `enable` + `Restart=always`.
5. **Healthcheck:** `openclaw-healthcheck.timer` → `/usr/local/sbin/openclaw-healthcheck.sh` every 2 min.
6. **Safe update:** `/usr/local/sbin/openclaw-safe-update.sh`.
7. **Caddy (follow-on session content from home summary):**
   - Recovered blocks from `/etc/caddy/Caddyfile.bak`.
   - Removed non-DNS `www` redirects.
   - Removed redundant global WS upgrade headers (Caddy handles upgrades).
   - Validated via local scratch → `/tmp/Caddyfile` → `caddy validate` → deploy.
   - Verified multi-domain **200**: `openclaw`, `contabo213`, `prompting`, `prompts`, `prodemy.promedic1.com`.
8. **Runbook:** `/root/OPENCLAW_PERMANENT.md`.

#### D3. Final OpenClaw status (ai-developer)

- Gateway: **active**, `127.0.0.1:18789`
- Public: https://openclaw.addict.best/ → **200**
- Caddy: multi-site routing restored
- Persistence: system unit + healthcheck + `/opt` Node

---

### E. ZeroClaw on **contabo2** (`149.102.150.185`) — merged detail

#### E1. Failure modes observed (why it “looked up” but failed)

| Issue | Evidence | Effect |
|-------|----------|--------|
| HTML OK, chat dead | Page **200** | Users saw UI shell |
| WS unauthorized | Logs: `token_missing` / `closed before connect` code **1008** | Chat/session unusable |
| Auth misaligned | `gateway.auth.mode=token` + token **not** in Control UI | Reverse-proxy clients never sent token |
| Fragile process manager | PM2 `npx openclaw` via `/root/zeroclaw/start.sh` | Restarts/version drift |
| Dead user systemd | `openclaw-gateway.service` (user) **failed** since ~2026-06-28 | False sense of systemd coverage |
| Port dual-ownership risk | PM2 vs any systemd on `18789` | Conflicts |
| Schema validation crash (follow-on) | Invalid keys `autoApprove` / `autoRemoveFiles` in `/root/.openclaw/openclaw.json` | Startup validation failure |
| Wrong root config pollution | Root `~/.openclaw` vs project `/root/zeroclaw` | Confusing dual configs |

#### E2. Actions taken & how

1. **Diagnostics:** Caddy site `zero.master1.vip.caddy` → `reverse_proxy 127.0.0.1:18789`; PM2 show; journal/WS logs; `OPENCLAW_CONFIG_PATH=/root/zeroclaw/openclaw.json`.
2. **Auth fix (public Control UI):**
   - `gateway.auth.mode = "none"`
   - `allowInsecureAuth` + `dangerouslyDisableDeviceAuth`
   - Origins: `https://zeroclaw.addict.best`, local, `zero.master1.vip`, `auto.master1.vip`
   - Backup: `openclaw.json.bak.pre-authfix`
3. **Permanent runtime:** `/opt/node` + `/opt/openclaw` + `zeroclaw-gateway.service` + healthcheck timer + safe-update.
4. **Remove dual managers:** `pm2 delete zeroclaw-gateway` + `pm2 save`; remove failed user unit.
5. **Models/providers (follow-on from setting summary):**
   - Primary: `openrouter/qwen/qwen3-coder:free`
   - Fallbacks: `google/gemini-2.5-flash`, `openrouter/nvidia/nemotron-nano-9b-v2:free` (as documented in source B)
   - Keys from `api-new.md` (Gemini doctorhealthy) + openrouter config MD
6. **Schema cleanup:** stripped invalid keys from `/root/.openclaw/openclaw.json` so validators pass.
7. **Runbook:** `/root/ZEROCLAW_PERMANENT.md`.

#### E3. Final ZeroClaw status (contabo2)

- Gateway: **systemd** `zeroclaw-gateway` **active/enabled**
- Auth: **`{"mode":"none"}`** for public reverse-proxy UI
- Public: https://zeroclaw.addict.best/ + `/chat?session=main` → **200**
- Watchdog: `zeroclaw-healthcheck.timer` every 2 min
- PM2 gateway: **removed** (other PM2 apps left intact)

---

## 3. Final results matrix (with reasons)

| Deliverable | Result | Why it worked |
|-------------|--------|----------------|
| Catalog UX | VS Code mid-gray + bold | Tokens + no nested `:root` leak |
| VIP hub secrets | Complete APIs incl. Prodia | MD → cards; recovered unsaved Prodia |
| Hybrid 4.7 memory | Multi-agent always-on | JSON + skills + AGENTS.md |
| api-new on 3 VPS | Present `chmod 600` | Shared vault path |
| **openclaw.addict.best** | Healthy + permanent | Upstream up + Node range + system unit + Caddy restore |
| **zeroclaw.addict.best chat** | Healthy + permanent | Auth aligned with reverse proxy + single process owner |
| Multi-domain Caddy (ai-dev) | Several addict/promedic domains 200 | Restored modular/site blocks; no DNS-less redirects |

---

## 4. All output / modified files (full paths) + purpose

### Local Mac — content & reports

| Full path | Goal / use |
|-----------|------------|
| `/Users/khaledahmedmohamed/designs-content/vip/MERGED_AI_Viral_Video_Manga_Anime_Complete_2026_v2_Catalog.html` | Tools catalog UI theme. |
| `/Users/khaledahmedmohamed/designs-content/vip/setting/vip_hub_merged.html` | Credentials + API hub. |
| `/Users/khaledahmedmohamed/designs-content/vip/setting/api-new.md` | API vault source of truth. |
| `/Users/khaledahmedmohamed/designs-content/vip/Add this.md` | Chrome/Gemini source. |
| `/Users/khaledahmedmohamed/designs-content/vip/new-hybrid-4.7.md` | Hybrid master spec. |
| `/Users/khaledahmedmohamed/designs-content/vip/setting/openrouter-configurations/qwen-open-router-config.md` | OpenRouter free-tier key source (ZeroClaw models). |
| `/Users/khaledahmedmohamed/openclaw-session-summary.md` | **Source A** — ai-developer OpenClaw + Caddy notes. |
| `/Users/khaledahmedmohamed/designs-content/vip/setting/openclaw-session-summary.md` | **Source B** — contabo2 ZeroClaw notes. |
| `/Users/khaledahmedmohamed/designs-content/vip/setting/SESSION_SUMMARY_2026-07-14_to_15_Catalog_Hub_Hybrid47_OpenClaw_ZeroClaw.md` | **This comprehensive merge** (canonical). |
| `/Users/khaledahmedmohamed/.gemini/antigravity-cli/brain/7b94ebc8-9c8c-4487-8253-50b6e6bb470b/scratch/Caddyfile` | Local Caddy merge/validate scratch before push (ai-dev). |

### Local Mac — agent permanent memory

| Full path | Goal / use |
|-----------|------------|
| `~/.gemini/config/permanent_memory/hybrid_47_social_memory.json` | Hybrid 4.7 structured memory. |
| `~/.gemini/config/permanent_memory/README.md` | Load order docs. |
| `~/.agents/skills/hybrid-47-social/**` | Skill + pro-tips + anti-block. |
| `~/.grok/skills/hybrid-47-social` | Symlink for Grok. |
| `~/.grok/AGENTS.md` · `~/Claude.md` · `~/Agents.md` · `~/.config/opencode/AGENTS.md` | Always-on contracts. |
| `~/.claude/projects/-Users-khaledahmedmohamed/memory/hybrid-47-social-permanent.md` | Claude Code memory. |
| `~/.claude/projects/-Users-khaledahmedmohamed/memory/MEMORY.md` | Index. |

### Remote — ai-developer (`213.199.36.17`)

| Full path | Purpose |
|-----------|---------|
| `/root/designs-content/vip/setting/api-new.md` | API vault copy. |
| `/etc/caddy/Caddyfile` (+ bak) | Multi-site reverse proxy (restored/cleaned). |
| `/tmp/Caddyfile` | Safe validate staging. |
| `/etc/systemd/system/openclaw-gateway.service` | Boot-permanent gateway. |
| `/etc/systemd/system/openclaw-healthcheck.{service,timer}` | Auto-heal. |
| `/usr/local/sbin/openclaw-healthcheck.sh` | Probe + restart. |
| `/usr/local/sbin/openclaw-safe-update.sh` | Safe package upgrade. |
| `/opt/node/current` · `/opt/openclaw` | Stable Node + OpenClaw. |
| `/usr/local/bin/openclaw` · `/etc/profile.d/openclaw-node.sh` | CLI + PATH. |
| `/root/OPENCLAW_PERMANENT.md` | Ops runbook. |

### Remote — contabo2 (`149.102.150.185`)

| Full path | Purpose |
|-----------|---------|
| `/root/designs-content/vip/setting/api-new.md` | API vault copy. |
| `/root/zeroclaw/openclaw.json` (+ `.bak.pre-authfix`) | Auth/origins/models for ZeroClaw. |
| `/root/.openclaw/openclaw.json` | Root config cleaned of invalid schema keys. |
| `/root/zeroclaw/start.sh` | Fallback start using `/opt`. |
| `/etc/systemd/system/zeroclaw-gateway.service` | Boot-permanent gateway. |
| `/etc/systemd/system/zeroclaw-healthcheck.{service,timer}` | Auto-heal. |
| `/usr/local/sbin/zeroclaw-healthcheck.sh` · `zeroclaw-safe-update.sh` | Probe / upgrade. |
| `/opt/node/current` · `/opt/openclaw` | Stable runtime. |
| `/usr/local/bin/openclaw` · `/etc/profile.d/zeroclaw-node.sh` | CLI + PATH. |
| `/root/ZEROCLAW_PERMANENT.md` | Ops runbook. |
| `/etc/caddy/sites/zero.master1.vip.caddy` | Site block → proxy 18789. |

### Remote — hetzner (`46.62.228.173`)

| Full path | Purpose |
|-----------|---------|
| `/root/designs-content/vip/setting/api-new.md` | API vault copy only (this session). |

---

## 5. Future learn (merged — keep forever)

### From full session + both OpenClaw summaries

1. **Nested CSS `:root` pollutes globals** — scope component tokens.
2. **HTTP 200 ≠ working agent UI** — always verify WebSocket connect + logs.
3. **`auth.mode=token` behind public reverse proxy without UI token = chat death** — use `auth.mode=none` only with loopback bind + Caddy + origin allowlist, **or** put `&token=` on dashboard URL deliberately.
4. **`openclaw update` stops the daemon** — use `*-safe-update.sh` only.
5. **Node ≥24.15 for OpenClaw 2026.7.x** — system 24.13.1 is a landmine.
6. **One process owner for port 18789** — never PM2 + systemd together.
7. **Never overwrite Caddyfile for one new site** — modular `import /etc/caddy/sites/*`.
8. **Don’t force WebSocket Upgrade headers in Caddy** — breaks HTTP/2; let `reverse_proxy` handle it.
9. **Only proxy hostnames that have DNS** — no `www` LE challenges without A/AAAA.
10. **Validate OpenClaw config before start:** `openclaw config validate` / catch unknown keys (`autoApprove`, etc.).
11. **Dual config dirs confuse ops** — know which of `/root/zeroclaw` vs `~/.openclaw` is live via env.
12. **Unsaved editor buffers** can hold secrets missing from disk — recover then persist.
13. **Permanent agent memory needs multi-home write.**
14. **Build-on, don’t replace** configs/hubs/unified merges.

---

## 6. Errors avoidance checklist (merged)

| Risk | Avoid by |
|------|----------|
| 503 public | `ss -lntp \| grep 18789` then Caddy reverse_proxy |
| WS token_missing | Align auth mode with public UI; or append `&token=` |
| Schema crash on start | `openclaw config validate`; strip unknown keys |
| Port fight | Single manager; `pm2 delete` + `pm2 save` if systemd owns it |
| Node range fail | Systemd ExecStart → `/opt/node/current/bin/node` |
| Caddy wipe | `import sites/*`; never full-file replace for one domain |
| HTTP/2 WS break | No manual `Upgrade`/`Connection` header_up |
| LE fail on www | DNS first, then cert |
| Mid-update downtime | Safe-update script + healthcheck timer |
| Model empty after change | Keep previous model in `fallbacks[]` when patching providers |
| Theme / catalog damage | Edit CSS tokens only; leave DATA JSON unless intentional |

---

## 7. Build-on (not replace) strategy

### Principles
1. **Append cards** in hub arrays; never wipe `const APIS`.
2. **Backup before mutate** (`*.bak.pre-*`, Caddyfile.bak).
3. **Patch JSON with jq/programmatic merge** for models/providers — don’t rewrite whole file blindly.
4. **Caddy modular files** per domain under `/etc/caddy/sites/`.
5. **Low-review content:** add `### Expert tips` / `### Ops notes` sections; don’t delete body text.
6. **Fallback anchors** when changing primary models.

### Enhancement targets

| File / area | Build-on next step |
|-------------|-------------------|
| `vip_hub_merged.html` | Filter by API tag; optional `expires` on secrets. |
| `api-new.md` | Append `## Changelog` + `## Archived`; never delete old keys silently. |
| Catalog HTML DATA | Add optional `tips` field per tool. |
| Hybrid pro-tips | Expand playbooks under same § IDs. |
| Caddy (both servers) | Enforce `import /etc/caddy/sites/*` pattern everywhere. |
| Server runbooks | Add “status card” links into hub MISC. |
| Source A/B openclaw summaries | Keep as short historical; **update this comprehensive file** for new work. |

### Modular Caddy recipe (from source A)
```caddyfile
{
    email admin@ielts.fast
    grace_period 30s
}
import /etc/caddy/sites/*
```
One file per domain → adding a site never wipes OpenClaw.

---

## 8. Tips & recommendations

1. Prefer `openclaw-safe-update.sh` / `zeroclaw-safe-update.sh` on prod.
2. After any gateway change: curl local `18789` **and** open chat WS once.
3. If auth stays `token`, document token in vault and use dashboard URL with token — don’t leave operators guessing.
4. Consider Caddy basic-auth or IP allowlist **in front of** `auth.mode=none` (add layer, don’t remove loopback bind).
5. Keep primary + 2 free fallbacks on OpenRouter/Gemini for ZeroClaw.
6. Continue `SESSION_SUMMARY_*.md` after multi-server sessions; merge satellite summaries into the comprehensive file.

### Alternatives researched (source B — optional future)

| Option | Why considered |
|--------|----------------|
| **NanoClaw** | Container sandboxing / security |
| **Nanobot** | Lightweight Python + MCP |
| **n8n / Activepieces** | Visual messaging workflows |
| **Dify** | Enterprise model orchestrator UI |

Keep OpenClaw/ZeroClaw where already production-hardened; evaluate alternatives only for greenfield agents.

---

## 9. Quick ops reference

### ai-developer — OpenClaw
```bash
ssh -i ~/.ssh/ai_developer_key root@213.199.36.17
systemctl status openclaw-gateway openclaw-healthcheck.timer
journalctl -u openclaw-gateway -f
/usr/local/sbin/openclaw-safe-update.sh
```

### contabo2 — ZeroClaw
```bash
ssh -i ~/.ssh/contabo2_new1 root@149.102.150.185
systemctl status zeroclaw-gateway zeroclaw-healthcheck.timer
journalctl -u zeroclaw-gateway -f
/usr/local/sbin/zeroclaw-safe-update.sh
```

| Command | Use |
|---------|-----|
| `ssh ...` | Enter the VPS |
| `systemctl status ...` | Is gateway + heal timer up? |
| `journalctl -u ... -f` | Live logs (WS auth, crash loops) |

### Public URLs
- https://openclaw.addict.best/
- https://zeroclaw.addict.best/chat?session=main
- Hub: `file:///Users/khaledahmedmohamed/designs-content/vip/setting/vip_hub_merged.html`

---

## 10. Session scorecard

| Area | Outcome |
|------|---------|
| Catalog UX | Done |
| Secrets hub | Done |
| Hybrid 4.7 memory | Done |
| api-new × 3 VPS | Done |
| OpenClaw (ai-developer) permanence + Caddy | Done |
| ZeroClaw (contabo2) auth + permanence + models | Done |
| Merge of both openclaw-session-summary.md into one comprehensive file | **Done (this file)** |

---

## 11. Source files absorbed

| Source | Path | Content absorbed |
|--------|------|------------------|
| A | `/Users/khaledahmedmohamed/openclaw-session-summary.md` | ai-dev Caddy truncate/www/WS headers; multi-domain 200s; modular Caddy; scratch Caddyfile path |
| B | `/Users/khaledahmedmohamed/designs-content/vip/setting/openclaw-session-summary.md` | contabo2 schema keys; OpenRouter/Gemini model config; alternatives list; validate tip |
| C | This comprehensive path (prior revision) | Catalog, hub, Hybrid 4.7, SCP, both permanent stacks |

**Canonical future updates:** edit **this comprehensive file only**; leave A/B as dated satellites or add a one-line pointer at their top:

```markdown
> Superseded for comprehensive history by:
> `/Users/khaledahmedmohamed/designs-content/vip/setting/SESSION_SUMMARY_2026-07-14_to_15_Catalog_Hub_Hybrid47_OpenClaw_ZeroClaw.md`
```

---

*End of comprehensive merge. Build on this document; do not replace satellite notes without pointing here.*