Add data export/import (Phase 8): portable JSON backup of config + credentials

GET /api/data/export serializes all integrations (with decrypted secrets, for
cross-instance portability), bookmark categories, bookmarks, and tunnels;
POST /api/data/import restores them additively in a transaction with old->new
id remapping. Wires the Settings "Data & Backup" section to download/upload the
backup file. Verified end-to-end including cross-instance portability under a
different ARCHNEST_SECRET_KEY, plus browser verification of the Settings UI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbJV5nm8KPVH1oNJYKpnoF

TERMIX_MIGRATION.md

@@ -179,9 +179,26 @@ The **frontend transfer UI was also browser-verified** (Playwright/Chromium): lo
 
 All test artifacts (test `sshd`, both test OS users + their home dirs, test backend instance, test DB, temp files) were cleaned up afterward.
 
+### Phase 8 — Data Export / Import (DONE)
+
+**Architecture decision**: a single-file JSON backup/restore of the user's configuration — all integrations (with their credentials), bookmark categories + bookmarks, and tunnels. Secrets are exported **decrypted** on purpose: that makes a backup portable to a different ArchNest instance whose `ARCHNEST_SECRET_KEY` differs (an encrypted export would be useless after a key change / on a fresh install). The export is only ever served to an authenticated user — the same person who can already read those secrets via the integrations they own — and the UI labels it as containing plaintext credentials. Import is **additive** (insert-as-new, never destructive), with old→new id remapping so tunnels and bookmarks keep pointing at their correct newly-created parents, all wrapped in a single SQLite transaction.
+
+**What was built:**
+- `backend/src/routes/data.ts` — `GET /api/data/export` (serializes integrations+decrypted secrets, bookmark categories, bookmarks, tunnels with a `version` field) and `POST /api/data/import` (zod-validated, transactional, additive, with `integrationIdMap`/`categoryIdMap` remapping; tunnels referencing an integration absent from the import are skipped rather than orphaned).
+- `src/lib/api.ts` — `exportData()`/`importData()` + `DataExport` type.
+- `src/pages/Settings.tsx` — wired the previously-placeholder "Data & Backup" section to the real endpoints: Export downloads `archnest-backup-<date>.json`; Import reads a chosen file and POSTs it, with success/error feedback. (Replaced the old mock "Export Bookmarks"/"Clear Cache"/"Reset" buttons.)
+
+**Verified end-to-end** against a real backend (not mocked): seeded an instance with an SSH integration (password + passphrase secrets), a bookmark category + bookmark, and a tunnel; then:
+- **Export** returned `version: 1` with the secrets correctly **decrypted** to plaintext and all four entity types present.
+- **Additive import** into the same instance doubled every count, and the new tunnel's `integrationId` pointed at the *newly-created* integration (id remapping confirmed, not the stale original id).
+- **Cross-instance portability**: imported the backup into a second backend started with a **completely different `ARCHNEST_SECRET_KEY`**; re-exporting from that instance showed the credentials decrypt correctly under the new key — proving they were re-encrypted on import, which is the whole point of the decrypted-export design.
+- **Browser-verified** (Playwright/Chromium): the Settings → Data & Backup page exports a real downloaded JSON file (correct contents + success message) and imports an uploaded backup file (correct "Imported N integrations…" confirmation).
+
+All test artifacts (two test backend instances, test DBs, downloaded backup files, temp files) were cleaned up afterward.
+
 ### Also worth checking during/after the phases above
 
-- Data export/import of SSH hosts/credentials/file-manager data — a nice-to-have, not yet scheduled.
+_All previously-listed follow-ups are now complete: host-metrics widgets (Phase 6), host-to-host transfer (Phase 7), and data export/import (Phase 8) are done, and the verification gaps noted in Phases 1, 5, and 6 have been closed (cert auth, Telnet, RDP, guacd compose wiring, host-metrics network/ports/firewall + browser UI, and the Phase 1b/7 UI click-throughs)._
 
 ## Tracking
 

backend/src/routes/data.ts

@@ -0,0 +1,205 @@
+import type { FastifyInstance } from 'fastify'
+import { z } from 'zod'
+import { db, logEvent } from '../db/index.js'
+import { encryptSecret, decryptSecret } from '../db/crypto.js'
+
+/**
+ * Backup / restore of the user's configuration: integrations (with their
+ * credentials), bookmarks, and tunnels. Secrets are exported *decrypted* on
+ * purpose — that makes the backup portable to another ArchNest instance with a
+ * different ARCHNEST_SECRET_KEY. The export therefore contains plaintext
+ * credentials and should be treated as sensitive (it's only ever served to an
+ * authenticated user, over the same channel they'd use to read those secrets
+ * anyway via the integrations they own).
+ */
+
+const EXPORT_VERSION = 1
+
+interface IntegrationRow {
+  id: number
+  type: string
+  name: string
+  enabled: number
+  config_json: string
+}
+interface SecretRow {
+  key: string
+  value_encrypted: string
+}
+interface CategoryRow {
+  id: number
+  name: string
+  icon: string | null
+  sort_order: number
+}
+interface BookmarkRow {
+  id: number
+  category_id: number | null
+  title: string
+  url: string
+  icon: string | null
+  favorite: number
+}
+interface TunnelRow {
+  id: number
+  name: string
+  integration_id: number
+  mode: string
+  source_port: number
+  endpoint_host: string
+  endpoint_port: number
+  auto_start: number
+  max_retries: number
+  retry_interval_ms: number
+}
+
+const importSchema = z.object({
+  version: z.number().optional(),
+  integrations: z
+    .array(
+      z.object({
+        id: z.number(),
+        type: z.string(),
+        name: z.string(),
+        enabled: z.boolean().default(true),
+        config: z.record(z.string(), z.string()).default({}),
+        secrets: z.record(z.string(), z.string()).default({}),
+      }),
+    )
+    .default([]),
+  bookmarkCategories: z
+    .array(z.object({ id: z.number(), name: z.string(), icon: z.string().nullable().default(null), sortOrder: z.number().default(0) }))
+    .default([]),
+  bookmarks: z
+    .array(
+      z.object({
+        categoryId: z.number().nullable().default(null),
+        title: z.string(),
+        url: z.string(),
+        icon: z.string().nullable().default(null),
+        favorite: z.boolean().default(false),
+      }),
+    )
+    .default([]),
+  tunnels: z
+    .array(
+      z.object({
+        name: z.string(),
+        integrationId: z.number(),
+        mode: z.string(),
+        sourcePort: z.number(),
+        endpointHost: z.string().default(''),
+        endpointPort: z.number().default(0),
+        autoStart: z.boolean().default(false),
+        maxRetries: z.number().default(3),
+        retryIntervalMs: z.number().default(5000),
+      }),
+    )
+    .default([]),
+})
+
+export async function dataRoutes(app: FastifyInstance) {
+  app.addHook('onRequest', app.authenticate)
+
+  app.get('/api/data/export', async () => {
+    const integrations = (db.prepare('SELECT id, type, name, enabled, config_json FROM integrations').all() as IntegrationRow[]).map((row) => {
+      const secretRows = db.prepare('SELECT key, value_encrypted FROM secrets WHERE integration_id = ?').all(row.id) as SecretRow[]
+      const secrets: Record<string, string> = {}
+      for (const s of secretRows) {
+        try {
+          secrets[s.key] = decryptSecret(s.value_encrypted)
+        } catch {
+          // a secret we can't decrypt (wrong key) is skipped rather than failing the whole export
+        }
+      }
+      return { id: row.id, type: row.type, name: row.name, enabled: !!row.enabled, config: JSON.parse(row.config_json), secrets }
+    })
+
+    const bookmarkCategories = (db.prepare('SELECT id, name, icon, sort_order FROM bookmark_categories').all() as CategoryRow[]).map((c) => ({
+      id: c.id,
+      name: c.name,
+      icon: c.icon,
+      sortOrder: c.sort_order,
+    }))
+
+    const bookmarks = (db.prepare('SELECT category_id, title, url, icon, favorite FROM bookmarks').all() as BookmarkRow[]).map((b) => ({
+      categoryId: b.category_id,
+      title: b.title,
+      url: b.url,
+      icon: b.icon,
+      favorite: !!b.favorite,
+    }))
+
+    const tunnels = (db.prepare('SELECT name, integration_id, mode, source_port, endpoint_host, endpoint_port, auto_start, max_retries, retry_interval_ms FROM tunnels').all() as TunnelRow[]).map(
+      (t) => ({
+        name: t.name,
+        integrationId: t.integration_id,
+        mode: t.mode,
+        sourcePort: t.source_port,
+        endpointHost: t.endpoint_host,
+        endpointPort: t.endpoint_port,
+        autoStart: !!t.auto_start,
+        maxRetries: t.max_retries,
+        retryIntervalMs: t.retry_interval_ms,
+      }),
+    )
+
+    return { version: EXPORT_VERSION, exportedAt: new Date().toISOString(), integrations, bookmarkCategories, bookmarks, tunnels }
+  })
+
+  app.post('/api/data/import', async (req, reply) => {
+    const parsed = importSchema.safeParse(req.body)
+    if (!parsed.success) {
+      return reply.code(400).send({ error: parsed.error.issues[0]?.message ?? 'Invalid import file' })
+    }
+    const { integrations, bookmarkCategories, bookmarks, tunnels } = parsed.data
+
+    const counts = { integrations: 0, bookmarkCategories: 0, bookmarks: 0, tunnels: 0 }
+
+    // Additive import: insert everything as new rows, remapping old ids -> new ids so
+    // tunnels and bookmarks keep pointing at the right (newly-created) parents.
+    const runImport = db.transaction(() => {
+      const integrationIdMap = new Map<number, number>()
+      const insertIntegration = db.prepare('INSERT INTO integrations (type, name, enabled, config_json) VALUES (?, ?, ?, ?)')
+      const insertSecret = db.prepare('INSERT INTO secrets (integration_id, key, value_encrypted) VALUES (?, ?, ?)')
+      for (const i of integrations) {
+        const res = insertIntegration.run(i.type, i.name, i.enabled ? 1 : 0, JSON.stringify(i.config))
+        const newId = Number(res.lastInsertRowid)
+        integrationIdMap.set(i.id, newId)
+        for (const [key, value] of Object.entries(i.secrets)) {
+          insertSecret.run(newId, key, encryptSecret(value))
+        }
+        counts.integrations += 1
+      }
+
+      const categoryIdMap = new Map<number, number>()
+      const insertCategory = db.prepare('INSERT INTO bookmark_categories (name, icon, sort_order) VALUES (?, ?, ?)')
+      for (const c of bookmarkCategories) {
+        const res = insertCategory.run(c.name, c.icon, c.sortOrder)
+        categoryIdMap.set(c.id, Number(res.lastInsertRowid))
+        counts.bookmarkCategories += 1
+      }
+
+      const insertBookmark = db.prepare('INSERT INTO bookmarks (category_id, title, url, icon, favorite) VALUES (?, ?, ?, ?, ?)')
+      for (const b of bookmarks) {
+        const mappedCategory = b.categoryId !== null ? categoryIdMap.get(b.categoryId) ?? null : null
+        insertBookmark.run(mappedCategory, b.title, b.url, b.icon, b.favorite ? 1 : 0)
+        counts.bookmarks += 1
+      }
+
+      const insertTunnel = db.prepare(
+        'INSERT INTO tunnels (name, integration_id, mode, source_port, endpoint_host, endpoint_port, auto_start, max_retries, retry_interval_ms) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)',
+      )
+      for (const t of tunnels) {
+        const mappedIntegration = integrationIdMap.get(t.integrationId)
+        if (mappedIntegration === undefined) continue // tunnel referenced an integration not present in this import — skip
+        insertTunnel.run(t.name, mappedIntegration, t.mode, t.sourcePort, t.endpointHost, t.endpointPort, t.autoStart ? 1 : 0, t.maxRetries, t.retryIntervalMs)
+        counts.tunnels += 1
+      }
+    })
+
+    runImport()
+    logEvent('data_imported', `Imported ${counts.integrations} integrations, ${counts.bookmarks} bookmarks, ${counts.tunnels} tunnels`)
+    return { ok: true, imported: counts }
+  })
+}

backend/src/server.ts

@@ -15,6 +15,7 @@ import { dockerRoutes, dockerExecRoutes } from './routes/docker.js'
 import { guacamoleRoutes } from './routes/guacamole.js'
 import { metricsRoutes } from './routes/metrics.js'
 import { transferRoutes } from './routes/transfer.js'
+import { dataRoutes } from './routes/data.js'
 import { startAutoStartTunnels } from './tunnels/manager.js'
 
 const JWT_SECRET = process.env.ARCHNEST_JWT_SECRET
@@ -49,6 +50,7 @@ await app.register(dockerExecRoutes)
 await app.register(guacamoleRoutes)
 await app.register(metricsRoutes)
 await app.register(transferRoutes)
+await app.register(dataRoutes)
 
 app.get('/api/health', async () => ({ ok: true }))
 

src/lib/api.ts

@@ -152,6 +152,22 @@ export const api = {
   listTransfers: () => apiFetch<{ transfers: TransferProgress[] }>('/transfers'),
   getTransfer: (id: string) => apiFetch<TransferProgress>(`/transfers/${id}`),
   cancelTransfer: (id: string) => apiFetch<{ ok: boolean }>(`/transfers/${id}/cancel`, { method: 'POST' }),
+
+  exportData: () => apiFetch<DataExport>('/data/export'),
+  importData: (data: DataExport) =>
+    apiFetch<{ ok: boolean; imported: { integrations: number; bookmarkCategories: number; bookmarks: number; tunnels: number } }>('/data/import', {
+      method: 'POST',
+      body: JSON.stringify(data),
+    }),
+}
+
+export interface DataExport {
+  version: number
+  exportedAt?: string
+  integrations: Array<{ id: number; type: string; name: string; enabled: boolean; config: Record<string, string>; secrets: Record<string, string> }>
+  bookmarkCategories: Array<{ id: number; name: string; icon: string | null; sortOrder: number }>
+  bookmarks: Array<{ categoryId: number | null; title: string; url: string; icon: string | null; favorite: boolean }>
+  tunnels: Array<{ name: string; integrationId: number; mode: string; sourcePort: number; endpointHost: string; endpointPort: number; autoStart: boolean; maxRetries: number; retryIntervalMs: number }>
 }
 
 export interface AuthUser {

src/pages/Settings.tsx

@@ -894,31 +894,83 @@ function NotificationsSection() {
 }
 
 function DataBackupSection() {
+  const [busy, setBusy] = useState(false)
+  const [message, setMessage] = useState<string | null>(null)
+  const [error, setError] = useState<string | null>(null)
+  const importRef = useRef<HTMLInputElement | null>(null)
+
+  async function handleExport() {
+    setBusy(true)
+    setError(null)
+    setMessage(null)
+    try {
+      const data = await api.exportData()
+      const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' })
+      const url = URL.createObjectURL(blob)
+      const a = document.createElement('a')
+      a.href = url
+      a.download = `archnest-backup-${new Date().toISOString().slice(0, 10)}.json`
+      a.click()
+      URL.revokeObjectURL(url)
+      setMessage(`Exported ${data.integrations.length} integrations, ${data.bookmarks.length} bookmarks, ${data.tunnels.length} tunnels.`)
+    } catch (err) {
+      setError(err instanceof Error ? err.message : 'Export failed')
+    } finally {
+      setBusy(false)
+    }
+  }
+
+  async function handleImportFile(file: File) {
+    setBusy(true)
+    setError(null)
+    setMessage(null)
+    try {
+      const text = await file.text()
+      const parsed = JSON.parse(text)
+      const result = await api.importData(parsed)
+      const c = result.imported
+      setMessage(`Imported ${c.integrations} integrations, ${c.bookmarkCategories} categories, ${c.bookmarks} bookmarks, ${c.tunnels} tunnels.`)
+    } catch (err) {
+      setError(err instanceof Error ? err.message : 'Import failed — is this a valid ArchNest backup file?')
+    } finally {
+      setBusy(false)
+    }
+  }
+
   return (
     <div style={cardBase}>
       <h3 style={sectionTitle}>Data & Backup</h3>
-      <div className="flex flex-col gap-3" style={{ maxWidth: '320px' }}>
+      <p style={{ fontSize: '12px', color: '#7A7D85', marginBottom: '16px', maxWidth: '460px' }}>
+        Export a portable backup of all integrations (including their credentials), bookmarks, and tunnels as a single JSON file, or
+        import one into this instance. Imports are additive — existing data is kept and the backup's items are added alongside it.
+        The backup contains plaintext credentials, so store it securely.
+      </p>
+      <div className="flex flex-col gap-3" style={{ maxWidth: '460px' }}>
         <div className="flex items-center justify-between">
-          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Export Bookmarks (JSON)</span>
+          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Export all data (JSON)</span>
-          <GoldButton><Download size={13} /> Export</GoldButton>
+          <GoldButton onClick={handleExport} disabled={busy}>
+            <Download size={13} /> Export
+          </GoldButton>
         </div>
         <div className="flex items-center justify-between">
-          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Import Bookmarks (JSON)</span>
+          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Import from backup (JSON)</span>
-          <GoldButton><Upload size={13} /> Import</GoldButton>
+          <GoldButton onClick={() => importRef.current?.click()} disabled={busy}>
-        </div>
+            <Upload size={13} /> Import
-        <div className="flex items-center justify-between">
+          </GoldButton>
-          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Export Settings</span>
+          <input
-          <GoldButton><Download size={13} /> Export</GoldButton>
+            ref={importRef}
-        </div>
+            type="file"
-        <div className="border-t" style={{ borderColor: 'rgba(231,76,60,0.15)', margin: '8px 0' }} />
+            accept="application/json,.json"
-        <div className="flex items-center justify-between">
+            className="hidden"
-          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Clear Cache</span>
+            onChange={(e) => {
-          <GoldButton danger><Trash2 size={13} /> Clear</GoldButton>
+              const file = e.target.files?.[0]
-        </div>
+              if (file) handleImportFile(file)
-        <div className="flex items-center justify-between">
+              e.target.value = ''
-          <span style={{ fontSize: '13px', color: '#E8E6E0' }}>Reset to Defaults</span>
+            }}
-          <GoldButton danger><RotateCcw size={13} /> Reset</GoldButton>
+          />
         </div>
+        {message && <p style={{ fontSize: '12px', color: '#2ECC71' }}>{message}</p>}
+        {error && <p style={{ fontSize: '12px', color: '#E74C3C' }}>{error}</p>}
       </div>
     </div>
   )