dev_arc_aws/docs/rdp-debug-handoff.md

RDP Connection Debugging — Handoff Doc

✅ WORKING NOW (2026-06-22) — XFCE over xrdp, full input + 1080p

RDP through ArchNest works end-to-end on the test Fedora VM (192.168.122.55, user sam) with the XFCE desktop. Confirmed by the user: logged in, desktop renders, mouse + keyboard work, 1080p. The full chain of fixes that got it there (each was a separate blocker, each its own deployed PR):

1. Auth/security (PR #41): gnome-remote-desktop mandates NLA, which guacd's FreeRDP 2 can't do → replaced it with xrdp on the VM (xrdp interoperates with guacd FreeRDP 2).
2. Desktop session (PR #42): GNOME 50 is Wayland-only and won't run on xrdp's Xorg backend → installed XFCE + created /etc/xrdp/startwm.sh to launch it.
3. Rendering (PR #43): blank despite XFCE running → disabled xfwm4 compositing (use_compositing=false) + LIBGL_ALWAYS_SOFTWARE=1 (no GL on the Virtio GPU).
4. Display scaling (PR #44): client scaled the Guacamole display to the panel.
5. Connection stability (PR #46): ArchNest backend now echoes the guacamole-common-js tunnel ping (0.,4.ping,...) that guacamole-lite 1.2.0 wrongly forwarded to guacd, which fixed the flicker→drop→reconnect loop ("User is not responding"). See guacamole.ts.
6. Input (PR #47): wired Guacamole.Mouse (scaled coords) + Guacamole.Keyboard in RemoteDesktop.tsx — the display rendered but no input was captured before this.
7. Default resolution (PR #48): guacd connectionDefaultSettings now 1920x1080.

ArchNest connection setting that works: Security = any (blank also fine). nla/rdp do not work against xrdp's default config.

⚠️ Desktop environment support matrix (the open task)

Goal (from the user): get GNOME and KDE working too, on latest versions — not just XFCE.

Desktop	Over xrdp (X11 backend)	Over native Wayland RDP	Works with guacd (FreeRDP 2)?
XFCE	✅ works (current setup)	n/a (X11 DE)	✅ yes, via xrdp
KDE Plasma 6	✅ should work via startplasma-x11 (X11 session still shipped through ~early 2027)	KRdp (Wayland) — newer, finicky	✅ via xrdp X11 session; ⚠️ via KRdp
GNOME 50	❌ no Xorg session exists (Wayland-only) — xrdp can't launch it	gnome-remote-desktop mandates NLA	❌ blocked both ways with FreeRDP 2

Why GNOME is the hard one (verified on the VM, not theory)

• GNOME 50 ships only Wayland sessions (/usr/share/xsessions/ has no gnome*.desktop; only gnome.desktop/gnome-classic.desktop under wayland-sessions/). So the xrdp route that works for XFCE/KDE-X11 cannot launch GNOME at all.
• GNOME's own Wayland RDP (gnome-remote-desktop) requires NLA and grdctl exposes no way to disable it (set-auth-methods only accepts credentials/kerberos, both NLA). guacd's FreeRDP 2 can't complete that NLA handshake → "wrong security type". Confirmed again this session: grdctl --help shows no TLS-only/disable-NLA option in v50.1.

Researched paths to make GNOME work (pick when picking this up)

1. Best long-term: get FreeRDP 3 into guacd. FreeRDP 3 can do gnome-remote-desktop NLA. Apache's official guacamole/guacd images (1.5.5 and 1.6.0) still ship FreeRDP 2.11.x, so this needs a custom guacd image built against FreeRDP 3 (guacamole-server has FreeRDP-3 build support; ~30-min from-source build to maintain in docker-compose.yml). This would unblock GNOME's native Wayland RDP and is the cleanest fix for any modern GNOME/Ubuntu 24.04+ target other ArchNest users hit. Recommended primary direction. Validate by rebuilding the guacd service and re-running the security=nla connect test against gnome-remote-desktop.
2. GNOME headless "system" RDP (GNOME 46+/48+), the intended modern path. gnome-remote-desktop has a system daemon + GDM handover mode (a separate path from the per-user gnome-remote-desktop.service; note the VM already has a gnome-remote-desktop-configuration.service unit present but disabled). It starts a headless GNOME Wayland session on connect, no monitor needed. BUT it still negotiates NLA, so it only helps once guacd has FreeRDP 3 (path 1). Refs: SUSE "Headless remote sessions in GNOME" parts 2–3; jamesnorth.net GRD setup; RHEL 10 "Administering RHEL by using the GNOME desktop" → remote desktop on headless server.
3. Fallback that works with FreeRDP 2 today: don't use GNOME's RDP at all — run a GNOME Xorg session under xrdp. Requires installing the GNOME-on-Xorg session (gnome-session-xsession / the classic Xorg session package) so a gnome.desktop appears in /usr/share/xsessions/, then point startwm.sh/~/.xsession at gnome-session. On Virtio GPU also force software GL (LIBGL_ALWAYS_SOFTWARE=1) and expect GNOME Shell to be heavy without GPU accel — GNOME Classic or "GNOME on Xorg" is lighter. Fedora has been dropping the GNOME Xorg session, so this may not be installable on 44+; verify availability first. Lower priority.

KDE Plasma 6 (should be straightforward, same pattern as XFCE)

• KDE is not installed on the VM yet. Two routes:
  • xrdp + Plasma X11 session (recommended now): install Plasma, ensure startplasma-x11 exists, point startwm.sh/~/.xsession at it (exec startplasma-x11). Same model as XFCE; works with guacd FreeRDP 2. Plasma keeps the X11 session through ~early 2027 (Plasma 6.8 goes Wayland-only). May also need compositing tweaks on the Virtio GPU (KWin: export KWIN_COMPOSE=Q for software/XRender, or disable compositing) to avoid a black screen.
  • KRdp (native Wayland, future-proof): Plasma 6's own Wayland RDP server. Newer, has its own black-screen/portal pitfalls, and (like gnome-remote-desktop) check its NLA behavior against guacd FreeRDP 2 before relying on it. Prefer once guacd has FreeRDP 3.

Suggested order of work for the next agent

1. Build & wire a FreeRDP-3 guacd image (path 1) — unblocks GNOME natively and benefits all users. Test against the VM's gnome-remote-desktop with security=nla.
2. Install KDE Plasma on the VM and prove the xrdp + startplasma-x11 route (quick win, no guacd change needed). Document the compositing/GL tweak if a black screen appears.
3. Only if FreeRDP 3 lands: enable GNOME headless system RDP and validate end-to-end.
4. Make the desktop/session choice per-host configurable rather than a global startwm.sh (e.g. xrdp session-select, or per-user ~/.xsession) so one VM can offer XFCE/KDE/GNOME.

---

✅ RESOLVED (2026-06-22) — root cause found, proven end-to-end

Root cause: guacd 1.5.5 ships FreeRDP 2.11.5, whose NLA/CredSSP client cannot complete authentication against gnome-remote-desktop, which mandates NLA.

Proven at every layer (not a theory — the EGL/Mesa/Zink hypothesis below was a red herring):

1. Server mandates NLA. Direct xfreerdp (v3) from the Fedora VM to its own gnome-remote-desktop returns, for /sec:tls and /sec:rdp: [WARN][com.freerdp.core.nego] Error: HYBRID_REQUIRED_BY_SERVER [0x00000005] → Protocol Security Negotiation Failure. grdctl rdp set-auth-methods only offers credentials (NLA) and kerberos — there is no non-NLA / plain-RDP mode to turn off.
2. guacd's FreeRDP 2 can't do NLA against it. Driving the real guacd path (guacd 172.18.0.2:4822 → VM) with security = nla, tls, rdp, AND any all return the identical Guacamole error Server refused connection (wrong security type?) (code 519). guacd's own log confirms it tried correctly: Security mode: NLA … then RDP server closed/refused connection: Server refused connection (wrong security type?). The fact that all four modes fail identically was the tell — it's not a mode mismatch, it's that FreeRDP 2's CredSSP handshake is incompatible with gnome-remote-desktop's.
3. Bumping guacd does NOT fix it. guacamole/guacd:1.6.0 still ships FreeRDP 2.11.7 (verified by inspecting the image). FreeRDP 3.x is what fixes gnome-remote-desktop NLA interop, and Apache's guacd image doesn't ship FreeRDP 3 yet. So an image bump is wasted.

Fix / recommendation (general — other ArchNest users will hit this)

gnome-remote-desktop is not a reliable RDP target for guacd-based gateways (this affects Fedora/Ubuntu 22.04+ desktops using GNOME's built-in "Remote Desktop"). The fix applied here, plus the alternative considered:

• Applied & verified (operational, per-VM): replaced gnome-remote-desktop with xrdp on the test VM. xrdp's RDP-security path interoperates with guacd's FreeRDP 2. Steps run: sudo dnf install -y xrdp && sudo systemctl enable --now xrdp; then disabled + masked gnome-remote-desktop's user service (systemctl --user mask gnome-remote-desktop.service) and killed the lingering daemon that was still holding port 3389 so xrdp could bind it. Verified end-to-end through the real guacd path: with security=any, guacd authenticates and streams live desktop frames. security MUST be any (or blank → defaults to any) for xrdp's default config — nla fails (Security negotiation failed) and rdp errors out. Note: xrdp gives a fresh X login session, not a takeover of the existing Wayland session.

  Gotcha (connected-but-blank screen): after installing xrdp you may connect successfully but see a blank/black screen. Cause: GNOME 50 on Fedora is Wayland-only (no Xorg session in /usr/share/xsessions/), so it cannot run on xrdp's Xorg backend — the session starts and dies in ~2s (xrdp-sesman logs Session on display N has finished). Fix: install a lightweight X11 desktop and point xrdp at it:

  sudo dnf install -y @xfce-desktop      # XFCE: X11, no GL needed, reliable over xrdp
  # /etc/xrdp/startwm.sh was missing on this VM — create it:
  sudo tee /etc/xrdp/startwm.sh >/dev/null <<'WM'
  #!/bin/sh
  [ -r /etc/profile ] && . /etc/profile
  export XDG_SESSION_TYPE=x11
  export XDG_CURRENT_DESKTOP=XFCE
  exec startxfce4
  WM
  sudo chmod +x /etc/xrdp/startwm.sh
  sudo systemctl restart xrdp-sesman xrdp
  

  Verified: xfce4-session + xfwm4 stay running and guacd streams sustained desktop frames.

  Second gotcha (still blank even with XFCE running): if all XFCE processes (xfce4-session/xfwm4/xfdesktop) are alive but the screen is still blank, it's xfwm4's GPU compositor failing on a GPU with no GL driver. Tell-tale lines in ~/.xsession-errors: xfwm4-WARNING: Another compositing manager is running on screen 0, glx: failed to create dri3 screen, failed to load driver: virtio_gpu. Fix — disable xfwm4 compositing and force software GL for the RDP user, then reconnect (kill the stale session first so it respawns):

  mkdir -p ~/.config/xfce4/xfconf/xfce-perchannel-xml
  cat > ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfwm4.xml <<'XML'
  <?xml version="1.0" encoding="UTF-8"?>
  <channel name="xfwm4" version="1.0">
    <property name="general" type="empty">
      <property name="use_compositing" type="bool" value="false"/>
    </property>
  </channel>
  XML
  # force llvmpipe so any GL app falls back to software instead of failing on virtio_gpu
  sed -i '1a export LIBGL_ALWAYS_SOFTWARE=1' ~/.xsession
  sudo pkill -u "$USER" xfce4-session; sudo pkill -u "$USER" Xvnc   # then reconnect
  

  Verified: fresh session has zero "Another compositing manager" errors and guacd streams a clean rendering desktop. This is the likely culprit on any VM with a Virtio/virtual GPU.

• Alternative (infra, affects everyone): a custom guacd build with FreeRDP 3. Not worth it yet — it's a 30+ min from-source build to maintain in docker-compose.yml, for one upstream gap that Apache will eventually close. Revisit if/when guacamole/guacd ships FreeRDP 3.

No ArchNest code change was required — the security field + ignore-cert handling in backend/src/routes/guacamole.ts (added earlier this debugging arc) are correct and remain useful for other RDP servers. The blocker was purely the guacd↔gnome NLA incompatibility.

The original investigation notes below are kept for history.

---

Goal

ArchNest is a self-hosted dashboard product. One of its integrations is a "Remote Desktop" connection type that proxies RDP/VNC/Telnet sessions through guacd (Apache Guacamole's proxy daemon) so users can open a remote desktop session in the browser. This needs to work reliably for any user's RDP server, not just this one — so the immediate goal is to get this specific connection working, but treat every root cause found as a potential general fix (config option, docs, code change) since other users will hit the same servers (gnome-remote-desktop, xrdp, Windows RDP, etc).

You have hands-on access to both machines involved. Use that — actively connect to both, run diagnostics on both sides simultaneously, and correlate logs/timestamps. Do not guess from one side alone; multiple times in this debugging session, a theory formed from only one machine's logs turned out to be wrong once the other machine's logs were checked.

The two machines

1. racknerd-712b73a — the VPS running the ArchNest stack (this repo) in Docker.

   • Container archnest-backend — the Node/Fastify backend. Route of interest: backend/src/routes/guacamole.ts — bridges a browser WebSocket to guacd using guacamole-lite's ClientConnection/Crypt classes. Builds a Guacamole connection token (protocol, hostname, port, username, password, domain, security, ignore-cert) and hands it to guacd.
   • Container archnest-guacd — Apache Guacamole's guacd (v1.5.5), the proxy daemon that actually speaks RDP/VNC/Telnet to the target. Listens on port 4822. On the archnest_default Docker network, internal IP 172.18.0.2, DNS aliases archnest-guacd/guacd. Backend env vars: ARCHNEST_GUACD_HOST=guacd, ARCHNEST_GUACD_PORT=4822.
   • Diagnostic command: docker logs -f archnest-guacd — shows each connection attempt, the security mode negotiated, certificate validation results, and the final success/refusal message from FreeRDP (the RDP client library guacd uses internally).
   • Also useful: docker exec archnest-backend env | grep ARCHNEST_GUACD, docker inspect archnest-guacd (to confirm network/IP), nc -zv 192.168.122.55 3389 (already confirmed reachable from racknerd).

2. Fedora VM (192.168.122.55) — appears to be a libvirt VM co-located on the same physical host as racknerd (it's in libvirt's default NAT range, and is reachable from racknerd over a private 192.168.x address despite racknerd otherwise looking like a public VPS). Running Fedora 44, GPU is a Red Hat, Inc. Virtio 1.0 GPU (rev 01) (confirmed via lspci). User sam, password happy2026 (test/lab credentials, not a real secret).

   • RDP is served by gnome-remote-desktop (GNOME's built-in RDP/VNC daemon), running as a per-user systemd service: systemctl --user status gnome-remote-desktop, systemctl --user restart gnome-remote-desktop.
   • Configured via the grdctl CLI: grdctl status --show-credentials, grdctl rdp enable, grdctl rdp set-credentials <user> <pass>, grdctl rdp set-tls-cert/set-tls-key, grdctl rdp disable-view-only.
   • Diagnostic command: journalctl --user -u gnome-remote-desktop -f — shows the daemon's own startup/shutdown/error logs.
   • There is a confirmed active, unlocked, real graphical session: loginctl list-sessions showed session 51 (seat0, tty2, class user), and loginctl show-session 51 -p Type -p State -p Active returned Type=wayland, Active=yes, State=active. So gnome-remote-desktop has a real Wayland session to attach to — this is NOT a "no session" problem.

What's already been fixed (confirmed working, do not re-investigate these)

1. DNS: an earlier hostname (fedora) didn't resolve from the backend container — resolved by using the IP 192.168.122.55 directly instead.
2. Self-signed cert rejection: FreeRDP/guacd rejected the target's self-signed RDP cert by default. Fixed in code — backend/src/routes/guacamole.ts now sets settings['ignore-cert'] = 'true' whenever protocol === 'rdp'. Confirmed deployed via docker exec archnest-backend grep -A2 "ignore-cert" /app/dist/routes/guacamole.js.
3. No way to override RDP security mode: added a security field to the connection token (settings.security = security || 'any') and exposed it in the Settings UI (src/pages/Settings.tsx, field key security, hint text about NLA). User has tried any, nla, tls, and rdp — all fail identically (see below).
4. GNOME's own RDP TLS cert was corrupt: journalctl showed [ERROR][com.freerdp.crypto] - [x509_utils_from_pem]: BIO_new failed for certificate / RDP server certificate is invalid. Fixed by regenerating the cert/key on the Fedora VM:

   openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/CN=fedora" \
     -keyout ~/.local/share/gnome-remote-desktop/rdp-tls.key \
     -out ~/.local/share/gnome-remote-desktop/rdp-tls.crt
   grdctl rdp set-tls-cert ~/.local/share/gnome-remote-desktop/rdp-tls.crt
   grdctl rdp set-tls-key ~/.local/share/gnome-remote-desktop/rdp-tls.key
   systemctl --user restart gnome-remote-desktop
   

   Confirmed fixed — later journal output shows no cert error on startup.
5. RDP sharing disabled at the gnome-remote-desktop level (grdctl status showed Status: disabled even though the daemon process was running and the port was listening). Fixed via grdctl rdp enable + systemctl --user restart gnome-remote-desktop.
6. Credentials missing / GNOME Keyring locked: grdctl rdp set-credentials sam happy2026 failed with Cannot create an item in a locked collection because the keyring wasn't unlocked (likely an artifact of an SSH-only login rather than a real unlocked graphical login). Fixed via:

   echo -n 'your-login-password' | gnome-keyring-daemon --unlock
   grdctl rdp set-credentials sam happy2026
   

   grdctl status --show-credentials now consistently shows Unit status: active, RDP: Status: enabled, Username: sam, Password: happy2026.

UPDATE: connection now succeeds, but screen is blank

The "Server refused connection (wrong security type?)" failure described below has since been resolved (cause not fully pinned down before it started working — likely one of the cert/credential/security-mode fixes finally lined up). The ArchNest Guacamole viewer now shows "Connected" in the top-right status, with the session named Fedora-WS — but the viewport is solid black, no desktop content ever renders. Confirmed NOT a lock-screen issue (user confirmed gnome is unlocked). journalctl --user -u gnome-remote-desktop -n 50 checked at the time showed the daemon has been running continuously since 10:34:45 with no crash (unlike earlier attempts where "RDP server started" was immediately followed by "RDP server stopped") — so this is a different failure mode than the original refusal: negotiation and session start now succeed, but no frame data is ever captured/sent.

This is strong evidence for the EGL/Mesa/Zink theory below — the daemon accepts the connection and starts the RDP server but apparently cannot capture real screen content, producing a connected-but-blank session instead of crashing outright. Next diagnostic step (not yet completed): tail journalctl --user -u gnome-remote-desktop -f AND journalctl --user -f | grep -i -E "pipewire|portal|screencast|monitor" live, while the black-screen session is open and the user clicks/moves the mouse in the Guacamole viewport, to catch any PipeWire/portal screencast error that doesn't appear in the regular unit log.

Original unresolved problem (superseded by the above, kept for history)

Connecting through ArchNest (browser → backend → guacd → Fedora VM) used to fail outright with:

Error: Server refused connection (wrong security type?)

This was tried with security set to any, nla, tls, and rdp — identical failure every time, regardless of mode. That's suspicious: if it were a genuine security negotiation mismatch, different modes should fail differently (or some should succeed). The fact that they all failed identically suggested the real failure might be happening after security negotiation succeeds — e.g. at session-start/framebuffer-creation time — and FreeRDP's client-side error message is a generic/misleading bucket for "the connection didn't complete," not literally a security-type mismatch. This symptom is no longer reproducing (see UPDATE above) — leave this section for historical context only.

Open theory (unconfirmed)

journalctl --user -u gnome-remote-desktop shows, on every daemon startup, EGL/Mesa/Zink rendering errors:

libEGL warning: failed to get driver name for fd -1
MESA-LOADER: failed to retrieve device information
MESA: error: ZINK: failed to choose pdev
libEGL warning: egl: failed to create dri2 screen

There was also one observed instance of "RDP server started" immediately followed by "RDP server stopped" with timing consistent with an actual connection attempt. The theory is that gnome-remote-desktop can't create a renderable framebuffer for screen capture (no working GPU/software-render path) and crashes/aborts when a client actually tries to start a session — which a FreeRDP client then reports as "wrong security type" because that's the generic refusal message FreeRDP shows for several different underlying failure modes.

This theory has NOT been confirmed. It's a leading hypothesis based on log timing correlation only — no one has yet proven the EGL/Mesa errors are causal vs. just noise from gnome-remote-desktop probing GPU paths at startup (which may be harmless/expected on a Virtio-GPU VM that falls back to software rendering anyway).

Diagnostic step that was in progress, never completed

A direct xfreerdp test, bypassing guacd entirely, to isolate whether gnome-remote-desktop rejects ANY RDP client (not just guacd/FreeRDP-via-guacd), or whether this is specific to how guacd's embedded FreeRDP negotiates. freerdp/xfreerdp has now been installed on both machines, but the actual test was never run/reported back. This should be your first move:

# From racknerd (mimics guacd's exact network path: container -> VM):
xfreerdp /v:192.168.122.55 /sec:tls /cert-ignore /u:sam /p:happy2026 +auth-only
xfreerdp /v:192.168.122.55 /sec:nla /cert-ignore /u:sam /p:happy2026 +auth-only
xfreerdp /v:192.168.122.55 /sec:rdp /cert-ignore /u:sam /p:happy2026 +auth-only

# From the Fedora VM itself (rules out networking, tests gnome-remote-desktop alone):
xfreerdp /v:localhost /sec:tls /cert-ignore /u:sam /p:happy2026 +auth-only

Run these WHILE simultaneously tailing both:

# on racknerd:
docker logs -f archnest-guacd
# on the Fedora VM:
journalctl --user -u gnome-remote-desktop -f

Correlate the exact moment of failure across both logs. This is the single most valuable piece of evidence currently missing.

Instructions

1. Get hands-on access to both racknerd-712b73a and the Fedora VM (192.168.122.55).
2. Run the xfreerdp direct tests above, with both logs tailing simultaneously, and read the actual FreeRDP client-side error output (not just "wrong security type" — xfreerdp's raw stderr/exit code will usually have more detail than what bubbles up through guacd/Guacamole's client to the ArchNest UI).
3. If xfreerdp succeeds where ArchNest's guac connection fails, the bug is in how backend/src/routes/guacamole.ts builds the connection settings/token, or in the guacamole-lite/guacd version compatibility — debug from there, comparing exactly what settings xfreerdp used successfully vs. what ArchNest sends.
4. If xfreerdp also fails identically, the problem is squarely on the gnome-remote-desktop / Fedora VM side. Investigate the EGL/Mesa/Zink rendering theory directly — check whether software rendering (llvmpipe) is available (glxinfo -B from an actual Wayland session, not an SSH shell — note: an earlier attempt from an SSH shell failed with Error: unable to open display, which is expected and not informative; you need to run it from within session 51 or equivalent), and whether the VM's libvirt XML has virtio-gpu with working 3D/virgl acceleration configured on the hypervisor side.
5. If gnome-remote-desktop turns out to be fundamentally unable to serve a real client (vs. screen-sharing GNOME's own "Remote Login" feature, which is its primary intended use case), consider recommending xrdp as a replacement RDP server on the Fedora VM, and note this in your report as a general product recommendation (since other ArchNest users may hit the same gnome-remote-desktop limitation).
6. Keep ArchNest's product goal in mind throughout: any fix that's specific to this user's VM is fine for unblocking them, but if you find a root cause that's likely to recur for other users (e.g. a guacd config default, a missing Settings field, a code bug in backend/src/routes/guacamole.ts), make the corresponding code/config fix in this repo, not just a one-off operational fix on this VM.

What to report back when done

Write a concise report (for the engineer/AI who handed this off) covering:

• The root cause, with the specific log lines/evidence that proved it (not just a theory).
• The exact fix applied, including any commands run on either machine and any code changes made in this repo (with file paths and diffs).
• Whether the fix is specific to this VM or represents a general product issue that other ArchNest users could hit — and if general, what was changed in the codebase to address it.
• Current working/non-working status of the connection after the fix, with the actual test performed to confirm it works end-to-end through ArchNest's UI (not just via direct xfreerdp).