xdm runs as root so it's looking for a different magic cookie and finding it.  It isn't really solving your 'problem'.  As @pin  writes, using a static IP and putting your hostname in /etc/hosts should "solve" that.
You don't need to reinstall anything to use or not use xdm.  Just change /etc/rc.conf xdm=YES or xdm=NO
Really all xdm does for you is give you graphical login and an express ticket to X.  Unless you always want to boot and go straight to X, it's easier to not use it.  If you want to run it but not at startup, set it to NO in rc.conf and doas service xdm onestart or just doas xdm
You can startx cwm, no?
There are a dozen or so scripts that run at X startup; any one of them could be causing ctwm to start.
Until you get familiar with ironing out these issues, I wouldn't use desktop installer  (mostly because I don't know what it all does)