If I had the space (current desktop is a Windows box for several reasons), I'd probably be running WindowMaker.
There's no well-functioned window manager out there that can beat it for efficiency. It's small and it is fast.
My last speed comparison on various startup X environments was long ago, and I didn't have a Beastie box to compare with.
All times below are from hitting Return after the last character of the password to the time the desktop was fully up and functional, including a command-line window on the desktop. Intel Core Duo processor, 16GB RAM, which, for 10 years ago, was pretty beefy.
Windows 7: 40s
Linux/WindowMaker: a second and a half.