I had Solaris 10 working wonderfully inside my vmware server on windows XP 64 bit. vmware tools was installed, and I could get full 1280x1024 resolution in full-screen mode.
Then I used Solaris update manager to install all its recommended patches.
Next time the guest OS came up, xorg.conf did not seem to be correct. The screen resolution was double what I set in xorg.conf. I ran the vmware-install.pl script again and chose 1280x1024 resolution. But now when I cntrl-alt-enter, I get an error telling me "Your host does not support the current guest settings in full screen mode. Try reducing your guest's resolution or color depth and try again. Failed to switch to full screen mode."
If I reduce the resolution to 1152x921 it will switch into full screen mode, but it doesn't fill the screen. I've played with the Hor/Vert sync, but no luck.
It "just worked" the first time I tried this, and now the patches have broken something. 