VMControl error -8: Invalid operation for virtual machine's current state

We were running some VMs then we shutdown the server without shutting down each VM. We saw VMWare saying it was stopping them then it said it was  unable to stop all VMs. After the reboot any VM we tried to start would get:

VMControl error -8: Invalid operation for virtual machine's current state

Here is the vmware.log from that VM:

Feb 23 09:14:11: vmx| Log for VMware Server pid=13381 version=e.x.p build=build-20925 option=Release

Feb 23 09:14:11: vmx| Command line: "/usr/local/vmware/lib/bin/vmware-vmx" "-C" "-@" """" "/home/mycompany/vms/SERVER1/SERVER.vmx"

Feb 23 09:14:11: vmx| vmxvmdb: Index name being generated from config file

Feb 23 09:14:11: vmx| VMXVmdbConnectServerd - Trying to discover serverd

Feb 23 09:14:11: vmx| MStat: Creating Stat system.cpuusage

Feb 23 09:14:11: vmx| MStat: Creating Stat system.ram

Feb 23 09:14:11: vmx| MStat: Creating Stat system.uptime

Feb 23 09:14:11: vmx| MStat: Creating Stat system.load

Feb 23 09:14:11: vmx| pcpu #0 CPUID numEntries=1 AuthcAMDenti

Feb 23 09:14:11: vmx| pcpu #0 CPUID version=0xf51 id1.edx=0x78bfbff id1.ecx=0x0 id1.ebx=0x862

Feb 23 09:14:11: vmx| pcpu #0 CPUID id80.eax=80000018 id81.edx=0xe1d3fbff id81.ecx=0x0

Feb 23 09:14:11: vmx| CPUID id1.edx: 0x78bfbff id1.ecx: 0 id81.edx: 0xe1d3fbff id81.ecx: 0

Feb 23 09:14:11: vmx| CPUID id88.ecx: 0 id88.edx: 0

Feb 23 09:14:11: vmx| Setup symlink /var/run/vmware/%2Fhome%2Fmycompany%2Fvms%2FSERVER1%2FSERVER%2Evmx -> /var/run/vmware/mycompany/13381

Feb 23 09:14:11: vmx| ACL_InitCapabilities: here 1 (bug 63252)

Feb 23 09:14:11: vmx| changing directory to /home/mycompany/vms/SERVER1/.

Feb 23 09:14:11: vmx| Config file: /home/mycompany/vms/SERVER1/SERVER.vmx

Feb 23 09:14:12: vmx| LOG failed to remove /home/mycompany/vms/SERVER1/vmware-2.log failed: No such file or directory

Feb 23 09:14:12: vmx| Snapshot failed due to version error: 7

Feb 23 09:14:12: vmx| SNAPSHOT:SnapshotConfigInfoRead: Error 9

Feb 23 09:14:12: vmx| SNAPSHOT:GetTree failed 9

Feb 23 09:14:12: vmx| VMXVmdbGetCheckpoints: unable to initialize snapshot tree Snapshots are not allowed on this virtual machine.

Feb 23 09:14:12: vmx| Read from FIFO 32 -- connecting to serverd...

Feb 23 09:14:12: vmx| VMDB: Connected to serverd

Feb 23 09:14:12: vmx| Accepted new connection at 30 for thread control (0x84c2198)

Feb 23 09:14:12: vmx| VUINewControlConnection: before slow ACL gunk (bug 63252).

Feb 23 09:14:12: vmx| ACL_InitCapabilities: here 2 (bug 63252)

Feb 23 09:14:12: vmx| VUINewControlConnection: after slow ACL gunk (bug 63252).

Feb 23 09:14:12: vmx| VUI: A new VMControl client connected.

Feb 23 09:14:12: vmx| IPC version negotiation version: VMX returning 2.1 to control

Feb 23 09:14:12: vmx| IPC vmcontrol-temp version: VMX returning 11.4 to control that tried 11.4

Feb 23 09:14:12: vmx| TOOLS delaying state change request to state 3

Feb 23 09:14:12: vmx| PowerOn

Feb 23 09:14:12: vmx| Host ACPI: can't find SRAT

Feb 23 09:14:12: vmx| Snapshot failed due to version error: 7

Feb 23 09:14:12: vmx| SNAPSHOT:SnapshotConfigInfoRead: Error 9

Feb 23 09:14:12: vmx| SNAPSHOT:GetTree failed 9

Feb 23 09:14:12: vmx| VMXVmdbGetCheckpoints: unable to initialize snapshot tree Snapshots are not allowed on this virtual machine.

Feb 23 09:14:12: vmx| DISKUTIL: Offline toolsVersion = 4101

Feb 23 09:14:12: vmx| UPGRADE: VM is non-painful guest winNetStandard: It is alright to upgrade HW with out-of-date tools

Feb 23 09:14:12: vmx| Msg_Hint: msg.upgrade.legacyVM (not shown)

Feb 23 09:14:12: vmx| HOST sysname Linux, nodename localhost, release 2.6.15-gentoo-r1, version #5 PREEMPT Thu Feb 23 06:18:04 CST 2006, machine i686, hz=250

Feb 23 09:14:12: vmx| DICT --- USER PREFERENCES

Feb 23 09:14:12: vmx| DICT       pref.grabOnKeyPress = FALSE

Feb 23 09:14:12: vmx| DICT    pref.autoFitFullScreen = fitHostToGuest

Feb 23 09:14:12: vmx| DICT     pref.view.navBar.type = favorites

Feb 23 09:14:12: vmx| DICT     pref.mruDest0.present = TRUE

Feb 23 09:14:12: vmx| DICT  pref.mruDest0.destString = localhost

Feb 23 09:14:12: vmx| DICT        pref.mruDest0.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest1.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest1.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest1.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest2.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest2.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest2.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest3.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest3.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest3.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest4.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest4.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest4.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest5.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest5.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest5.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest6.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest6.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest6.user =

Feb 23 09:14:12: vmx| DICT     pref.mruDest7.present = FALSE

Feb 23 09:14:12: vmx| DICT  pref.mruDest7.destString =

Feb 23 09:14:12: vmx| DICT        pref.mruDest7.user =

Feb 23 09:14:12: vmx| DICT          pref.view.navBar = TRUE

Feb 23 09:14:12: vmx| DICT                  pref.kvm = FALSE

Feb 23 09:14:12: vmx| DICT       pref.placement.left = 0

Feb 23 09:14:12: vmx| DICT        pref.placement.top = 0

Feb 23 09:14:12: vmx| DICT      pref.placement.right = 1024

Feb 23 09:14:12: vmx| DICT     pref.placement.bottom = 696

Feb 23 09:14:12: vmx| DICT pref.console.openedObj0.present = TRUE

Feb 23 09:14:12: vmx| DICT pref.console.openedObj0.type = vm

Feb 23 09:14:12: vmx| DICT pref.console.openedObj0.path = /vm/#f6ddc19ce5b3e29e/

Feb 23 09:14:12: vmx| DICT pref.console.openedObj0.file = /home/mycompany/vms/CLIENT1/client.vmx

Feb 23 09:14:12: vmx| DICT pref.console.openedObj0.dest = /host2/#695f4e9b8d250684/

Feb 23 09:14:12: vmx| DICT pref.console.openedObj.maxNum = 3

Feb 23 09:14:12: vmx| DICT pref.console.currentObj.path = /vm/#54fa11e674119603/

Feb 23 09:14:12: vmx| DICT pref.console.currentObj.type = vm

Feb 23 09:14:12: vmx| DICT pref.console.openedObj1.present = TRUE

Feb 23 09:14:12: vmx| DICT pref.console.openedObj1.type = vm

Feb 23 09:14:12: vmx| DICT pref.console.openedObj1.path = /vm/#54fa11e674119603/

Feb 23 09:14:12: vmx| DICT pref.console.openedObj1.file = /home/mycompany/vms/SERVER1/SERVER.vmx

Feb 23 09:14:12: vmx| DICT pref.console.openedObj1.dest = /host2/#695f4e9b8d250684/

Feb 23 09:14:12: vmx| DICT        hint.vmui.poweroff = FALSE

Feb 23 09:14:12: vmx| DICT pref.console.openedObj2.present = TRUE

Feb 23 09:14:12: vmx| DICT pref.console.openedObj2.type = vm

Feb 23 09:14:12: vmx| DICT pref.console.openedObj2.path = /vm/#62db48dadbde3bf4/

Feb 23 09:14:12: vmx| DICT pref.console.openedObj2.file = /home/mycompany/vms/SERVER2/SERVER.vmx

Feb 23 09:14:12: vmx| DICT pref.console.openedObj2.dest = /host2/#695f4e9b8d250684/

Feb 23 09:14:12: vmx| DICT --- USER DEFAULTS

Feb 23 09:14:12: vmx| DICT --- HOST DEFAULTS

Feb 23 09:14:12: vmx| DICT    vmnet1.hostonlyaddress = 10.10.10.1

Feb 23 09:14:12: vmx| DICT     serverd.init.fullpath = /usr/local/vmware/lib/serverd/init.pl

Feb 23 09:14:12: vmx| DICT         authd.client.port = 902

Feb 23 09:14:12: vmx| DICT          control.fullpath = /usr/local/vmware/bin/vmware-cmd

Feb 23 09:14:12: vmx| DICT            authd.fullpath = /usr/local/vmware/sbin/vmware-authd

Feb 23 09:14:12: vmx| DICT             loop.fullpath = /usr/local/vmware/bin/vmware-loop

Feb 23 09:14:12: vmx| DICT                    libdir = /usr/local/vmware/lib

Feb 23 09:14:12: vmx| DICT           vmware.fullpath = /usr/local/vmware/bin/vmware

Feb 23 09:14:12: vmx| DICT    vmnet1.hostonlynetmask = 255.255.255.0

Feb 23 09:14:12: vmx| DICT                     vmdir = /home/mycompany/vms

Feb 23 09:14:12: vmx| DICT            dhcpd.fullpath = /usr/local/vmware/bin/vmnet-dhcpd

Feb 23 09:14:12: vmx| DICT          serverd.fullpath = /usr/local/vmware/sbin/vmware-serverd

Feb 23 09:14:12: vmx| DICT            datastore.name = local

Feb 23 09:14:12: vmx| DICT       datastore.localpath = /home/mycompany/vms/

Feb 23 09:14:12: vmx| DICT --- SITE DEFAULTS

Feb 23 09:14:12: vmx| DICT                  tag.help = introduction.htm

Feb 23 09:14:12: vmx| DICT   tag.configurationEditor = config_editor_newvm.htm

Feb 23 09:14:12: vmx| DICT             tag.ideConfig = devices_virtualdrive.htm

Feb 23 09:14:12: vmx| DICT          tag.floppyConfig = devices_floppy.htm

Feb 23 09:14:12: vmx| DICT           tag.mouseConfig = devices_mouse.htm

Feb 23 09:14:12: vmx| DICT             tag.netConfig = devices_netadapter.htm

Feb 23 09:14:12: vmx| DICT        tag.parallelConfig = devices_parallel.htm

Feb 23 09:14:12: vmx| DICT          tag.serialConfig = devices_serial.htm

Feb 23 09:14:12: vmx| DICT           tag.soundConfig = devices_sound.htm

Feb 23 09:14:12: vmx| DICT             tag.memConfig = configvm_memory.htm

Feb 23 09:14:12: vmx| DICT            tag.miscConfig = configvm.htm

Feb 23 09:14:12: vmx| DICT             tag.usbConfig = devices_usb.htm

Feb 23 09:14:12: vmx| DICT         tag.displayConfig = configvm_display-problems.htm

Feb 23 09:14:12: vmx| DICT                 tag.tools = vmtools.htm

Feb 23 09:14:12: vmx| DICT --- COMMAND LINE

Feb 23 09:14:12: vmx| DICT          gui.managementUI = TRUE

Feb 23 09:14:12: vmx| DICT --- CONFIGURATION

Feb 23 09:14:12: vmx| DICT               displayName = Windows Server 2003 Standard Edition

Feb 23 09:14:12: vmx| DICT                   memsize = 640

Feb 23 09:14:12: vmx| DICT            ide0:0.present = TRUE

Feb 23 09:14:12: vmx| DICT             scsi0.present = TRUE

Feb 23 09:14:12: vmx| DICT          scsi0.virtualDev = lsilogic

Feb 23 09:14:12: vmx| DICT           scsi0:0.present = TRUE

Feb 23 09:14:12: vmx| DICT          scsi0:0.fileName = root.vmdk

Feb 23 09:14:12: vmx| DICT              scsi0:0.mode = persistent

Feb 23 09:14:12: vmx| DICT           scsi0:1.present = TRUE

Feb 23 09:14:12: vmx| DICT          scsi0:1.fileName = shared.vmdk

Feb 23 09:14:12: vmx| DICT              scsi0:1.mode = persistent

Feb 23 09:14:12: vmx| DICT           scsi0:2.present = TRUE

Feb 23 09:14:12: vmx| DICT          scsi0:2.fileName = page.vmdk

Feb 23 09:14:12: vmx| DICT              scsi0:2.mode = independent-nonpersistent

Feb 23 09:14:12: vmx| DICT           scsi0:3.present = TRUE

Feb 23 09:14:12: vmx| DICT          scsi0:3.fileName = ISOs.vmdk

Feb 23 09:14:12: vmx| DICT              scsi0:3.mode = persistent

Feb 23 09:14:12: vmx| DICT          floppy0.fileType = file

Feb 23 09:14:12: vmx| DICT          floppy0.fileName = guestinfo.flp

Feb 23 09:14:12: vmx| DICT    floppy0.startConnected = TRUE

Feb 23 09:14:12: vmx| DICT         ethernet0.present = TRUE

Feb 23 09:14:12: vmx| DICT     ethernet0.addressType = static

Feb 23 09:14:12: vmx| DICT         ethernet0.Address = 00:50:56:00:AA:03

Feb 23 09:14:12: vmx| DICT  ethernet0.connectionType = hostonly

Feb 23 09:14:12: vmx| DICT            ethernet0.vnet = VMnet3

Feb 23 09:14:12: vmx| DICT      ethernet0.virtualDev = vmxnet

Feb 23 09:14:12: vmx| DICT         ethernet1.present = TRUE

Feb 23 09:14:12: vmx| DICT     ethernet1.addressType = generated

Feb 23 09:14:12: vmx| DICT ethernet1.generatedAddress = 00:0c:29:23:45:61

Feb 23 09:14:12: vmx| DICT ethernet1.generatedAddressOffset = 10

Feb 23 09:14:12: vmx| DICT  ethernet1.connectionType = bridged

Feb 23 09:14:12: vmx| DICT            ethernet1.vnet = VMnet4

Feb 23 09:14:12: vmx| DICT      ethernet1.virtualDev = vmxnet

Feb 23 09:14:12: vmx| DICT            tools.syncTime = FALSE

Feb 23 09:14:12: vmx| DICT            config.version = 7

Feb 23 09:14:12: vmx| DICT         virtualHW.version = 3

Feb 23 09:14:12: vmx| DICT             sound.present = FALSE

Feb 23 09:14:12: vmx| DICT            sound.fileName = -1

Feb 23 09:14:12: vmx| DICT                   guestOS = winnetstandard

Feb 23 09:14:12: vmx| DICT          priority.grabbed = normal

Feb 23 09:14:12: vmx| DICT        priority.ungrabbed = normal

Feb 23 09:14:12: vmx| DICT        powerType.powerOff = default

Feb 23 09:14:12: vmx| DICT         powerType.powerOn = default

Feb 23 09:14:12: vmx| DICT         powerType.suspend = default

Feb 23 09:14:12: vmx| DICT           powerType.reset = default

Feb 23 09:14:12: vmx| DICT           ide0:0.fileName = C:\Program Files\mycompany\cd_images\tools.iso

Feb 23 09:14:12: vmx| DICT         ide0:0.deviceType = cdrom-image

Feb 23 09:14:12: vmx| DICT            ide0:1.present = TRUE

Feb 23 09:14:12: vmx| DICT           ide0:1.fileName = C:\Program Files\mycompany\cd_images\files.ISO

Feb 23 09:14:12: vmx| DICT         ide0:1.deviceType = cdrom-image

Feb 23 09:14:12: vmx| DICT     ide0:1.startConnected = FALSE

Feb 23 09:14:12: vmx| DICT              scsi0:2.redo =

Feb 23 09:14:12: vmx| DICT             uuid.location = 56 4d 37 b2 d4 b2 c9 22-ca da 72 61 11 23 45 57

Feb 23 09:14:12: vmx| DICT                 uuid.bios = 56 4d 37 b2 d4 b2 c9 22-ca da 72 61 11 23 45 57

Feb 23 09:14:12: vmx| DICT               usb.present = FALSE

Feb 23 09:14:12: vmx| DICT     ide0:0.startConnected = FALSE

Feb 23 09:14:12: vmx| DICT        scsi0:0.deviceType = scsi-hardDisk

Feb 23 09:14:12: vmx| DICT        scsi0:1.deviceType = scsi-hardDisk

Feb 23 09:14:12: vmx| DICT        scsi0:2.deviceType = scsi-hardDisk

Feb 23 09:14:12: vmx| DICT        scsi0:3.deviceType = scsi-hardDisk

Feb 23 09:14:12: vmx| DICT  Ethernet1.startConnected = FALSE

Feb 23 09:14:12: vmx| DICT              scsi0:0.redo =

Feb 23 09:14:12: vmx| DICT              scsi0:1.redo =

Feb 23 09:14:12: vmx| DICT              scsi0:3.redo =

Feb 23 09:14:12: vmx| DICT --- USER DEFAULTS

Feb 23 09:14:12: vmx| DICT --- HOST DEFAULTS

Feb 23 09:14:12: vmx| DICT    vmnet1.hostonlyaddress = 10.10.10.1

Feb 23 09:14:12: vmx| DICT     serverd.init.fullpath = /usr/local/vmware/lib/serverd/init.pl

Feb 23 09:14:12: vmx| DICT         authd.client.port = 902

Feb 23 09:14:12: vmx| DICT          control.fullpath = /usr/local/vmware/bin/vmware-cmd

Feb 23 09:14:12: vmx| DICT            authd.fullpath = /usr/local/vmware/sbin/vmware-authd

Feb 23 09:14:12: vmx| DICT             loop.fullpath = /usr/local/vmware/bin/vmware-loop

Feb 23 09:14:12: vmx| DICT                    libdir = /usr/local/vmware/lib

Feb 23 09:14:12: vmx| DICT           vmware.fullpath = /usr/local/vmware/bin/vmware

Feb 23 09:14:12: vmx| DICT    vmnet1.hostonlynetmask = 255.255.255.0

Feb 23 09:14:12: vmx| DICT                     vmdir = /home/mycompany/vms

Feb 23 09:14:12: vmx| DICT            dhcpd.fullpath = /usr/local/vmware/bin/vmnet-dhcpd

Feb 23 09:14:12: vmx| DICT          serverd.fullpath = /usr/local/vmware/sbin/vmware-serverd

Feb 23 09:14:12: vmx| DICT            datastore.name = local

Feb 23 09:14:12: vmx| DICT       datastore.localpath = /home/mycompany/vms/

Feb 23 09:14:12: vmx| DICT --- SITE DEFAULTS

Feb 23 09:14:12: vmx| DICT                  tag.help = introduction.htm

Feb 23 09:14:12: vmx| DICT   tag.configurationEditor = config_editor_newvm.htm

Feb 23 09:14:12: vmx| DICT             tag.ideConfig = devices_virtualdrive.htm

Feb 23 09:14:12: vmx| DICT          tag.floppyConfig = devices_floppy.htm

Feb 23 09:14:12: vmx| DICT           tag.mouseConfig = devices_mouse.htm

Feb 23 09:14:12: vmx| DICT             tag.netConfig = devices_netadapter.htm

Feb 23 09:14:12: vmx| DICT        tag.parallelConfig = devices_parallel.htm

Feb 23 09:14:12: vmx| DICT          tag.serialConfig = devices_serial.htm

Feb 23 09:14:12: vmx| DICT           tag.soundConfig = devices_sound.htm

Feb 23 09:14:12: vmx| DICT             tag.memConfig = configvm_memory.htm

Feb 23 09:14:12: vmx| DICT            tag.miscConfig = configvm.htm

Feb 23 09:14:12: vmx| DICT             tag.usbConfig = devices_usb.htm

Feb 23 09:14:12: vmx| DICT         tag.displayConfig = configvm_display-problems.htm

Feb 23 09:14:12: vmx| DICT                 tag.tools = vmtools.htm

Feb 23 09:14:12: vmx| DICT --- GLOBAL SETTINGS

Feb 23 09:14:12: vmx| WSSCAN: reserved mem (in MB) min=32 max=7904 recommended=7904

Feb 23 09:14:12: vmx|         hostMem=8000 maxAllowedAll=-1 maxAllowedVM=3600

Feb 23 09:14:12: vmx|         totOverhead=16

Feb 23 09:14:12: vmx| WSSCAN: used rec mem (in MB) 7904

Feb 23 09:14:12: vmx| WSSCAN: Overhead 170225 paged 6303 nonpaged 4096 maxFBSize

Feb 23 09:14:12: vmx| WSSCAN 1 1 2023424 -1 2023424 -1 50 0

Feb 23 09:14:12: vmx| LICENSE using: '/etc/vmware/license.vs.1.0-80'

Feb 23 09:14:12: vmx| HOSTINFO: Seeing AMD CPU, numCoresPerCPU 1 numThreadsPerCore 1.

Feb 23 09:14:12: vmx| HOSTINFO: This machine has 1 physical CPUS, 1 total cores, and 1 logical CPUs.

Feb 23 09:14:12: vmx| STATDECLGROUP stats Root "" null

Feb 23 09:14:12: vmx| Host CPUID features: version 0xf51 id1.edx 0x78bfbff id1.ecx 0x0 id81.edx 0xe1d3fbff id81.ecx 0x0

Feb 23 09:14:12: vmx| CPU.cpuFeatures = 0xc83dff00

Feb 23 09:14:12: vmx| CPUID after masking: version 0xf51 id1.edx 0x78bbbff id1.ecx 0x0 id81.edx 0xc1d3fbff id81.ecx 0x0 id88.ecx 0x0

Feb 23 09:14:12: vmx| CPU.cpuFeatures = 0xc83dfe00

Feb 23 09:14:12: vmx| VMMon_AllocLowPages: failed, numPages = 2, errno = 22

Feb 23 09:14:12: vmx| KHZEstimate 1603940

Feb 23 09:14:12: vmx| MHZEstimate 1604

Feb 23 09:14:12: vmx| NumVCPUs 1

Feb 23 09:14:12: vmx| Module Monitor power on failed.

Feb 23 09:14:12: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0

Feb 23 09:14:12: vmx| VMX IPC closed the connection with thread control (0x84c2198)

Feb 23 09:14:12: vmx| VMX: Remote VMControl client control disconnected.

Feb 23 09:14:12: vmx| VMX idle exit

Feb 23 09:14:12: vmx| Flushing VMX VMDB connections

Feb 23 09:14:12: vmx| IPC_exit: disconnecting all threads

Feb 23 09:14:12: vmx| VMX exit.

Feb 23 09:14:12: vmx| AIOMGR-S : stat o=56 r=112 w=0 i=0 br=372736 bw=0