I've inherited an environment with two ESXi hosts and a vCenter Appliance Server that is running on one of those hosts. It does not have a history of issues but after a recent unexpected loss of power to the SAN the vCenter Appliance, which resides on the SAN, is now malfunctioning. Our VM's are running fine and can be managed on the hosts directly but I need to get the vCenter appliance functional again.
Following KB steps to perform an FSCK resulted in issues being found and fixed but the underlying problem remains. I found that /var/logs/vmware didn't exist, instead there seemed to be a file named vmware in /var/logs.
The only vCenter backup is a snapshot of the appliance from over a year ago. Changes to the environment since then have been the removal of some VM's, no new VM's were added, some VM's virtual hard drives were expanded, and one VM had an additional VHD added. There have been no infrastructure changes (SAN, domain, etc.) in that time. I was able to perform a current backup of the vCenter Appliance database as detailed in KB2091961 using the backup_lin.py script. I then made a current snapshot and reverted to the 1+ Year old snapshot. The vCenter server seemed to come up functional, ESXi hosts were visible, but of course it showed VM's that had since been removed, etc. I proceeded to restore the backup that I had just made and it broke the vCenter Server with the same issue of VXPD will not start. The /var/logs/vmware folder is however now accessible and I have followed the suggested steps of ensuring there is adequate storage and making sure the time is correct, etc.
Attempting to access vCenter via web browser results in '503 Service Unavailable' however I can still log in to management at port 5480 and can SSH into it. The best lead I currently have is the following from a log file which makes me feel like something is unhappy up in the database regarding the datastore details.
info vpxd[7F5ABDDB37A0] [Originator@6876 sub=Vsan] Done creating VSAN dynamic MOs for -15.
error vpxd[7F5ABDDB37A0] [Originator@6876 sub=Datastore] [VpxdDatastore] Failed to load DATASTORE table from database: vmodl.fault.ManagedObjectNotFound
warning vpxd[7F5ABDDB37A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 3719 ms
error vpxd[7F5ABDDB37A0] [Originator@6876 sub=Main] [VpxdMain] Failed to initialize 5Vmomi5Fault21ManagedObjectNotFound9ExceptionE(vmodl.fault.ManagedObjectNotFound)
error vpxd[7F5ABDDB37A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...
info vpxd[7F5ABDDB37A0] [Originator@6876 sub=Default] Forcing shutdown of VMware VirtualCenter now
After exhaustive research I've about given up on fixing the problem and would like to look at rebuilding. My questions are...
- If I restore an old snapshot is there a way to get vCenter to refresh itself in regards to the VM tweaks (VHD size increases/additions) that have been made since then? If yes then it seems my problem is solved and I would just need to delete the VM's that were removed awhile back.
- If no on question 1 then what are some suggested next steps to get a working vCenter appliance backup up and running? I have read about resetting the inventory service but I'm unsure exactly what the effects of that are.
- What is the behavior when an old vCenter snapshot is restored, how does it affect the VM's whose settings have been modified since?
Thank you.
