VMware VCSA 7.0U2 Upgrade Issue: “Encountered an internal error”

Last week, I was working on an customer environment tasked to upgrade their VMware vSphere 6.7U3 environment to the latest version 7.0U2. As always with such upgrades, I started with upgrading the vCenter Server Appliance before upgrading any ESXi host. Unexpectedly, I received an unrecoverable error during Stage 2 of the VCSA upgrade with lots of details starting with “Encountered an internal error”. Read on to learn how I resolved this issue.

Prerequisites

Before moving on with upgrading the vCenter Server Appliance from 6.7U3 to 7.0U2, I ensured that the customer environment met the prerequisites. First thing to check is whether there is a direct upgrade path available from the existing version to the target version. This checked out accompanied with the standard “Back in time Upgrade Restriction” information as can be retrieved from: https://interopmatrix.vmware.com/#/Upgrade?productId=2.

Before actually upgrading, I also ensure that other prerequisites were met such as:

- Checking the Health status of the vSphere environment
- Checking the vCenter Appliance Management Interface (VAMI) for any health, storage or other issues
- Setting DRS to Manual on the vSphere cluster which hosts the VCSA virtual machine
- Creating a VM snapshot (with memory) of the VCSA virtual machine
- Allocating a temporary IP address as part of the VCSA upgrade process (more on this below)

Upgrade Process

The upgrade process that I followed was to first download the vCenter Server Appliance 7.0U2 ISO image from the VMware website: https://my.vmware.com/en/web/vmware/downloads/details?downloadGroup=VC70U2B&productId=974&rPId=68517.

Thereafter, I mounted the ISO image on a management server that had access to the vSphere environment. I followed the “Upgrade” wizard in the UI installer from that ISO image and filled in all required details from the source environment as well as target VCSA 7.0U2 environment. For the latter, I needed to use a temporary IP address. Basically, the standard method for upgrading from 6.7U3 to 7.0U2.

In case you do not know; the wizard goes through two stages in this upgrade process. In the first stage a new VCSA 7.0U2 virtual machine will be deployed using that temporary IP address and the VM name that you have given beforehand. When Stage 1 is complete, you can carry on with Stage 2 that starts with asking you whether you want to export/import configuration data only or configuration data combined with historical log data. When you have confirmed your choice, this second stage starts by exporting configuration (and log) data, importing that data into the new VCSA 7.0U2 virtual machine, shutting down the old 6.7U3 VCSA virtual machine and switching the new VCSA 7.0U2 virtual machine to the existing IP address.

Stage 2 Failed: “Encountered an internal error”

Stage 1 completed successfully without any trouble in approximately 10 minutes.

I kicked off Stage 2 immediately afterwards, which failed at step 2 when “Starting VMware Security Token Service”. When clicking on the “Messages” button, I was presented with the following error details:

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 249, in securityctx_modifier yield File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 360, in register_service svc_create_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [] } During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1861, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 369, in boot self.registerTokenServiceWithLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 656, in registerTokenServiceWithLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 652, in registerTokenServiceWithLookupService dynVars=dynVars) File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 710, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1118, in do_lsauthz_operation ls_obj.register_service(svc_id, svc_create_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__ raise RuntimeError("generator didn't stop after throw()") RuntimeError: generator didn't stop after throw()
Resolution
This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

After this unrecoverable error, I was unable to get the new VCSA 7.0U2 instance to properly work. Rebooting the virtual machine did not help and I was unable to access the VAMI at port 5480 as I was presented with the failed deployment screen immediately. I tried logging on via SSH and manually start services, using command “service-control –start –all” but that also did not work.

To confirm that this was not just a random issue, I deleted the newly VCSA 7.0U2 virtual machine and powered on the old VCSA 6.7U virtual machine. I closed the ISO image UI installer on the management server and waited for the old VCSA to become fully operational. When done, I restarted the entire Upgrade process and waited to see what happened at Stage 2 this time. Unfortunately, it failed again on the same step with the same error message. This was actually a good thing because no IT admin wants to see random errors on Production environments. 😉

Solution

As it was now confirmed that this was not an incidental issue, I started to properly troubleshoot the error. After some digging around on the customer’s environment and VMware Communities and Knowledge Base articles, I stumbled across the following: https://communities.vmware.com/t5/vSphere-Upgrade-Install/Upgrade-VCSA-6-7-to-VCSA-7-0-fails-on-Stage-2-Starting-VMware/td-p/2837281. The error message stated in that forum post was exactly the same as the one I was seeing. Ajay1988 (VMware Employee) responded by stating that the SSO admin account seemed not to be part of the SSO Administrators group. I checked the same on the customer side and BINGO! The same was the case for me as the “administrator@vsphere.local” account was (somehow) not part of the “Administrators” group.

I added the “administrator@vsphere.local” to the “Administrators”, rolled back my second attempt of the VCSA upgrade and restarted the entire process for a third time. Third time lucky… Yes it was! Stage 2 went past that point of “Starting VMware Security Token Service” at 22% and all remaining steps completed successfully.

Thank You

Thank you VMware Communities and especially Lukas_Ro and Ajay1988 for your forum post and provided solution. I hope that this MCL blog post also helps others who are encountering this same error.

Multi-Cloud Life