VMware vRealize Orchestrator Not Logging In

This occurred for me when upgrading to or installing a new vCenter 7 and replacing the self-signed certificate. I tested in a lab and was able to successfully install both vCenter 7 and Orchestrator 8.3. I was able to successfully configure both appliances and log in, as well. I did use vSphere Authentication as the Orchestrator’s Identity Provider. As soon as I replaced the self-signed certificate on vCenter, I immediately received the following when logging into Orchestrator:

Uh-oh! So after two weeks or so and lot’s of doing this and that and trying this and that, I think I finally found the resolution. This is actually in the VMware documents, but the document is not quite complete with the information needed to successfully run the commands. Here is the document, https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html. I have detailed the process in full later in this blog post.

For Completeness Sake

For completeness sake, I am going to show the entire process. Please feel free to scroll to the interesting sections below to resolve. I am not going to show how to deploy the appliances, just that they will be in vSphere and available as a starting point.

Install and Check Services

Installed, configured, and checking the services for a “known good”.

VMware vCenter 7.0

When I navigate to my vCenter appliance, I can see that it is using an untrusted certificate.

I perform the necessary steps to continue on. Your browser may be different and your organization’s policies may be different. If your organization is using HTTP Strict Transport Security (HSTS), you will likely be unable to continue without some very tricky manipulation or replacing the self-signed certificate to a known and trusted certificate. This is likely how or why you are in this predicament in the first place and had to search for this blog post.

The log in window is presented to me.

I verified I was able to successfully log in.

VMware vRealize Orchestrator

Navigate to the Orchestrator 8.3 appliance, I am presented with the following.

Since this appliance is fresh, I need to click on the Start the Control Center link and establish an authentication provider. I have to log in with the root account.

Click on Configure Authentication Provider

On this page, I chose vSphere for the Authentication mode setting and the Host address is my vCenter 7 appliance. I am presented with an Accept Certificate box. This will accept the current self-signed certificate, since that is all that is available. NOTE: You could wait to do this step until after you alter the TLS certificate on vCenter, but this article assumes you did not or that you already had an Orchestrator appliance deployed like I did.

Complete the Identity Service window with an administrative or service account that allows users to be queried. Click Register.

Type in a group to use as an Admin group, I used admins, then click the Search button.

A window will display that allows you to pick a security group based off your search criteria. Click Save Changes.

The Orchestrator appliance will be configuring in the background. This is not a fast process! Click on the home icon and choose Validate Configuration. You will see a message stating that a server restart is required…This will automatically happen after a two minute wait. Please be patient here…

You can continue clicking the Refresh button until you have all green check marks. This signifies the appliance rebooted and all services are back up.

Go back to the vco tab in the browser and choose the START THE ORCHESTRATOR CLIENT link. You should be presented the VMware vSphere log on screen. This signifies that your authentication provider is set up correctly to use vSphere. Try logging in.

I can verify that I can successfully log in without trouble.

Let’s Break This!

Replacing the vCenter Server TLS Certificate in vSphere Client

Log in to vCenter server if you are not already. Lot’s of assumptions in the next few sections…I am going to assume you are logged in with an administrative user that also can perform cryptographic operations (https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-17568345-E59E-43A8-A811-92F8BE9C7719.html), then navigate to Menu > Administration > Certificate Management.

I am going to assume you know how to request a Certificate Signing Request (CSR), have already had the certificate signed, and have the necessary certificates in possession. If not, here is a VMware resource to get you started: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-E0609A99-A8D1-4336-BD3B-DE707E261A63.html. Under Machine SSL Certificate, click on Actions > Import and Replace Certificate.

In my case, I chose to use Replace with external CA certificate(requires private key) option and clicked the Next button.

Add in the machine certificate, the root and intermediate certificates chained together (for the chain, I always start with the root and add the intermediate certificates below), and the private key file. Click the Replace button.

If the certificates are successfully replaced, you should get logged out relatively quickly. Give this a few minutes as the vCenter server services are rebooting in the background.

Click the Login button after a few minutes. You may get a no healthy upstream message. Be patient and refresh your browser periodically, there is a lot going on with these appliances.

Eventually, you should get the vCenter log in screen. You can verify that your vCenter is secured by looking for the lock symbol. Go ahead and log back in to verify you can.

Click the Launch vSphere Client (HTML 5) button. Enter your credentials.

Click the Login button.

You may receive a Error occurred while fetching vmca root cert: com.vmware.vcenter.certificate_authority.get_root message. This just indicates that the vCenter server services are not fully restarted, yet. There may be a running task in Recent Tasks. Once this is complete, the message will go away after you refresh the User Interface (UI) or the browser window. Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.

Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.

We can confirm that we have logged in and that the message went away.

VMware vRealize Orchestrator

Ok, let’s try to log into the Orchestrator appliance.

So far, it looks promising. Click on the Start the Orchestrator Client link. Warning: you may actually get logged in. This is most likely due to a cookie on your browser. If you close your browser and try to log in again, you will most likely not be able to log in. That is what we are going to fix.

Enter your credentials and click the Login button.

Et voilà! There we are for us English speakers, the broken UI that is extremely frustrating to fix.

The Fix

Here is the article from VMware on how to solve this (https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html). Unfortunately, not all the details are there to run the commands and if you are not experienced with the underlying technology of the Orchestrator appliance, like I wasn’t and really still am not, then this will just likely frustrate you even further. Let’s break this down…I added an indicator where I added steps to the original documented procedure.

1. Log in to the vRealize Orchestrator command line as root. (Added) I used an SSH session, but you can do this on the console with VMRC. I just wanted to be able to copy and paste commands.

2. (Added) Obtain the name of the <vRO pod> you will need for the next step.

kubectl -n prelude get pods

3. Run the kubectl -n prelude exec command. (Added) I used the last line from the clue in the example command of vco-server-app. I really did not know and the document does not explain.

Command from document.

kubectl -n prelude exec -it <vRO pod> -c vco-server-app -- bash

Command used with the <vRO pod> substituted.

kubectl -n prelude exec -it vco-app-77c8fb6659-fsr5v -c vco-server-app -- bash

4. Run the rpm command.

rpm -hiv --nodeps /vco-cfg-cli.rpm

5. Navigate to the /usr/lib/vco-cli/bin/ directory.

6. Run the following ./vro-configure-inner.sh trust commands.

From the document.

./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept

With substituted <vSphere-Auth-Provider-URI>

./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri vcsa70.aaronrombaut.com --accept

A lot of information will scroll past. I am only including a screenshot of the end of the command.

From the document.

./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept

With substituted <vSphere-Auth-Provider-URI>

./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri vcsa70.aaronrombaut.com --accept

A lot of information will scroll past. I am only including a screenshot of the end of the command, again.

7. Log out of the vRealize Orchestrator Appliance by using the exit command and log in again. (Added) If you only type exit here once, you will only exit the rpm command. You actually have to end the SSH session or console. You can type exit a second time to close the SSH session.

8. Run the following deploy.sh commands.

/opt/scripts/deploy.sh --onlyClean

A lot of information will scroll past. I am only including a screenshot of the end of the command. This command will take a few minutes to complete.

/opt/scripts/deploy.sh

A lot of information will scroll past. This command will take even longer to complete than the last command. Notice: if you prematurely end this command, your appliance will likely not be recoverable. Trust me when I tell you this. Learn from my pain…

You may even see messages that state Exit code and + return 0 like the screenshot below.

This is not complete, yet. Keep waiting until you see the following screen. (If you are nervous or impatient, get up and take a walk, this seriously takes a really long time, the appliance is going through a restart as part of this process).

Confirm VMware vRealize Orchestrator Appliance Configuration

Navigate to the appliance. Click on the Start the Orchestrator Client link to log on.

Type in your credentials and click the Login button.

Assuming everything went well, you should now be able to log back into the VMware vRealize Orchestrator appliance without error.

Please let me know if this helped you or if something I typed did not line up with what you experienced.

Making a REST API Request in vRealize Orchestrator

This was a tough nut to crack for me. I definitely don’t ever want to struggle through learning this topic again. I hope if you are reading this, I can save you a ton of lost time. I use Postman to scope out the concepts before trying to translate them in vRealize Orchestrator (vRO). When you see text surrounded by curly braces like, {{text}}, you can replace with your environment variables. If you are using Postman, then set this as an environment variable. It will save you a lot of typing in the long run.

First, here is the URL for the Swagger UI:
https://{{vRealize Server Fully Qualified Domain Name (FQDN)}}/vco/api/doc/index.html