Failover Clustering

TLS Certificate Considerations

When adding a certificate for Windows Server Failover Clustering on general purpose servers, ensure the fully qualified domain name (FQDN) for the cluster name, essentially the load-balanced name, is in the Subject Name of the certificate using the Common Name (CN) attribute. The FQDN of the cluster name and the FQDN of all the individual servers making up the cluster should also be in the Subject Alternative Name (SAN) field of the certificate using the DNS attribute.

SubjectSubject Alternative Name
Load-balanced FQDNXX
Individual Server FQDNX
General Purpose Server

The exception to this is if you are setting up clustering for Microsoft SQL Server in preparation for an Always On High Availability group. In this case, the FQDN of the individual SQL server is required to be the Subject field, with the name of the cluster and all participating servers in the Subject Alternative Name field.

SubjectSubject Alternative Name
Load-balanced FQDNX
Individual Server FQDN XX
Microsoft SQL Server

Active Directory Considerations

Add your computer objects to an Organizational Unit (OU) in Active Directory. If you pre-stage the failover cluster computer object, ensure the object is disabled. This will let the Failover Cluster wizard know that object is not in use anywhere.

Networking Requirements

It is a good idea to add a Layer 2 network interface to each participating server for cluster heartbeats. It does not need to have a Layer 3 address as the traffic does not need to be routed.

Adding the Failover Cluster Feature

Open the Add Roles and Features Wizard from Server Manager.

Click the Next button.

Click the Next button.

Click the Next button.

Click the Next button. Choose the Failover Clustering checkbox.

A pop-up window should appear prompting to add Remote Server Administration Tools for the requested feature. Click the Add Features button.

You will be returned back to the Add Roles and Features Wizard.

Click the Next button.

Click the Install button. An installation progress bar will appear.

Wait for the feature installation to complete as indicated by the progress bar.

Click the Close button after receiving the message that a restart is pending. Restart the server.

Repeat the installation on each server that is going to participate in the Failover Cluster.

Configuring the Failover Cluster

From Server Manager, select Tools, then Failover Cluster Manager.

In the Actions pane, choose Validate Configuration…

Click the Next button.

Click the Browse… button.

The object type, Computers, should already be selected. Type in the first few characters of the group of servers to lookup. Click the Check Names button.

Choose all the participating servers by either using the shift key or ctrl key. Ensure the cluster computer object is not selected.

Click the OK button.

Click the OK button.

Click the Next > button.

Click the Next > button.

Click the Next > button.

The validation tests will start to run. Once complete, a summary screen should appear.

Assuming the validation checks all pass, click the checkbox to Create the cluster now using the validated node… and then the Finish button.

Click the Next button. Type in the Cluster Name in the text box and provide an IP address in the appropriate octet.

Click the Next button.

Uncheck the checkbox to Add all eligible storage to the cluster. Click the Next button.

A progress bar should display the cluster configuration.

Assuming the cluster configures, click the Finish button.

Expand the cluster object in the Failover Cluster Manager and choose Nodes.

Ensure all the participating servers are present and the status is Up.

Expand the cluster object in the Failover Cluster Manager and choose Networks.

If a second network interface was added and configured prior to the cluster configuration, there should be at least two networks in the list. The default configuration (as seen in the image) is acceptable.

Conclusion

This concludes the installation and configuration of the Windows Server Failover Cluster (WSFC) feature.

VMware vRealize Orchestrator Not Logging In

This occurred for me when upgrading to or installing a new vCenter 7 and replacing the self-signed certificate. I tested in a lab and was able to successfully install both vCenter 7 and Orchestrator 8.3. I was able to successfully configure both appliances and log in, as well. I did use vSphere Authentication as the Orchestrator’s Identity Provider. As soon as I replaced the self-signed certificate on vCenter, I immediately received the following when logging into Orchestrator:

Uh-oh! So after two weeks or so and lot’s of doing this and that and trying this and that, I think I finally found the resolution. This is actually in the VMware documents, but the document is not quite complete with the information needed to successfully run the commands. Here is the document, https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html. I have detailed the process in full later in this blog post.

For Completeness Sake

For completeness sake, I am going to show the entire process. Please feel free to scroll to the interesting sections below to resolve. I am not going to show how to deploy the appliances, just that they will be in vSphere and available as a starting point.

Install and Check Services

Installed, configured, and checking the services for a “known good”.

VMware vCenter 7.0

When I navigate to my vCenter appliance, I can see that it is using an untrusted certificate.

I perform the necessary steps to continue on. Your browser may be different and your organization’s policies may be different. If your organization is using HTTP Strict Transport Security (HSTS), you will likely be unable to continue without some very tricky manipulation or replacing the self-signed certificate to a known and trusted certificate. This is likely how or why you are in this predicament in the first place and had to search for this blog post.

The log in window is presented to me.

I verified I was able to successfully log in.

VMware vRealize Orchestrator

Navigate to the Orchestrator 8.3 appliance, I am presented with the following.

Since this appliance is fresh, I need to click on the Start the Control Center link and establish an authentication provider. I have to log in with the root account.

Click on Configure Authentication Provider

On this page, I chose vSphere for the Authentication mode setting and the Host address is my vCenter 7 appliance. I am presented with an Accept Certificate box. This will accept the current self-signed certificate, since that is all that is available. NOTE: You could wait to do this step until after you alter the TLS certificate on vCenter, but this article assumes you did not or that you already had an Orchestrator appliance deployed like I did.

Complete the Identity Service window with an administrative or service account that allows users to be queried. Click Register.

Type in a group to use as an Admin group, I used admins, then click the Search button.

A window will display that allows you to pick a security group based off your search criteria. Click Save Changes.

The Orchestrator appliance will be configuring in the background. This is not a fast process! Click on the home icon and choose Validate Configuration. You will see a message stating that a server restart is required…This will automatically happen after a two minute wait. Please be patient here…

You can continue clicking the Refresh button until you have all green check marks. This signifies the appliance rebooted and all services are back up.

Go back to the vco tab in the browser and choose the START THE ORCHESTRATOR CLIENT link. You should be presented the VMware vSphere log on screen. This signifies that your authentication provider is set up correctly to use vSphere. Try logging in.

I can verify that I can successfully log in without trouble.

Let’s Break This!

Replacing the vCenter Server TLS Certificate in vSphere Client

Log in to vCenter server if you are not already. Lot’s of assumptions in the next few sections…I am going to assume you are logged in with an administrative user that also can perform cryptographic operations (https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-17568345-E59E-43A8-A811-92F8BE9C7719.html), then navigate to Menu > Administration > Certificate Management.

I am going to assume you know how to request a Certificate Signing Request (CSR), have already had the certificate signed, and have the necessary certificates in possession. If not, here is a VMware resource to get you started: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-E0609A99-A8D1-4336-BD3B-DE707E261A63.html. Under Machine SSL Certificate, click on Actions > Import and Replace Certificate.

In my case, I chose to use Replace with external CA certificate(requires private key) option and clicked the Next button.

Add in the machine certificate, the root and intermediate certificates chained together (for the chain, I always start with the root and add the intermediate certificates below), and the private key file. Click the Replace button.

If the certificates are successfully replaced, you should get logged out relatively quickly. Give this a few minutes as the vCenter server services are rebooting in the background.

Click the Login button after a few minutes. You may get a no healthy upstream message. Be patient and refresh your browser periodically, there is a lot going on with these appliances.

Eventually, you should get the vCenter log in screen. You can verify that your vCenter is secured by looking for the lock symbol. Go ahead and log back in to verify you can.

Click the Launch vSphere Client (HTML 5) button. Enter your credentials.

Click the Login button.

You may receive a Error occurred while fetching vmca root cert: com.vmware.vcenter.certificate_authority.get_root message. This just indicates that the vCenter server services are not fully restarted, yet. There may be a running task in Recent Tasks. Once this is complete, the message will go away after you refresh the User Interface (UI) or the browser window. Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.

Again, be patient as this may seem like an eternity in computer time or that it is broken, but it should come back up.

We can confirm that we have logged in and that the message went away.

VMware vRealize Orchestrator

Ok, let’s try to log into the Orchestrator appliance.

So far, it looks promising. Click on the Start the Orchestrator Client link. Warning: you may actually get logged in. This is most likely due to a cookie on your browser. If you close your browser and try to log in again, you will most likely not be able to log in. That is what we are going to fix.

Enter your credentials and click the Login button.

Et voilà! There we are for us English speakers, the broken UI that is extremely frustrating to fix.

The Fix

Here is the article from VMware on how to solve this (https://docs.vmware.com/en/vRealize-Orchestrator/8.3/com.vmware.vrealize.orchestrator-install-config.doc/GUID-66B37DF2-052E-44A0-929E-E4F53E1BCCE3.html). Unfortunately, not all the details are there to run the commands and if you are not experienced with the underlying technology of the Orchestrator appliance, like I wasn’t and really still am not, then this will just likely frustrate you even further. Let’s break this down…I added an indicator where I added steps to the original documented procedure.

1. Log in to the vRealize Orchestrator command line as root. (Added) I used an SSH session, but you can do this on the console with VMRC. I just wanted to be able to copy and paste commands.

2. (Added) Obtain the name of the <vRO pod> you will need for the next step.

kubectl -n prelude get pods

3. Run the kubectl -n prelude exec command. (Added) I used the last line from the clue in the example command of vco-server-app. I really did not know and the document does not explain.

Command from document.

kubectl -n prelude exec -it <vRO pod> -c vco-server-app -- bash

Command used with the <vRO pod> substituted.

kubectl -n prelude exec -it vco-app-77c8fb6659-fsr5v -c vco-server-app -- bash

4. Run the rpm command.

rpm -hiv --nodeps /vco-cfg-cli.rpm

5. Navigate to the /usr/lib/vco-cli/bin/ directory.

6. Run the following ./vro-configure-inner.sh trust commands.

From the document.

./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept

With substituted <vSphere-Auth-Provider-URI>

./vro-configure-inner.sh trust --alias vco.vsphere.lookup-service.ssl.certificate --uri vcsa70.aaronrombaut.com --accept

A lot of information will scroll past. I am only including a screenshot of the end of the command.

From the document.

./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri <vSphere-Auth-Provider-URI> --accept

With substituted <vSphere-Auth-Provider-URI>

./vro-configure-inner.sh trust --alias vco.sso.ssl.certificate --uri vcsa70.aaronrombaut.com --accept

A lot of information will scroll past. I am only including a screenshot of the end of the command, again.

7. Log out of the vRealize Orchestrator Appliance by using the exit command and log in again. (Added) If you only type exit here once, you will only exit the rpm command. You actually have to end the SSH session or console. You can type exit a second time to close the SSH session.

8. Run the following deploy.sh commands.

/opt/scripts/deploy.sh --onlyClean

A lot of information will scroll past. I am only including a screenshot of the end of the command. This command will take a few minutes to complete.

/opt/scripts/deploy.sh

A lot of information will scroll past. This command will take even longer to complete than the last command. Notice: if you prematurely end this command, your appliance will likely not be recoverable. Trust me when I tell you this. Learn from my pain…

You may even see messages that state Exit code and + return 0 like the screenshot below.

This is not complete, yet. Keep waiting until you see the following screen. (If you are nervous or impatient, get up and take a walk, this seriously takes a really long time, the appliance is going through a restart as part of this process).

Confirm VMware vRealize Orchestrator Appliance Configuration

Navigate to the appliance. Click on the Start the Orchestrator Client link to log on.

Type in your credentials and click the Login button.

Assuming everything went well, you should now be able to log back into the VMware vRealize Orchestrator appliance without error.

Please let me know if this helped you or if something I typed did not line up with what you experienced.

Configuring VMware Horizon Connection Server – Windows Server

ADSI Edit

If you do not change the expiration period, Connection Server will stop accepting SAML assertions from the SAML authenticator, such as a Unified Access Gateway appliance or a third-party identity provider, after 24 hours, and the metadata exchange must be repeated.

1. Start the ADSI Edit utility on your Connection Server host.

2. In the console tree, right-click ADSI Edit and select Connect to.

Connection Settings

3. In the Select or type a Distinguished Name or Naming Context text box, type the distinguished name DC=vdi,DC=vmware,DC=int.

4. In the Computer pane, select or type localhost:389 or the fully qualified domain name (FQDN) of the Connection Server host followed by port 389.

For example: localhost:389 or mycomputer.example.com:389

Connection Settings – Filled Out

5. Expand the ADSI Edit tree, expand OU=Properties, select OU=Global, and double-click CN=Common in the right pane.

6. In the Properties dialog box, edit the pae-NameValuePair attribute to add the following values

cs-samlencryptionkeyvaliditydays=number-of-days 
cs-samlsigningkeyvaliditydays=number-of-days

In this example, number-of-days is the number of days that can elapse before a remote Connection Server stops accepting SAML assertions. After this period of time, the process of exchanging SAML metadata must be repeated.

Multi-valued String Editor

Click the OK button on the Multi-valued String Editor window to return to the CN=Common Properties window.

Click on the OK button on the CN=Common Properties window to return to the ADSI Edit window.

Mark your calendar for the time frame entered above. The day after that expiration date will mean that users can no longer log in. The SAML assertion metadata will need to be regenerated and exchanged (probably to a Unified Access Gateway) in order for the trust relationship to be re-established.

locked.properties

If any of the following are added to the locked.properties file, save the file and restart the Connection Server Service. For more information, please reference the Horizon Installation guide located at https://docs.vmware.com/en/VMware-Horizon/2106/horizon-installation.pdf. The two most common properties used in a deployment are the balancedHost and portalHost. More information on these two properties follows the table below.

balancedHostAllow HTML Access Through a Load Balancer
portalHostAllow HTML Access Through a Gateway
serverPortReplace the Default HTTP Ports or NICs for Horizon Connection Server Instances
serverPortNonSslReplace the Default HTTP Ports or NICs for Horizon Connection Server Instances
serverProtocolProperty to off-load SSL for client connections
psgControlPortReplace the Default Control Port for PCoIP Secure Gateway on Connection Server Instances
frontMappingHttpDisabledChange the Port Number for HTTP Redirection to Connection Server
frontMappingHttpDisabledPrevent HTTP Redirection for Client Connections to Connection Server
Common Properties in locked.properties file

Allow HTML Access Through a Load Balancer

Connection Server instances that are directly behind a load balancer or load-balanced gateway must know the address by which browsers will connect to the load balancer when users use HTML Access.

You must perform this procedure for each Connection Server that is behind the load balancer or load-balanced gateway.

1. Create or edit the locked.properties file in the gateway configuration folder on the Connection Server host.
For example: install_directory\VMware\VMware View\Server\sslgateway\conf\locked.properties

2. Add the balancedHost property and set it to the address of the load balancer. For example, if users type https://view.example.com in a browser to reach any of the load-balanced Connection Servers, add balancedHost=view.example.com to the locked.properties file.

3. Save the locked.properties file.

4. Restart the Connection Server service to make your changes take effect.

Allow HTML Access Through a Gateway

Connection Server instances that are directly behind a gateway, such as Unified Access Gateway, must know the address by which browsers will connect to the gateway when users use HTML Access.

1. Create or edit the locked.properties file in the gateway configuration folder on the Connection Server host. For example: install_directory\VMware\VMware View\Server\sslgateway\conf\locked.properties

2. Add the portalHost property and set it to the address of the gateway.

For example, if https://view-gateway.example.com is the address that browsers use to access VMware Horizon through the gateway, add portalHost=view-gateway.example.com to the locked.properties file. If the Connection Server instance is behind multiple gateways, you can specify each gateway by adding a number to the portalHost property, for example:

portalHost.1=view-gateway-1.example.com
portalHost.2=view-gateway-2.example.com

You must also specify multiple portalHost properties if a single gateway machine is known by more than one name.

3. Save the locked.properties file.

4. Restart the Connection Server service to make your changes take effect.

Windows Server Shutdown Event Tracker

Every time I log in to my Windows Servers, I am greeted with the Shutdown Event Tracker. I know I cleanly shutdown, restarted, or only logged off, though. This has been an issue in my lab as well as on customer networks. I wanted to see if there was an easy and reliable fix, which turns out, there is!

Open Registry Editor (regedit.msc).

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Reliability

There are likely two keys here, DirtyShutdown and DirtyShutdownTime that are present. Simply delete these two keys and you should no longer get the Shutdown Event Tracker on subsequent logons.


Configuring VMware Horizon Connection Server – Horizon Console (HTML5)

Open a web browser and navigate to the Horizon Console with the fully qualified domain name of the load-balanced address or individual connection server and append /admin to the end. Don’t navigate using the short name, as it will likely result in logon issues when trying to authenticate.

https://hcs000v.aaronrombaut.com/admin

For the very first log on, use the domain account that was added to the computer’s Local Administrators group on the server where Horizon Connection Server is installed. Once in the UI, additional administrators can be configured.

Also on the very first log on, you will be presented with a license screen. Go ahead and enter the license key and click the OK button.

Settings in Horizon Console 2013

Servers

From this menu, vCenter Servers, Gateways, and Connection Servers can be configured.

Domains

From this menu, Instant Clone Engine Domain Accounts can be configured, Connection Server domains can be viewed, and untrusted domains can be added in the Domain menu.

Product Licensing and Usage

From this menu, Licensing can be modified, license Usage can be viewed, or the Customer Experience Program settings can be managed.

Global Settings

From this menu, General Settings, Security Settings, and Client Restriction Settings can be edited.

Registered Machines

From this menu, RDS Hosts and Other machines can be viewed.

Administrators

From this menu, users, administrators, groups, and roles can be configured in the Administrators and Groups, Role Privileges, Role Permissions, and Access Groups sub menus.

Cloud Pod Architecture

This menu item is for linking multiple pods to create a single pod federation.

Event Configuration

From this menu, the Event Database can be configured as well as the Syslog and Events to File System settings.

Global Policy

[Placeholder]

Troubleshooting

Certificates; Event Database Configuration; License

Installing VMware Horizon Connection Server

Certificate Requirements

When you install VMware Horizon Connection Server, a self-signed certificate will be installed into the Personal certificate store.

You should use a Certificate Authority (CA) signed certificate for the computer account and store it in the Personal certificate store before installing VMware Horizon Connection server. During the installation, if a machine certificate is already installed, it will be used. If you are going to use a load balancer, ensure the load balanced fully qualified domain name is in the Common Name (CN) attribute of the certificate and each Horizon Connection server’s fully qualified domain name is added to the Subject Alternative Name (SAN) attribute of the certificate.

Open the Certificates – Local Machine (certlm.msc) console and install the machine certificate. Make sure that you check the box to Mark this key as exportable. If you forget this step, just import the certificate again. The private key is required to bring up the VMware Horizon View Blast Secure Gateway service. Blast Secure Gateway logs (C:\ProgramData\VMware\VDM\logs\Blast Secure Gateway) will report that the private key is not accessible if this step is forgotten.

Once installed, right-click the certificate, choose properties, and change the Friendly name to vdm, making sure vdm is lowercase.

Click the OK button.

Service Accounts

A service account should be added to the local administrator security group as a fail-safe way to log in.

Open the Computer Management console (compmgmt.msc). Expand System Tools > expand Local Users and Groups > click Groups folders.

Double-click the Administrators folder and add the Active Directory Service Account. A user account on the local computer cannot be used.

Installing VMware Horizon Connection Server Software

Using an administrative account, double-click the VMware Horizon Connection Server installer.

Answer the optional User Account Control window.

The Installation Wizard opens.

Click the Next button.

Review the License Agreement.

Choose the I accept the terms in the license agreement radio button.

Click the Next button.

Leave the default installation location or change if necessary.

Click the Next button.

For the first Connection Server, choose Horizon 7 Standard Server. For subsequent, or replica servers, choose Horizon 7 Replica Server. This same dialog works for Horizon 8, the UI will just say Horizon Standard Server instead of Horizon 7 Standard Server and Horizon Replica Server instead of Horizon 7 Replica Server.

You can leave the Install HTML Access checkbox selected, but know that if your users use Smart Cards, they will not be able to use HTML. The VMware Horizon Client must be used.

Choose IPv4 for the IP Protocol.

If you are in the Federal Government, your systems most likely are Federal Information Processing Standards (FIPS) compliant. You will likely want to choose Enabled for FIPS compliant cryptography. If FIPS is not configured in your operating system, this option will not display for you.

Click the Next button.

Enter a data recovery password and re-enter it.

Optionally, enter a password reminder.

Click the Next button.

Leave the Configure Windows Firewall automatically radio button selected.

Click the Next button.

Choose the Authorize the local Administrators group radio button. An administrative security group can be configured later after the installation.

Click the Next button.

Choose to Join the VMware Customer Experience Improvement Program or not.

Click the Next button.

Review the installation location.

Click the Install button.

Choose to Show the documentation or not.

Click the Finish button.

Repeat the installation on all subsequent connection servers. Be sure to choose Horizon 7 Replica Server (for Horizon 7) or Horizon Replica Server (for Horizon 8) depending on the version of VMware Horizon Connection server you are installing.

Continue to Configuring VMware Horizon Connection Server – Windows Server.

References

Install Horizon Connection Server with a New Configuration: https://docs.vmware.com/en/VMware-Horizon-7/7.13/horizon-installation/GUID-9F93A59F-C35F-4388-B3D6-CE4F50D8BAFD.html


Adding Active Directory Computer Accounts to Microsoft SQL Server

Launch Microsoft SQL Server Management Studio (SSMS) with an administrative account.

Type the fully qualified domain name in the Server name: textbox.

Change the Authentication method from the drop-down box.

Click the Connect button.

Right-click the Security folder > choose New > click Login…

​Provide the computer name in the Login name textbox. A computer name consists of a domain name, backslash, followed by a host name, and ending with a dollar sign ($). See below for reference.

<DOMAIN-NAME>\<HOSTNAME>$
AARONROMBAUT\APP-001v$

Choose a Default Database from the drop-down if one has been created.

Choose User Mapping from Select a page.

Click the checkbox for the database this computer account should be associated with in the Users mapped to this login: pane.

Click the checkbox for the db_owner in the Database role membership for: <Database Name> pane.

Click the OK button.