Installing VMware App Volumes 4 Manager – Part 3

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-3E512951-4045-4C15-A642-69BBCF4C34B6.html

This section should really be called Configuring VMware App Volumes 4 Manager, but since it is part of a multi-part series, I kept it for uniformity. If you followed along in part 2 (Installing VMware App Volumes 4 Manager – Part 2), then most of the hard work will already be accomplished and this will just be a matter of configuring each page.

Start out by navigating to your App Volumes 4 Manager in the browser. The URL is the fully qualified domain name. In my case, I will go to https://app-001v.aaronrombaut.com.

If you receive an error like this, follow the Replace the Self-Signed Certificate with CA-signed Certificate article.

Click the Get Started button.

If you have a license to upload, edit that here. Since I do not have a license, I am skipping this. Click the Next button.

If you fill in the details on the AD Domains page and receive the Request Error, follow the Connecting Securely to Active Directory article.

Fill in the details and click the Register button. You should not get a Request Error like above.

Verify that the details are correct and click the Next button.

Type the group you want to use for administering App Volumes 4 applications in the Search Groups: box and click the Search button. Once you find the group, click the Assign button.

Click the Next button.

Fill in the Hostname:, Username:, and Password: fields and click the Save button.

If you receive the Request Error, like seen above, follow the vCenter Certificate becomes unverified article.

Review the page and when ready, click on the Next button.

Choose the datastores for Packages and Writable Volumes using the appropriate Default Storage Location: selectors. Click the Next button.

Choose the Import volumes immediately radio button and click the Set Defaults button.

Check the top checkbox to select all the templates to upload. Click the Upload button.

Review the settings and click the Upload button.

On the Settings page, I choose to Allow Non-Domain Entities: under Active Directory. This is my personal choice in my lab and may not be necessary for your environment.

Review the settings on the page and click the Save button.

This concludes the installation of App Volumes 4 Manager. At this point, you are ready to start creating Applications and provisioning them to user groups. Your infrastructure should be secure as well if you have followed all of the steps provided. Feel free to make comments or provide feedback if something doesn’t work for you or I “glazed” over a detail somewhere. This is software, so it’s likely to change rapidly.

Installing VMware App Volumes 4 Manager – Part 1

Installing VMware App Volumes 4 Manager – Part 2

Installing VMware App Volumes 4 Manager – Part 2

This is a prerequisite phase before configuring App Volumes Manager in the browser. A lot of this should already be accomplished in a production environment except the newly installed App Volumes Manager specific settings. This page can serve more than just for configuring security for App Volumes Manager.

At this point, the App Volumes Manager software should be installed. Reference the following article to install the App Volumes Manager: Installing VMware App Volumes 4 Manager – Part 1.

Rather than reiterate what’s already been written elsewhere that makes sense, I am just going to drop the appropriate links below and make notes where appropriate for reference.

Using SSL Certificates with App Volumes Manager

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-266871B4-ACEA-455C-8388-20BDD7B239D2.html


Connecting Securely to Active Directory

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-FAEDA013-2768-4854-B813-B39BB3F7E598.html

I added the certificate in the following order: Root+Intermediate+Machine and named it adCA.pem. This file gets added to the C:\Program Files (x86)\CloudVolumes\Manager\config directory and then the App Volumes Manager service gets restarted.


Replace the Self-Signed Certificate with CA-signed Certificate

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-EFCC36A2-1609-4B47-969E-2A0CF9BC9B68.html

Add the CA signed certificate and key to the following directory: C:\Program Files (x86)\CloudVolumes\Manager\nginx\conf

Directory to nginx.conf configuration file: C:\Program Files (x86)\CloudVolumes\Manager\nginx\conf — Make a copy before editing!

Edit lines 57 and 58 to reflect the certificate and key added earlier.

Restart the App Volumes Manager service.


Add a Trusted Root Certificate to the Certificate Store

ref: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-B635BDD9-4F8A-4FD8-A4FE-7526272FC87D.html

Add Custom Certificates

ref: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-15A4FD99-37E2-41E5-8A35-247B8FDB019D.html


Convert Certificate Files to One-Line PEM Format

ref: https://docs.vmware.com/en/Unified-Access-Gateway/2009/uag-deploy-config/GUID-870AF51F-AB37-4D6C-B9F5-4BFEB18F11E9.html

Use the following commands to convert .p12 or .pfx files to PEM.

openssl pkcs12 -in mycaservercert.pfx -nokeys -out mycaservercert.pem
openssl pkcs12 -in mycaservercert.pfx -nodes -nocerts -out mycaservercertkey.pem
openssl rsa -in mycaservercertkey.pem -check -out mycaservercertkeyrsa.pem

Create a Custom vCenter Server Role [for App Volumes Manager]

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-8C79E140-A8DA-4A1F-B09D-DE9A332114E2.html?hWord=N4IghgNiBcIE4HsIFMQF8g

Create a Custom vCenter Server Role Using PowerCLI [for App Volumes Manager]

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.admin.doc/GUID-505624F3-F3EB-428C-BEA0-5BD7F6095A1F.html#GUID-505624F3-F3EB-428C-BEA0-5BD7F6095A1F


App Volumes 2.12.1 – vCenter Certificate becomes unverified. (2150281)

ref: https://kb.vmware.com/s/article/2150281

Use this article to have App Volumes Manager trust the vCenter Server certificate. I could not find this information anywhere else besides this kb.

To resolve the issue, add the vCenter CA certificate to the cacert.pem

Make sure that the cacert.pem file has the complete chain including the vCenter server certificate.

  • Generated the cacert.pem and placed it under the directory “C:/Program Files (x86)/CloudVolumes/Manager/config/cacert.pem”
  • Imported the vCenter CA cert and vCenter server cert to trusted store of the App Volume Manager.
  • Restarted the App Volume Manager service.

Update: I added the certificates in the following order: Root+Intermediate+Machine to cacert.pem and copied the file into the C:/Program Files (x86)/CloudVolumes/Manager/config/ directory. I then restarted the App Volumes Manager service. When configuring the Machine Managers, the certificate error is not present.

Adding a Computer Account to MS SQL Server for a VMware App Volumes Manager Database

ref: https://www.enhansoft.com/updated-how-to-create-a-sql-server-computer-account-login/

This post will probably work for other use cases, but I am specifically needing it for VMware App Volumes Manager.

Open SQL Server Management Studio (SSMS)

Expand Security

Right-click Logins

Select New Login…

1. Do not use the Search… button! Type the Login name: as

DOMAIN\computer-name$

The dollar sign is necessary to signify the account as a computer and not the name of a user. (ref: https://social.technet.microsoft.com/Forums/en-US/eec574c0-5421-4d7a-a806-a3c5af3d29bf/why-in-samaccount-name-of-computer-account-in-active-directory?forum=winserverDS)

2. Choose the Windows authentication radio button.

3. Select the Default database for App Volumes Manager if it was already created. You can assign it later after creating the database if needed.

4. Select the Default language

Do not click OK!

On the Server Roles page, choose the sysadmin checkbox to grant the role to the user. Don’t click OK, yet.

On the User Mapping page, Choose the checkbox next to the database being mapped to the user (computer) account (assuming the database has already been created).

Click OK.

Verify the computer account is added to the list of logins.

This concludes this post.

Installing VMware App Volumes 4 Manager – Part 1

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.install.doc/GUID-2E6F56D8-E657-4290-BAE7-E18E7556ADDC.html (VMware App Volumes Installation Guide)

ref: https://docs.vmware.com/en/VMware-App-Volumes/4/com.vmware.appvolumes.install.doc/GUID-25B53F4E-C22B-4DBD-A253-D7FA33D965BF.html (Installing App Volumes)

I will start out this post to mention that if you try to install App Volumes 4 and are stuck on the SQL database portion trying to get Windows Integrated Authentication (WIA) working, you are not alone. I have spent countless hours troubleshooting this and looking for documentation. Documentation seems to be almost non-existent regarding the database requirements, besides which version to use, but nothing specific on database settings or user/computer/other authentication settings. It is very frustrating.

I will also mention that I have installed MS SQL Server 2016 and have set up TLS certificates on the Windows guest as well as configured SQL Server to use the certificate. I have also added the App Volumes server computer account to the database permissions. Please see the following articles, Configuring Microsoft SQL Server for VMware Horizon View, MSSQL SSL/TLS Certificate Chain Fix, and Adding a Computer Account to MS SQL Server for a VMware App Volumes Manager Database, for more information. If you read and follow the three articles provided, you will not encounter an error when configuring the App Volumes Manager database authentication!

I am going to assume that you have some level of technical proficiency if you are this far along in your journey and will not add mundane details like how to mount an ISO or what buttons to click unless following a particular workflow and feel the details are absolutely necessary. Let’s get started…

After downloading App Volumes 4 (https://my.vmware.com/web/vmware/downloads/info/slug/desktop_end_user_computing/vmware_app_volumes/4_x), mount the ISO to your App Volumes server. Navigate to the Installation directory.

Double-click setup.exe.

Click Next.

Check the I accept the terms in the License Agreement checkbox (you did read all that, right?) and Click Next.

Choose the Install App Volumes Manager radio button since we are installing the App Volumes Manager. Click Next.

Not Shown! – You may be presented with a Windows User Account Control (UAC) prompt. Move past this accordingly. The installation up to this point has been a bootstrap of sorts. The next screens will do the actual Manager installation.

Déjà vu? Groundhog Day? Now we proceed with the actual installation of App Volumes Manager. Click Next.

Choose the Connect to an existing SQL Server Database radio button. Click Next.

There is a lot going on at this step. If you read the beginning of this post and followed the two additional articles provided, you should not have any trouble using Windows Integrated Authentication here. To quickly recap:

  • Your MS SQL server and App Volumes Manager servers can resolve DNS forward and reverse lookup records
  • Your MS SQL server has a TLS certificate in the local machine certificate store
  • The user running your MS SQL service has been provided access to the MS SQL server’s private key
  • The MS SQL server has been configured to use the machine’s TLS certificate
  • The MS SQL service or server has been restarted.
  • The App Volumes database has been created.
  • The App Volumes server account (the computer account) has a log in on the MS SQL server and is associated with the App Volumes database. Adding a Computer Account to MS SQL Server for a VMware App Volumes Manager Database
  1. Enter the FQDN for the MS SQL server.
  2. Choose the Windows Integrated Authentication radio button.
  3. Click the Browse… button. If your authentication works properly, this is a good test as it will either show you the databases or it will present you an error.

Choose the database specified for use with App Volumes and click Next.

4. Check the Enable SQL Server certificate validation checkbox.

Click Next.

Almost there, if you are to this point, the hardest part of the installation is over! Leave the default ports unless you like to complicate your life. Click Next.

Optional: Change the installation location if necessary.

Click Next.

Click Install. Take a break! The installer will need a few minutes to install the software.

Confirm that the Wizard completed successfully. See Troubleshooting section below if you did not have a successful installation.

Click Finish.

You have completed the install, but there are some prerequisite steps to take to make the configuration easier and work the first time. Check out the article here: Installing VMware App Volumes 4 Manager – Part 2.

Troubleshooting

Be sure to read the finished dialog! You may receive the following, reporting that the Wizard ended prematurely. You will have to open up the logs and investigate; navigate to your installation directory and look at the logs. The issue that generated the following screenshot for me was either a self-signed certificate issue or a SQL Server issue because I didn’t add the App Volumes Manager server login to the MS SQL server and assign it to the App Volumes database (see below for specific error message and how to fix). Delete the Cloud Volumes directory and try the installation again.

From the inst_ruby_script.log file, you see the following:

E, [2020-11-19T11:43:30.666302 #3196] ERROR -- : ERROR: 28000 (18456) [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'AARONROMBAUT\APP-001V$'.
E, [2020-11-19T11:43:30.666571 #3196] ERROR -- : ["script/odbc_driver_validator.rb:145:in drvconnect'", "script/odbc_driver_validator.rb:145:inconnect_using_conn_string'", "script/odbc_driver_validator.rb:34:in odbc_connection'", "script/odbc_driver_validator.rb:76:invalidate_conn_and_version'", "script/odbc_driver_validator.rb:183:in <encoded>'", "script/odbc_driver_validator.rb:2:inRGLoader_load'", "script/odbc_driver_validator.rb:2:in `
'"]
Unable to connect to SQLServer. Exiting with status -1.

To resolve this, follow this article: Adding a Computer Account to MS SQL Server for a VMware App Volumes Manager Database.

PowerCLI Script to add vCenter Privileges for VMware Horizon 7

ref: https://docs.vmware.com/en/VMware-Horizon-7/7.12/horizon-installation/GUID-A878F876-B359-42FC-9124-A1E34BFB3319.html

ref: https://code.vmware.com/docs/11794/cmdlet-reference/doc/Get-VIPrivilege.html

ref: https://code.vmware.com/docs/11794/cmdlet-reference/doc/New-VIRole.html

ref: https://code.vmware.com/docs/11794/cmdlet-reference/doc/Set-VIRole.html

$VIRoleName = "View Manager Role"
$VIRolePrivileges = @(`
    # Folder  
    'Create Folder', 'Delete Folder',`
    # Datastore
    'Allocate space',`
    # Virtual Machine - Configuration
    'Add or remove device', 'Advanced configuration', 'Modify device settings',`
    # Virtual Machine - Interaction
    'Power off', 'Power on', 'Reset', 'Suspend', 'Perform wipe or shrink operations',`
    # Virtual Machine - Inventory
    'Create new', 'Create from existing', 'Remove',`
    # Virtual Machine - Provisioning
    'Customize guest', 'Deploy template', 'Read customization specifications', 'Clone template', 'Clone Virtual Machine',`
    # Resource
    'Assign virtual machine to resource pool',`
    # Global
    'Act as vCenter Server',`
    # Host
    'Advanced settings',`
    # Profile-driven Storage
    'Profile-driven storage view', 'Profile-driven storage update'
    )

try {
    # Get list of current Roles
    $VIRoles = Get-VIRole

    # Check if Role exists
    foreach($VIRole in $VIRoles) {
        if ($VIRole.Name -like $VIRoleName) {
            # Role exists
            exit
        } 
    }

    # Assume the Role does not exist
    # Create the new Role
    New-VIRole -Name $VIRoleName
    
    # Add the Privileges to the Role
    foreach($VIRolePrivilege in $VIRolePrivileges) { 
        Set-VIRole -Role $VIRoleName -AddPrivilege $VIRolePrivilege 
    }

} catch {
    
}

Copy and paste the contents above to a new PowerShell file. This script will check if the given Role exists and exit or it will create the Role and add the Privileges. This script will not check the current assigned Privileges if the Role exists.

Configuring Microsoft SQL Server for VMware Horizon View

Software being used:

  • Microsoft SQL Server 2016
  • Microsoft SQL Server Management Studio 18.4
  • VMware Horizon 7.12.0

Configure Microsoft SQL Server

After installing Microsoft SQL Server, a few things need to be configured:

1. Ensure TCP port 1433 is open on any firewall software running on the server.

2. Add a machine certificate to local machine personal store (certlm.msc)

2a. Be sure to add the account that runs the SQL service to the private key. See this post for more details. MSSQL SSL/TLS Certificate Chain Fix

3. Add certificate to Microsoft SQL server. (ref: https://support.microsoft.com/en-us/help/316898/how-to-enable-ssl-encryption-for-an-instance-of-sql-server-by-using-mi)

4. Open Sql Server Configuration Manager. Right-click Protocols for <INSTANCE-NAME>, choose Properties.

5. Click on the Certificate tab.

6. Choose the machine certificate you added earlier, click OK.

7. Back on the Sql Server Configuration Manager window, double-click TCP/IP.

8. Ensure the following settings:

9. Check the Active, Enabled, IP Address, and TCP Port settings. Modify as necessary and click OK. If you do make any modifications, be sure to restart the SQL service.

Creating the VMware Horizon Database and User

  1. Open Microsoft SQL Server Management Studio.
  2. Ensure the SQL server authentication is set to SQL Server and Windows Authentication mode. VMware Horizon does not use Integrated Windows Authentication. (https://docs.vmware.com/en/VMware-Horizon-7/7.12/horizon-installation/GUID-1360BFDF-9F90-47FD-8B6C-E842CF951A53.html)

3. Right-click Databases and choose New Database…

4. Fill in the Database name with a meaningful name. Click OK.

5. Expand Security and Right-click Logins.

6. On the General page, fill in the Login name with a meaningful name. Also choose a password and uncheck Enforce password policy unless you use a strong password. Change the Default database and Default language accordingly.

7. On the Server Roles page, check the sysadmin role.

8. On the User Mapping page, check the database name you created earlier. Click OK.

Configure the Event Database on Horizon View Connection Server

ref: https://docs.vmware.com/en/VMware-Horizon-7/7.12/horizon-installation/GUID-E04FDAC2-AD7B-4B09-B6E0-4A541646869B.html

Deploy VMware Unified Access Gateway with PowerShell

Each version of the Unified Access Gateway will also have PowerShell scripts available in a .zip file. For this post, I am using Unified Access Gateway 20.09. The components can be downloaded from https://my.vmware.com/web/vmware/downloads/info/slug/desktop_end_user_computing/vmware_unified_access_gateway/20_09. You will want to the appliance itself as well as the PowerShell scripts. For this post, I am going to use the FIPS version.

You are also going to need the ovftool that can be downloaded from https://code.vmware.com/web/tool/4.4.0/ovf. In this post, we are going to need the newest version, 4.4.1. Make sure you have installed the ovftool on your Windows machine you are going to deploy from. Also, ensure you can access ovftool from the command line. You may have to add this to the System PATH.

Optionally, you can use an Integrated Development Environment (IDE) of your choice. For this post, I am going to use Visual Studio Code, available at https://code.visualstudio.com/download.

I have TLS certificates from Let’s Encrypt, so I am going to deploy with them as well. For me, this is how I have created my directory structure.

  • certs – contains all of my TLS certificates needed
  • ova – contains the ova I am going to deploy
  • uag-001v_setting.ini – this is the settings file I use for one of my Unified Access Gateways (UAG). You would need one per UAG you want to deploy.
  • uagdeploy.ps1 – PowerShell script included in the PowerShell scripts zip
  • uagdeploy.psm1 – PowerShell module included in the PowerShell scripts zip

The following is my uag-001v_settings.ini file. You can get the “barebones” file after deploying and configuring at least one UAG in vSphere. Then you can export the settings and modify as necessary.


[General]
eth0ErrorMsg={"netmask":"SUCCESS","ip":"SUCCESS","defaultGateway":"SUCCESS"}
#netInternet: Portgroup used in vSphere for Internet/DMZ facing interface
netInternet=DMZ
#ip0: IP address for the netInternet interface
ip0=10.10.10.30
diskMode=
#source: The location of the OVA to deploy
source=.\ova\euc-unified-access-gateway-fips-20.09.0.0-16949983_OVF10.ova
#ip1: IP address for the internal interface
ip1=192.168.92.30
#defaultGateway: IP address for the gateway on the netInternet interface
defaultGateway=10.10.10.1
#target: User ([email protected]), 
#target: vCenter(vcenter.aaronrombaut.com), 
#target: host location (/Datacenter/host/Cluster/vmh-001p.aaronrombaut.com)
target=vi://[email protected]:[email protected]/Datacenter/host/Cluster/vmh-001p.aaronrombaut.com
#ds: Datastore to install to
ds=Synology-LUN01
netmask0=255.255.255.0
#netManagementNetwork= Portgroup used in vSphere
netManagementNetwork=LAN
#netBackendNetwork: Portgroup used in vSphere
netBackendNetwork=LAN
ip0AllocationMode=STATICV4
#name: Name of the Unified Access Gateway appliance
name=uag-001v
deploymentOption=twonic
ip1AllocationMode=STATICV4
netmask1=255.255.255.0
authenticationTimeout=300000
fipsEnabled=true
sysLogType=UDP
uagName=uag-001v
clockSkewTolerance=600
locale=en_US
tls12Enabled=true
tls13Enabled=false
ipMode=STATICV4
requestTimeoutMsec=10000
ipModeforNIC2=STATICV4
tls11Enabled=false
clientConnectionIdleTimeout=360
tls10Enabled=false
adminCertRolledBack=false
cookiesToBeCached=none
enableHTTPHealthMonitor=false
snmpEnabled=false
maxSystemCPUAllowed=100
healthCheckUrl=/favicon.ico
quiesceMode=false
#dns: The Domain Name System server in use
dns=192.168.92.10
isTLS13SetByUser=false
isCiphersSetByUser=false
tlsPortSharingEnabled=true
ceipEnabled=false
bodyReceiveTimeoutMsec=15000
monitorInterval=60
maxConnectionsAllowedPerSession=16
cipherSuites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
adminPasswordExpirationDays=90
httpConnectionTimeout=120
#dnsSearch: The domain name to use in queries
dnsSearch=aaronrombaut.com
isTLS11SetByUser=false
sessionTimeout=36000000
syslogSystemMessagesEnabled=false
ssl30Enabled=false
#ntpServers: Servers for providing time in the infrastructure
ntpServers=time.cloudflare.com
#sshEnabled: Leave this blank to NOT enable ssh which is recommended in Production
sshEnabled=

[Horizon]
#proxyDestinationUrl: URL for the VMware Horizon Connection Server
proxyDestinationUrl=https://hzn7cs-001v.aaronrombaut.com
disableHtmlAccess=false
rewriteOriginHeader=false
healthCheckUrl=/favicon.ico
proxyDestinationIPSupport=IPV4
queryBrokerInterval=300
matchWindowsUserName=false
windowsSSOEnabled=false
pcoipDisableLegacyCertificate=false
gatewayLocation=External
securityHeaders={"X-Frame-Options":"SAMEORIGIN","Strict-Transport-Security":"max-age=63072000; includeSubdomains; preload","X-Content-Type-Options":"nosniff","Content-Security-Policy":"default-src 'self';font-src 'self' data:;script-src 'self' 'unsafe-inline' 'unsafe-eval' data:;style-src 'self' 'unsafe-inline';img-src 'self' blob: data:","X-XSS-Protection":"1; mode=block"}
proxyDestinationUrlThumbprints=
tunnelExternalUrl=https://myhorizon.aaronrombaut.com:443
blastExternalUrl=https://myhorizon.aaronrombaut.com:8443
radiusClassAttributeList=
smartCardHintPrompt=false
logoutOnCertRemoval=false
redirectHostMappingList=
proxyPattern=(/|/view-client(.*)|/portal(.*)|/appblast(.*))
pcoipExternalUrl=72.225.4.11:4172

[SSLCert] #External facing
pemPrivKey=
pemCerts=
#pfxCerts: The location where the certificates are located
pfxCerts=.\certs\myhorizon-prod.p12
pfxCertAlias=

[SSLCertAdmin] #Internal facing
pemPrivKey=
pemCerts=
#pfxCerts: The location where the certificates are located
pfxCerts=.\certs\myhorizon-prod.p12
pfxCertAlias=

[PackageUpdates]
packageUpdatesScheme=OFF

Open an elevated PowerShell window and navigate to the working directory. You will want to type the following to start the deployment:

.\uagdeploy.ps1 .\uag-001v_settings.ini

Enter and re-enter the root password and admin password.

Enter Yes or No if you want to join the VMware Customer Experience Improvement Program (CEIP).

Enter the password for the vCenter you specified in the settings.ini file above.

You should receive a “deployed successfully” message at the end. From here, you should be able to navigate to the appliance in a web browser and if you used certificates, your page should be accessible, securely. For me, I access the UAG at https://uag-001v.aaronrombaut.com:9443.

If you have other Unified Access Gateways to deploy, just modify the settings.ini file and deploy. Make sure you have records for each UAG added to your DNS forward and reverse lookup zones.