How to generate a custom SSL certificate for use with ePO using the OpenSSL toolkit


Environment

ePolicy Orchestrator (ePO) 5.x

Summary

You can use a custom SSL certificate instead of the default self-signed certificate when browsers authenticate through the ePO server. This article describes one way to create a custom SSL certificate signed by a third party Certification Authority (CA), such as Verisign.

IMPORTANT:
  • The ePO platform provides the technical mechanism to support the integration of third-party certificates.
  • The generation, validation, or troubleshooting of third-party certificates isn't supported.

Solution 1

IMPORTANT: Before you begin this process, you must install and configure the OpenSSL toolkit. Installation and configuration of the toolkit makes sure that all needed libraries and configuration files for OpenSSL are in place:
  1. Download and install the OpenSSL toolkit.

    NOTES:
    • You must download and use the full version of OpenSSL. Don't use the light version.
    • The current release is Win64 OpenSSL v3.0.2 140-MB installer EXE MSI.
       
  2. Extract and install the version you downloaded based on your platform (32-bit or 64-bit).

    NOTE: For more information about obtaining the OpenSSL toolkit, see the OpenSSL site.
     
  3. Create the following folder: c:\ssl\keys
     
  4. Open a command prompt as an Administrator. Change the directory to the OpenSSL bin folder where OpenSSL is installed. Example: cd C:\OpenSSL-Win64\bin

Obtain a custom SSL certificate for use with ePO:
  1. Create a new private key using OpenSSL with 2048-bit strength and encrypt it using des3:

    openssl genpkey -out c:\ssl\keys\mcafee.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048
     
  2. Save a copy of the encrypted .key file. You need the key to create an unencrypted version in step 7 for importing into ePO.

    Check the MD5 value of the .key file created by running the OpenSSL switch below:

    openssl rsa -noout -modulus -in C:\ssl\keys\mcafee.key | openssl md5

    Example: 
    C:\Program Files\OpenSSL-Win64\bin>openssl rsa -noout -modulus -in C:\ssl\keys\mcafee.key | openssl md5
    rsa: Use -help for summary.
    MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e

     
  3. Later versions of Chrome and Edge require a certificate with the Subject Alt Name field populated. To create a Certificate Signing Request (CSR) for use with Chrome or Edge, you need to first create a configuration file by performing the steps below:
    1. Create a file named sancert.cnf with the information provided below.
    2. Save it in the C:\OpenSSL-Win64\bin folder, or where OpenSSL is installed, as long as it's in the bin folder.
    3. Don't change anything in the [req] or [ req_ext ] section below.
    4. Substitute information when needed in the [dn] and [alt_names] sections.

      Example:
[ req ]
default_bits = 2048
distinguished_name = dn
req_extensions = req_ext

[ dn ]
C=US
ST=State name (note: use full state name instead of abbreviation)
L=City
O=Organization name
OU=Domain name
CN=fqdn of server name

[ req_ext ]
subjectAltName = @alt_names

[alt_names]
DNS.1 = dns1
DNS.2 = dns2.com (optional)
DNS.3 = dns3.com (optional)
  1. Create the CSR with the Common Name (CN), which then sees sancert.cnf to add the Subject Alternative Name.
Run the command below, and substitute the correct values within the quotation marks. 

openssl req -key c:\ssl\keys\mcafee.key -config sancert.cnf -new -subj "/C=US/ST=state/L=city/O=OrgName/OU=domain.com/CN=servername.domain.com" -out c:\ssl\keys\mcafee.csr​

NOTE: The CN= value is what the certificate is issued to, whether it's the FQDN or NetBIOS name of the server. If the ePO is clustered, use the virtual cluster name. These values are entered into the subject field of the certificate.

CAUTION: Check that the MD5 values for .csr and .key are the same. Only then perform step 5 onward.

To check the MD5 value for the .csr file, run the OpenSSL switch below:

openssl rsa -noout -modulus -in C:\ssl\keys\mcafee.csr |openssl md5

Example:
C:\Program Files\OpenSSL-Win64\bin>openssl rsa -noout -modulus -in C:\ssl\keys\mcafee.csr | openssl md5
rsa: Use -help for summary.
MD5(stdin)= d41d8cd98f00b204e9800998ecf8427e
  1. ​To obtain the server certificate, submit the CSR to your third-party CA or Enterprise CA.
A custom SSL certificate can be certified using either of the below: 
  • An ECA (a domain CA in Windows):
    1. On the domain controller CA server, open a command prompt as an Administrator.
    2. Run the command below. This command forces the use of the web server template that generates the server certificate.

      certreq -submit -attrib "CertificateTemplate: WebServer" c:\ssl\keys\mcafee.csr

      NOTE: The template name might differ for your CA, as long as it's a webserver type template.
       
    3. Click OK when the dialog window is displayed for the LDAP server.
    4. Save the .cer to a location of your choice. A .cer file that contains the root CA certificate and ePO certificate is created.
       
  • A third-party CA, such as Verisign or Entrust:
    The third-party CA might provide a single certificate file (.cer), or sometimes more than one certificate file (.crt). The certificate obtained always contains the CA root certificate and server certificate at the least.
    For example:
    • A root certificate file representing the root certificate
    • A certificate file for an intermediate certificate
    • A server certificate 
The three certificate files (.crtare combined into the correct format for ePO to use with the following steps. But, if the CA can provide the certificate in PKCS#7 format, it doesn't need further conversion. You can supply the resulting (.cerfile to ePO.
 
If you receive separate .crt files, follow the steps below to combine the .crt files.
  1. Use the Export Certificate wizard in Windows, or OpenSSL for Windows or Linux using the command below:

    openssl crl2pkcs7 -nocrl -certfile cert1.crt -certfile cert2.crt -certfile cert3.crt -out outfile.p7b

    NOTE: The command creates a certificate chain file from the 'cert1.crt, cert2.crtcert3.crtfiles called outfile.p7b. The p7b file contains the entire certificate chain, which can be supplied to ePO. The order of the chain must be as follows: Root certificate, intermediate certificate (if it exists), and ePO certificate.
  1. Using the OpenSSL toolkit and the encrypted .key file from step 1, create an unencrypted version of the private key for inputting into ePO:

    For OpenSSL version 1.x, use the command below:

    openssl rsa -in c:\ssl\keys\mcafee.key -out c:\ssl\keys\unsecured.mcafee.pem

    For OpenSSL version 3.x, use the commands below:

    NOTE: This step is critical when using a higher version of OpenSSL or any method that generates the key in pkcs8 format. Currently, we don't support pkcs8 for the browser certificate, so it's necessary to convert the pkcs8 key to pkcs1. Both the commands are needed to be executed.

    openssl pkcs8 -in c:\ssl\keys\mcafee.key -topk8 -nocrypt -out c:\ssl\keys\pkcs8mcafee.pem

    openssl pkcs8 -in c:\ssl\keys\pkcs8mcafee.pem -traditional -nocrypt -out c:\ssl\keys\unsecured.mcafee.pem
     
  2. Use the new certificate and private key file to update the ePO certificate:

    IMPORTANT: Before importing any certificate into the ePO console, back up the <ePO installation folder> \Server\keystore\ folder.

    NOTE: In addition to this list, the browser certificate presented must be recognized as Trusted. This step is important so it can be trusted with your Enterprise CA when it's added to the Trusted Root CA list. For more details, see your operating system documentation.
     
    1. Log on to the ePO console.
    2. Click MenuConfigurationServer Settings.
    3. Click Server Certificate under Setting Categories, and then click Edit.
    4. Select Use the provided certificate and private key.
    5. Click Browse in the Certificate (P7B, PEM) field. Locate and select the certificate file (.p7b or .cer). Then, click Open.
    6. Click Browse in the Private key (PEM) field.
    7. Go to and select the private key file (unsecured.mcafee.pem in this case), and then click Open.
    8. Click Save.
    9. Restart the ePO services:
      1. Press Windows+R, type services.msc, and then click OK.
      2. Right-click each of the following ePO services and click Restart:

        McAfee ePolicy Orchestrator x.x.x Application Server
        McAfee ePolicy Orchestrator x.x.x Event Parser
        McAfee ePolicy Orchestrator x.x.x Server
         
      3. Close the services window.

Solution 2

If the application of the custom SSL certificate fails or needs to be rolled back:
  1. Go to <ePO installation folder>\Server\keystore\
  2. Locate the following files:
    • server.keystore
    • server.keystore.backup<date string>
  3. Rename server.keystore to server.keystore.bad.
  4. Rename server.keystore.backup<date string> to server.keystore.
  5. Restart the ePO services:
    1. Press Windows+R, type services.msc, and click OK.
    2. Right-click each of the following ePO services and click Restart:

      McAfee ePolicy Orchestrator x.x.x Application Server
      McAfee ePolicy Orchestrator x.x.x Event Parser
      McAfee ePolicy Orchestrator x.x.x Server
    3. Close the services window.