Knowledge Center

Third-party DLL injectors, code detours, and hooking
Technical Articles ID:   KB88085
Last Modified:  8/28/2017


McAfee Endpoint Security (ENS) Threat Prevention 10.x
McAfee VirusScan Enterprise (VSE) 8.x


This article explains a common behavior of software applications that run in Microsoft Windows environments, where the software applications inject code into a process that is not their own. While this behavior is similar to the behavior of malware, it is also a built-in mechanism to Microsoft Windows to allow software developers to provide a richer computing experience for the user. 

McAfee has encountered numerous software applications with legitimate reasons to load DLLs into McAfee processes, and while McAfee will attempt to block those DLL load attempts, technical limitations allow those injections to sometimes succeed, which requires an alternate response from McAfee to allow ENS to continue to operate normally. This has led to improvements in the ENS 10.5.2 release, and a new utility, MfeSysPrep.exe. MfeSysPrep.exe is available through Technical Support, and can be used as a DLL injector discovery tool, recommended for any environment experiencing symptoms caused by the presence of third-party DLLs within McAfee processes, including those described in the Problem statements below.

McAfee software considers third-party DLLs that have injected into McAfee processes untrusted. In other words, we did not write their code, we do not know what that code will do nor what it is capable of doing, or if it can ever be compromised and used maliciously. What we do know is that any work done from that third-party DLL's functions will appear to be coming from the injected McAfee process – so if it were malicious activity, it would look like the McAfee software was performing malicious operations. Allowing third-party DLL injections into McAfee processes is not acceptable, and we have taken measures using our Access Protection (AP) technology (also known as Arbitrary Access Control) to secure against third-party DLL injections. We have also implemented a Validation and Trust Protection (VTP) framework to secure against scenarios where injection may have occurred.

The Validation and Trust Protection framework was introduced with VirusScan Enterprise 8.7i. The Access Protection feature was introduced with VirusScan Enterprise 8.0i, and the underlying technology was updated in VirusScan Enterprise 8.8 Patch 5 (or Patch 4 + Hotfix 929019). The new technology is called Arbitrary Access Control (AAC). Other McAfee products have made similar technological transitions to adopt this shared technology, needed for securing the products against malware and untrusted entities.

How we use Validation and Trust Protection:
The McAfee Validation and Trust Protection service (MFEVTPS.exe) is a critical service that performs inspections of DLLs and running processes (including McAfee processes) that interact with McAfee code, to verify those objects are trusted. The service depends upon the Microsoft Cryptographic service (CryptSvc), Trust-related APIs, and the health of the Certificate store and/or Catalog files. If those dependencies are in a bad state, our service may not function properly.

To learn more about the signing of files, see https://msdn.microsoft.com/en-us/windows/hardware/drivers/install/digital-signatures:
  • A validation check occurs whenever McAfee code needs to ensure the acting process is trusted, that the target object is trusted, or both.
  • When McAfee processes initialize, the VTP service is used to validate that we are loading trusted code, and we use AP/AAC to ensure we load only trusted DLLs.
As stated above, only McAfee code and Microsoft code are implicitly trusted.

  • Results of a validation check are cached by MFEVTPS.exe to improve performance of future validation checks.
  • The cache is always examined first when performing a validation check.
  • If a validation check returned "untrusted," that object is cached as untrusted. If an object was cached as untrusted incorrectly, only a cache reset can correct it.
The cache is reset only when booting to Safe Mode (not Safe Mode with Networking) or by running the command: VTPInfo.exe /ResetVTPCache. McAfee can also reset the cache via the DAT, when necessary. Immediately following the cache reset, a user may experience a brief period of slow performance.

Trust Failures
A trust failure refers to a validation check that results in "untrusted" when the expected result was "trusted."

  • A McAfee process has been injected by a third party (we do not trust the third party, so the process fails a validation check). 
  • A Microsoft catalog signed file has invalid signing information so it cannot be verified, and fails to be loaded by a McAfee process. 
  • A valid DLL file was cached as "untrusted" incorrectly, and subsequent attempts to load it are denied. 
All of these examples can cause the affected McAfee process to fail, such as not loading properly, or not performing its expected duties properly. These failures will be occurring because of our own security mechanism (AAC) denying access to untrusted code.

How we use Access Protection/Arbitrary Access Control:
Access Protection was replaced by Arbitrary Access Control. This technology operates from the Windows kernel. It is capable of blocking access to objects, such as network, file, registry, and process objects. The feature is driven by a set of rules to determine what to block and what to allow. Rules describe "bad" or "unsafe" behaviors that should be blocked or denied. Many of the rules are under your control, exposed in the User Interface or ePolicy Orchestrator (ePO) policies; some rules that are not exposed yet are still in effect – we consider those rules critical to the operational health of the product. Rules that we expose for you can be enabled or disabled, set to report only, or modified to add other processes to protect or protect against, or to exclude so we no longer block a certain process from violating the rule. We also allow you to create your own behavior-blocking rules, making this feature one of the most powerful tools at your disposal to secure your environment.

To summarize how AAC works, it sees an operation that is trying to be carried out, steps in, and asks the following:
  • What is the process name?
  • What object is the process accessing?
  • What is the process trying to do to that object?
  • Is this allowed (Yes or No)?
An answer of "No" means we block it (and if "Report" is enabled, we log it and send an event to the ePO server). An answer of Yes" means the action is permitted.

Our private rules include additional criteria, such as:
  • Who wrote this code? (Checking the digital signature)
  • Do we trust that vendor's digital certificate? (We trust only McAfee and Microsoft by default)
The added criteria provides more security. Conversely, as explained above, if a validation check fails or yields an "untrusted" result, our own protections may block McAfee processes from accessing objects. As indicated in the Background statement above, that outcome is expected.


  • A third-party process is being blocked from accessing McAfee-protected files or processes. The third-party process receives error code 5, which means ACCESS_DENIED. (This is expected behavior and not a problem with the McAfee software.)
  • A third-party process that loads McAfee DLLs is blocked from accessing other McAfee processes, files, or folders. For example, Microsoft Outlook (which loads the ENS exploit prevention user-mode hooking DLLs) is blocked when trying to access other McAfee processes.


  • A McAfee process fails to start properly. For example, you are unable to launch the ENS console despite installation logs indicating success.
  • A McAfee process is running, but only partially operational. For example, ENS loads successfully but indicates other services are not running properly, yet the applicable service is running.
  • An installation of a McAfee product fails, and attempts to roll back which may also fail, leaving remnants of a failed installation attempt.
  • A McAfee process is blocked from accessing other files/folders belonging to a different McAfee product. For example, McScript_InUse.exe (belonging to the McAfee Agent) is blocked from accessing ENS folders.


At a high level, the cause is as follows:
Processes load DLLs, which contain code that is executed. All of this work is done by the process, using threads. A third-party DLL is one that is not built by the vendor (in this case, McAfee) or Microsoft. When a McAfee process loads a third-party DLL, that process has been "injected" by that third-party DLL, which means it now contains third-party functions, and can execute that third-party's code. The outcome is that the McAfee process can be performing operations that were never intended for that process to ever do – neither can there be any awareness of what those actions might be, because it is not McAfee code, yet the third-party code is active and resides within the McAfee process.

NOTE: Many third-party application vendors write legitimate software that leverages DLL injection as a means to facilitate their product functionality. Many examples exist, but to name a few vendors whose applications are commonly encountered in the corporate space: Citrix, Avecto, Lumension, Beyond Trust, nVidia, and so on.


ENS 10.2 introduces a process named MFECanary.exe that runs as a child process to MFEEsp.exe, and captures digital certificate details for any DLL that attempts to inject into it. The information is sent to ePO via an agent event, which is processed at the ePO server and then populated to the Endpoint Security Common policy.

When you see certificates are populated in the Endpoint Security Common policy, it is an indication that untrusted third-party software exists in the environment that is actively trying to load DLLs into McAfee processes. Should they succeed, their presence inside McAfee processes will lead to the Problem statements described above.

From the Endpoint Security Common policy, a checkbox you control decides whether client machines will trust or not trust the third-party certificate.


Technical Support assistance: 
Technical Support can also assist in identifying the third party and subsequently trusting a third-party digital certificate of signed third-party DLLs injecting into McAfee processes. This will require opening a Service Request and following normal escalation procedures. To expedite processing of the escalation, the steps to achieve these tasks are provided here.

Option 1 - Identifying the third party (easiest method):
Accompanying the release of ENS 10.5.2 is a utility available from Technical Support named MfeSysPrep.exe. MfeSysPrep.exe can be run standalone on any target system for DLL injector discovery, distributed through third-party tools, or deployed via ePO. The tool writes a log file locally, and sends ePO events for identified untrusted DLLs that could impact ENS functionality.

For each identified injector and corresponding digital certificate, if McAfee has deemed that specific certificate trustworthy, the tool will automatically update ENS to trust the certificate. If there are no outstanding untrusted DLLs that were discovered by the tool, no further action is necessary. See the Related Information section of this article for implications of trusting a third-party certificate.

For each untrusted and signed DLL that was identified, the certificate information is captured in the log and provided to ePO in an event (event ID 1092 or 1095). For each untrusted and unsigned DLL that was identified, a SHA1 and SHA2 hash is recorded for that DLL in the log and provided to ePO in an event (event ID 1092 or 1095).

Event ID 1092 indicates an injector was found, and we could not trust it.
Event ID 1095 indicates an injector was found, and we trusted it automatically.

Option 2 - Identifying the third party (hard method):
This is perhaps the most difficult undertaking of the whole process because the method to use for collecting this information depends on whether you are experiencing symptoms that affect product behavior. Refer to the sections below on collecting data when there are "No adverse symptoms" and when there are "Adverse symptoms:"
  • No adverse symptoms:
    1. Use Process Explorer, available from Microsoft.
    2. Run the tool as an Administrator.
    3. Select the McAfee process that has the third-party DLL inside it.
    4. Use the DLL view (or CTRL+D) and identify the third-party DLL, noting its location on disk (if there are multiple locations, note them all).
    5. Use Windows Explorer to locate that DLL on disk, and access its Properties.
    6. See Obtaining the .cer File below.
  • Adverse symptoms:
    • When the adverse symptoms occur during a Windows session, there are two data points to collect; they should ideally be collected in parallel (that is, run both tools at the same time and capture the issue once).
      1. Collect Process Monitor data:
        1. Use Process Monitor, available from Microsoft.
        2. Run the tool as an Administrator.
        3. Launch the McAfee process that is behaving unexpectedly.
        4. After it has reproduced the anomalous behavior, save the Procmon capture.
          IMPORTANT: Save All Events and use the Native PML format.
        5. Provide this information to Technical Support to help with identifying the third-party DLL being loaded by the McAfee process.
      2. Collect AMTrace data:
        1. Run AMTrace.exe (see KB86691) from an Administrator command prompt, providing the option to start immediately.
        2. Launch the McAfee process that is behaving unexpectedly.
        3. After it has reproduced the anomalous behavior, run AMTrace again, now providing the option to stop.
        4. Provide the created log to Technical Support to help with identifying the untrusted DLL.
    • When the adverse symptoms occur only Windows boot/startup, there are two data points to collect; they can be collected in serial or parallel:
      1. Collect Process Monitor data:
        1. Use Process Monitor, available from Microsoft.
        2. Run the tool as an Administrator.
        3. From the Options menu select Enable Boot Logging.
        4. Reset the VTP cache*.
        5. Reboot the system.
        6. Log on and run Process Monitor again.
        7. Save the Procmon boot logging capture.
          IMPORTANT: Save All Events and use the Native PML format.
        8. Provide this information to Technical Support to help with identifying the third-party DLL being loaded by the McAfee process.
      2. Collect AMTrace data.
        1. Run AMTrace.exe (see KB86691) from an Administrator command prompt, providing the option to start on boot.
        2. Reset the VTP cache*.
        3. Reboot the system.
        4. After it has reproduced the anomalous behavior, run AMTrace again, now providing the option to stop.
        5. Provide the created log to Technical Support to help with identifying the untrusted DLL.
* To reset the VTP cache:
  1. From an Administrator command prompt, navigate to \Program Files\Common Files\McAfee\SystemCore.
  2. Run the VTPInfo.exe utility with the parameter /resetvtpcache:

    VTPInfo.exe /ResetVTPCache

Obtaining the .cer file:
This procedure is required only if you performed Option 2 above to identify the third-party untrusted DLLs. Obtaining the .cer file is not necessary if you used Option 1 above.

Provide the .cer file to Technical Support, and in return you will receive an executable package that will automatically add the certificate to the McAfee Trust certificate store. The executable package will need to be run on any/all systems where the third-party digital certificate must be trusted. If an ePO-deployable version of the package is not already provided, one can be built using the ePO Endpoint Deployment Kit (EEDK) tool available from the McAfee Community: https://community.mcafee.com/community/business/toolexchange.
  1. Access the properties of the EXE or DLL file (right-click and select Properties).
  2. Click the Digital Certificates tab.
NOTE: If this tab is missing, the file is not digitally signed and there is no option for us to trust this vendor's code – you must contact that vendor and request they provide signed code.
  1. Select a certificate from the Signature list that you want to trust.
  2. Click Details.
  3. Click View Certificate.
  4. Click the Details tab.
  5. Click Copy to File.
  6. In the Certificate Export Wizard, click Next.
  7. Click Next again to accept creating a .cer file.
  8. Specify a file name and location (for example, C:\Temp\My3rdParty.cer).
  9. Click Next and Finish.

Trusting a third-party digital certificate:
If you used Option 1 above, and no further untrusted DLLs were found, no further action is required.

For untrusted and signed DLLs, we expect the digital certificate for that DLL will appear in the Endpoint Security Common policy, in which case you can refer to Solution 1.
If the certificate has not appeared in that policy, or you want to move the process along in a more predictable manner, after you have obtained the .cer file you should work with Technical Support to obtain an EXE that will update ENS to begin trusting the provided certificate. See the Related Information section of this article for implications of trusting a third-party certificate.

For untrusted and unsigned DLLs, we expect the vendor of that software to provide you with digitally signed code as it is an industry standard for identifying released software. In cases where vendors cannot or will not comply, Technical Support can assist if you provide the SHA1 and SHA2 hash for any unsigned DLLs identified by MfeSysPrep.exe as injectors.

Rate this document

Did this article resolve your issue?

Please provide any comments below

Beta Translate with

Select a desired language below to translate this page.

Glossary of Technical Terms

 Highlight Glossary Terms

Please take a moment to browse our Glossary of Technical Terms.