Knowledge Center

Third-party DLL injectors, code detours, and hooking
Technical Articles ID:   KB88085
Last Modified:  9/27/2019


McAfee Endpoint Security (ENS) Threat Prevention 10.x
McAfee VirusScan Enterprise (VSE) 8.x


This article explains a common behavior of software applications that run in Microsoft Windows environments, where the software applications inject code into a process that is not their own. Although this behavior is similar to the behavior of malware, it is also a built-in mechanism for Microsoft Windows. This mechanism allows software developers to provide a richer computing experience for the user. 

McAfee has encountered numerous software applications with legitimate reasons to load DLLs into McAfee processes. Although McAfee tries to block those DLL load attempts, technical limitations allow those injections to sometimes succeed, which requires an alternate response from McAfee, to allow ENS to continue to operate normally. This situation has led to improvements in the ENS 10.5.2 release, and a new utility, MfeSysPrep.exe. MfeSysPrep.exe is available through Technical Support, and can be used as a DLL injector discovery tool. This utility is recommended for any environment experiencing symptoms caused by the presence of third-party DLLs in McAfee processes, including those third-party DLLs described in the Problem statements below.

McAfee software considers third-party DLLs that have injected into McAfee processes untrusted. In other words, we did not write their code, we do not know what that code does, nor what it is capable of doing, or if it can ever be compromised and used maliciously. What we do know is that any work done from the DLL functions of that third party, will appear to be coming from the injected McAfee process. So, if it is malicious activity, it looks like the McAfee software is performing malicious operations. Allowing third-party DLL injections into McAfee processes is not acceptable, and we have taken measures using our Access Protection (AP) technology, also known as Arbitrary Access Control, to secure against third-party DLL injections. We have also implemented a Validation and Trust Protection (VTP) framework, to secure against scenarios where an injection might have occurred.

The Validation and Trust Protection framework was introduced with VirusScan Enterprise 8.7i. The Access Protection feature was introduced with VirusScan Enterprise 8.0i, and the underlying technology was updated in VirusScan Enterprise 8.8 Patch 5 (or update 4 + Hotfix 929019). The new technology is called Arbitrary Access Control (AAC). Other McAfee products have made similar technological transitions to adopt this shared technology, needed for securing the products against malware and untrusted entities.

How we use Validation and Trust Protection:
The McAfee Validation and Trust Protection service, MFEVTPS.exe, is a critical service that performs inspections of DLLs and running processes, including McAfee processes, that interact with McAfee code, to verify that those objects are trusted. The service depends on the Microsoft Cryptographic service (CryptSvc), Trust-related APIs, and the health of the Certificate store and Catalog files. If those dependencies are in a bad state, our service might not function properly.

To learn more about the signing of files, see https://msdn.microsoft.com/en-us/windows/hardware/drivers/install/digital-signatures:
  • A validation check occurs when McAfee code needs to make sure that the acting process is trusted, that the target object is trusted, or both.
  • When McAfee processes initialize, the VTP service is used to validate that we are loading trusted code, and we use AP/AAC to make sure that we load only trusted DLLs.
As stated above, only McAfee code and Microsoft code are implicitly trusted.

  • Results of a validation check are cached by MFEVTPS.exe to improve the performance of future validation checks.
  • The cache is always examined first when performing a validation check.
  • If a validation check returned "untrusted," that object is cached as untrusted. If an object was cached as untrusted incorrectly, only a cache reset can correct it.
The cache is reset only when booting to Safe Mode, but not Safe Mode with Networking, or by running the command: VTPInfo.exe /ResetVTPCache. McAfee can also reset the cache via the DAT, when needed. Immediately following the cache reset, a user might experience a brief period of slow performance.

Trust Failures
A trust failure sees a validation check that results in "untrusted" when the expected result was "trusted."

  • A McAfee process has been injected by a third party. We do not trust the third party, so the process fails a validation check. 
  • A Microsoft catalog signed file has invalid signing information so it can't be verified, and fails to be loaded by a McAfee process. 
  • A valid DLL file was cached as "untrusted" incorrectly, and subsequent attempts to load it are denied. 
All these examples can cause the affected McAfee process to fail, such as not loading properly, or not performing its expected duties properly. These failures occur because of our own security mechanism (AAC), denying access to untrusted code.

How we use Access Protection or Arbitrary Access Control:
Access Protection was replaced by Arbitrary Access Control. This technology operates from the Windows kernel. It can block access to objects, such as network, file, registry, and process objects. The feature is driven by a set of rules to determine what to block and what to allow. Rules describe "bad" or "unsafe" behaviors that must be blocked or denied. Many of the rules are under your control, exposed in the User Interface or ePolicy Orchestrator (ePO) policies. Some rules that are not exposed yet are still in effect – we consider those rules critical to the operational health of the product. Rules that we expose for you can be enabled or disabled, set to report only, or modified to add other processes to protect or protect against, or to exclude so we no longer block a certain process from violating the rule. We also allow you to create your own behavior-blocking rules, making this feature one of the most powerful tools at your disposal, to secure your environment.

To summarize how AAC works, it sees an operation that is trying to be carried out, steps in, and asks the following:
  • What is the process name?
  • What object is the process accessing?
  • What is the process trying to do to that object?
  • Is this allowed, Yes or No?
An answer of "No" means that we block it. If "Report" is enabled, we log it and send an event to the ePO server. An answer of "Yes" means that the action is permitted.

Our private rules include more criteria, such as:
  • Who wrote this code? Getting this information involves checking the digital signature.
  • Do we trust the digital certificate of that vendor? We trust only McAfee and Microsoft by default.
The added criteria provides more security. Conversely, as explained above, if a validation check fails or yields an "untrusted" result, our own protections might block McAfee processes from accessing objects. As indicated in the Background statement above, that outcome is expected.


  • A third-party process is being blocked from accessing McAfee-protected files or processes. The third-party process receives error code 5, which means ACCESS_DENIED. This behavior is expected and is not a problem with the McAfee software.
  • A third-party process that loads McAfee DLLs is blocked from accessing other McAfee processes, files, or folders. For example, Microsoft Outlook, which loads the ENS exploits prevention user-mode hooking DLLs, is blocked when trying to access other McAfee processes.


  • A McAfee process fails to start properly. For example, you are unable to launch the ENS console despite installation logs indicating success.
  • A McAfee process is running, but only partially operational. For example, ENS loads successfully, but indicates that other services are not running properly, yet the applicable service is running.
  • An installation of a McAfee product fails, and tries to roll back, which might also fail, leaving remnants of a failed installation attempt.
  • A McAfee process is blocked from accessing other files or folders belonging to a different McAfee product. For example, McScript_InUse.exe which belongs to the McAfee Agent, is blocked from accessing ENS folders.


At a high level, the cause is as follows:
Processes load DLLs, which contain code that is executed. All this work is done by the process, using threads. A third party DLL is one that is not built by the vendor, which in this case is McAfee, or Microsoft. When a McAfee process loads a third-party DLL, that process has been "injected" by that third-party DLL, which means it now contains third-party functions, and can execute that third party's code. The outcome is that the McAfee process can be performing operations that were never intended for that process to ever do. There will also be no awareness of what those actions might be, because it is not McAfee code, yet the third-party code is active and resides within the McAfee process.

NOTE: Many third-party application vendors write legitimate software that applies DLL injection as a means to facilitate their product functionality. Many examples exist, but these names are of some vendors whose applications are commonly encountered in the corporate space: Citrix, Avecto, Lumension, Beyond Trust, nVidia, and so on.


ENS 10.2 introduces a process named MFECanary.exe that runs as a child process to MFEEsp.exe, and captures digital certificate details for any DLL that attempts to inject into it. The information is sent to ePO via an agent event, which is processed at the ePO server and then populated to the Endpoint Security Common policy.

When you see that certificates are populated in the Endpoint Security Common policy, it is an indication that untrusted third-party software exists in the environment that is actively trying to load DLLs into McAfee processes. If they succeed, their presence inside McAfee processes leads to the Problem statements described above.

From the Endpoint Security Common policy, a checkbox you control, decides whether client systems trust or not trust the third-party certificate.


Technical Support assistance: 
Technical Support can also help in identifying the third party and later trusting a third-party digital certificate of signed third-party DLLs injecting into McAfee processes. This support requires opening a Service Request. To expedite processing, the steps to achieve these tasks are provided in this article.

Option 1 - Identifying the third party - easiest method:
Accompanying the release of ENS 10.5.2 is a utility available from Technical Support named MfeSysPrep.exe. MfeSysPrep.exe can be run standalone on any target system for DLL injector discovery, distributed through third-party tools, or deployed via ePO. The tool writes a log file locally, and sends ePO events for identified untrusted DLLs that could impact ENS functionality.

For each identified injector and corresponding digital certificate, if McAfee has deemed that specific certificate trustworthy, the tool automatically updates ENS to trust the certificate. If no outstanding untrusted DLLs were discovered by the tool, no further action is needed. See the Related Information section of this article for implications of trusting a third-party certificate.

For each untrusted and signed DLL that was identified, the certificate information is captured in the log and provided to ePO in an event, with event ID 1092 or 1095. For each untrusted and unsigned DLL that was identified, a SHA1 and SHA2 hash is recorded for that DLL in the log and provided to ePO in an event, with event ID 1092 or 1095.

Event ID 1092 indicates that an injector was found, and we could not trust it.
Event ID 1095 indicates that an injector was found, and we trusted it automatically.

Option 2 - Identifying the third party - hard method:
This identification is perhaps the most hard undertaking of the whole process because the method to use for collecting this information depends on whether you are experiencing symptoms that affect product behavior. See the sections below on collecting data when there are "No adverse symptoms" and when there are "Adverse symptoms:"
  • No adverse symptoms:
    1. Use Process Explorer, available from Microsoft.
    2. Run the tool as an Administrator.
    3. Select the McAfee process that has the third party DLL inside it.
    4. Use the DLL view, or press CTRL+D, and identify the third-party DLL, noting its location on disk. If there are multiple locations, note them all.
    5. Use Windows Explorer to locate that DLL on disk, and access its Properties.
    6. See Obtaining the .cer File below.
  • Adverse symptoms:
    • When the adverse symptoms occur during a Windows session, there are two data points to collect. These data points should ideally be collected in parallel, that is, run both tools at the same time and capture the issue once.
      1. Collect Process Monitor data:
        1. Use Process Monitor, available from Microsoft.
        2. Run the tool as an Administrator.
        3. Launch the McAfee process that is behaving unexpectedly.
        4. After it has reproduced the anomalous behavior, save the Procmon capture.
          IMPORTANT: Save All Events and use the Native PML format.
        5. Provide this information to Technical Support to help with identifying the third-party DLL being loaded by the McAfee process.
      2. Collect AMTrace data:
        1. Run AMTrace.exe by seeing KB86691, from an Administrator command prompt, providing the option to start immediately.
        2. Launch the McAfee process that is behaving unexpectedly.
        3. After it has reproduced the anomalous behavior, run AMTrace again, now providing the option to stop.
        4. Provide the created log to Technical Support to help with identifying the untrusted DLL.
    • When the adverse symptoms occur only on Windows boot/startup, there are two data points to collect, which can be collected in serial or parallel:
      1. Collect Process Monitor data:
        1. Use Process Monitor, available from Microsoft.
        2. Run the tool as an Administrator.
        3. From the Options menu, click Enable Boot Logging.
        4. Reset the VTP cache*.
        5. Reboot the system.
        6. Log on and run Process Monitor again.
        7. Save the Procmon boot logging capture.
          IMPORTANT: Save All Events and use the Native PML format.
        8. Provide this information to Technical Support to help with identifying the third-party DLL being loaded by the McAfee process.
      2. Collect AMTrace data.
        1. Run AMTrace.exe by seeing KB86691 from an Administrator command prompt, providing the option to start on boot.
        2. Reset the VTP cache*.
        3. Reboot the system.
        4. After it has reproduced the anomalous behavior, run AMTrace again, now providing the option to stop.
        5. Provide the created log to Technical Support to help with identifying the untrusted DLL.
* To reset the VTP cache:
  1. From an Administrator command prompt, navigate to \Program Files\Common Files\McAfee\SystemCore.
  2. Run the VTPInfo.exe utility with the parameter /resetvtpcache:

    VTPInfo.exe /ResetVTPCache

Obtaining the .cer file:
This procedure is needed only if you performed Option 2 above to identify the third-party untrusted DLLs. Obtaining the .cer file is not needed if you used Option 1 above.

Provide the .cer file to Technical Support, and in return you receive an executable package that will automatically add the certificate to the McAfee Trust certificate store. The executable package needs to be run on any or all systems where the third-party digital certificate must be trusted. If an ePO-deployable version of the package is not already provided, one can be built using the ePO Endpoint Deployment Kit (EEDK) tool available from the McAfee Community: https://community.mcafee.com/t5/Documents/ePO-Endpoint-Deployment-Kit-9-6-1-Enterprise-Edition/ta-p/553541.
  1. To access the properties of the EXE or DLL file, right-click and select Properties.
  2. Click the Digital Certificates tab.
NOTE: If this tab is missing, the file is not digitally signed and there is no option for us to trust this vendor's code. You must contact that vendor and request they provide signed code.
  1. Select a certificate from the Signature list that you want to trust.
  2. Click Details.
  3. Click View Certificate.
  4. Click the Details tab.
  5. Click Copy to File.
  6. In the Certificate Export Wizard, click Next.
  7. Click Next again to accept creating a .cer file.
  8. Specify a file name and location. For example, C:\Temp\My3rdParty.cer.
  9. Click Next and Finish.

Trusting a third-party digital certificate:
If you used Option 1 above, and no further untrusted DLLs were found, no further action is needed.

For untrusted and signed DLLs, we expect that the digital certificate for that DLL appears in the Endpoint Security Common policy, in which case you can see Solution 1.
If the certificate has not appeared in that policy, or if you want to move the process along in a more predictable manner, after you have obtained the .cer file, you must work with Technical Support to obtain an EXE that will update ENS, to begin trusting the provided certificate.
See the Related Information section of this article for implications of trusting a third-party certificate.

For untrusted and unsigned DLLs, we expect the vendor of that software to provide you with digitally signed code, as it is an industry standard for identifying released software. In cases where vendors can't or will not comply, Technical Support can help you. You need to provide the SHA1 and SHA2 hashes for any unsigned DLLs identified by MfeSysPrep.exe as injectors, and the binary for the DLL itself. The SHA hashes are captured in the MfeSysPrep logging.

IMPORTANT: Before trust can be added to a third-party DLL, you must submit the DLL to McAfee Labs as a sample using the instructions in KB85567 to confirm that it is non-malicious. Include the following information when submitting such DLL samples:
  • Software manufacturer 
  • Parent executable or process  
  • Is this application internal or publicly available?  
  • Brief description of the software's function and purpose including the DLL submitted

Rate this document

Beta Translate with

Select a desired language below to translate this page.


This article is available in the following languages:

English United States

Glossary of Technical Terms

 Highlight Glossary Terms

Please take a moment to browse our Glossary of Technical Terms.