Knowledge Center

Third-party DLL injectors, code detours, and hooking
Technical Articles ID:   KB88085
Last Modified:  1/25/2017


McAfee Endpoint Security (ENS) Threat Prevention 10.x
McAfee VirusScan Enterprise (VSE) 8.x


This article explains a common behavior of software applications that run in Microsoft Windows environments, where the software applications inject code into a process that is not their own. While this behavior is similar to the behavior of malware, it is also a built-in mechanism to Microsoft Windows to allow software developers to provide a richer computing experience for the user. 

McAfee software considers third-party DLLs that have injected into McAfee processes untrusted. In other words, we did not write their code, we do not know what that code will do nor what it is capable of doing, or if it can ever be compromised and used maliciously. What we do know is that any work done from that third-party DLL's functions will appear to be coming from the injected McAfee process – so if it were malicious activity, it would look like the McAfee software was performing malicious operations. Allowing third-party DLL injections into McAfee processes is not acceptable, and we have taken measures using our Access Protection (AP) technology (also known as Arbitrary Access Control) to secure against third-party DLL injections. We have also implemented a Validation and Trust Protection (VTP) framework to secure against scenarios where injection may have occurred.

The Validation and Trust Protection framework was introduced with VirusScan Enterprise 8.7i. The Access Protection feature was introduced with VirusScan Enterprise 8.0i, and the underlying technology was revamped in VirusScan Enterprise 8.8 Patch 5 (or Patch 4 + Hotfix 929019). The new technology is called Arbitrary Access Control (AAC). Other McAfee products have made similar technological transitions to adopt this shared technology, needed for securing the products against malware and untrusted entities.

How we use Validation and Trust Protection:
The McAfee Validation and Trust Protection service (MFEVTPS.exe) is a critical service that performs inspections of DLLs and running processes (including McAfee processes) that interact with McAfee code, to verify those objects are trusted. The service depends upon the Microsoft Cryptographic service (CryptSvc), Trust-related APIs, and the health of the Certificate store and/or Catalog files. If those dependencies are in a bad state, our service may not function properly.

To learn more about the signing of files, see: https://msdn.microsoft.com/en-us/windows/hardware/drivers/install/digital-signatures.
  • A validation check occurs whenever McAfee code needs to ensure the acting process is trusted, that the target object is trusted, or both.
  • When McAfee processes initialize, the VTP service is used to validate that we are loading trusted code, and we use AP/AAC to ensure we load only trusted DLLs.
As stated above, only McAfee code and Microsoft code are implicitly trusted.

  • Results of a validation check are cached by MFEVTPS.exe to improve performance of future validation checks.
  • The cache is always examined first when performing a validation check.
  • If a validation check returned "untrusted," that object is cached as untrusted. If an object was cached as untrusted incorrectly, only a cache reset can correct it.
The cache is reset when booting to Safe Mode only (not Safe Mode with Networking) or by running the command: VTPInfo.exe /ResetVTPCache. McAfee can also reset the cache via the DAT, when necessary. Immediately following the cache reset, a user may experience a brief period of slow performance.

Trust Failures
A trust failure refers to a validation check that results in "untrusted" when the expected result was "trusted."

  • A McAfee process has been injected by a third party (we do not trust the third party, so the process fails a validation check). 
  • A Microsoft catalog signed file has invalid signing information so it cannot be verified, and fails to be loaded by a McAfee process. 
  • A valid DLL file was cached as "untrusted" incorrectly, and subsequent attempts to load it are denied. 
All these examples can cause the affected McAfee process to fail, such as not loading properly, or not performing its expected duties properly. These failures will be occurring because of our own security mechanism (AAC) denying access to untrusted code.

How we use Access Protection/Arbitrary Access Control:
Access Protection was replaced by Arbitrary Access Control. This technology operates from the Windows kernel. It is capable of blocking access to objects, such as network, file, registry, and process objects. The feature is driven by a set of rules to determine what to block and what to allow. Rules describe "bad" or "unsafe" behaviors that should be blocked or denied. Many of the rules are under your control, exposed in the User Interface or ePolicy Orchestrator (ePO) policies; some rules that are not exposed yet are still in effect – we consider those rules critical to the operational health of the product. Rules that we expose for you can be enabled or disabled, set to report only, or modified to add other processes to protect or protect against, or to exclude so we no longer block a certain process from violating the rule. We also allow you to create your own behavior-blocking rules, making this feature one of the most powerful tools at your disposal to secure your environment.

To summarize how AAC works, it sees an operation that is trying to be carried out, steps in, and asks the following:
  • What is the process name?
  • What object is the process accessing?
  • What is the process trying to do to that object?
  • Is this allowed (Yes or No)?
An answer of "No" means we block it (and if "Report" is enabled, we log it and send an event to the ePO server). An answer of Yes" means the action is permitted.

Our private rules include additional criteria, such as:
  • Who wrote this code? (Checking the digital signature)
  • Do we trust that vendor's digital certificate? (We trust only McAfee and Microsoft by default)
The added criteria provides more security. Conversely, as explained above, if a validation check fails or yields an "untrusted" result, our own protections may block McAfee processes from accessing objects. As indicated in the Background statement above, that outcome is expected.

NOTE: It is not McAfee software that decides for you what code is trustworthy in your environment. From our software's perspective all third-party code (not McAfee and not Microsoft) is untrustworthy. However, because we inspect and validate even our own code and Microsoft's code, if those inspections return "untrusted," it is possible for our AAC protections to block our own processes from doing work, or to block Microsoft code. These validation failures can manifest a variety of symptoms, including the Problem statements below.


  • A third-party process is being blocked from accessing McAfee-protected files or processes. The third-party process receives error code 5, which means ACCESS_DENIED. (This is expected behavior and not a problem with the McAfee software.)
  • A third-party process that loads McAfee DLLs is blocked from accessing other McAfee processes, files, or folders. For example, Microsoft Outlook (which loads the Endpoint Security exploit prevention user-mode hooking DLLs) is blocked when trying to access other McAfee processes.


  • A McAfee process fails to start properly.
    For example, you are unable to launch the ENS console despite installation logs indicating success.
  • A McAfee process is running, but only partially operational.
    For example, ENS loads successfully but indicates other services are not running properly, yet the applicable service is running.
  • An installation of a McAfee product fails, and attempts to roll back which may also fail, leaving remnants of a failed installation attempt.
  • A McAfee process is blocked from accessing other files/folders belonging to a different McAfee product.
    For example, McScript_InUse.exe (belonging to the McAfee Agent) is blocked from accessing ENS folders.


At a high level, the cause is as follows. Processes load DLLs, which contain Functions, which contain Code, which is run/executed. All of this work is done by the Process using Threads. A third-party DLL is one that is not built by the vendor (in this case, McAfee), or Microsoft. When a McAfee process loads a third-party DLL, that process has been "injected" by that third-party DLL, which means it now contains third-party functions, and can execute that third-party's code. The outcome is that the McAfee process can be performing operations that were never intended for that process to ever do – neither can there be any awareness of what those actions might be, because it is not McAfee code, yet the third-party code is active and resides within the McAfee process.

NOTE: Many third-party application vendors write legitimate software that leverages DLL injection as a means to facilitate their product functionality. Countless examples exist but to name a few vendors whose applications are commonly encountered in the corporate space: Citrix, Avecto, Lumension, Beyond Trust, nVidia, and so on.


ENS 10.2 introduces a process named MFECanary.exe that runs as a child process to MFEEsp.exe, and captures digital certificate details for any DLL that attempts to inject into it. The information is sent to ePolicy Orchestrator via an agent event, which is processed at the ePolicy Orchestrator server and then populated to the Endpoint Security Common policy.

From the Endpoint Security Common policy, a checkbox you control decides whether client machines will trust or not trust the third-party certificate.


Technical Support assistance
Technical Support can also assist in identifying the third-party, obtaining the .cer file, and subsequently trusting a third-party digital certificate of signed third-party DLLs injecting into McAfee processes. This will require opening a Service Request and following normal escalation procedures. To expedite processing of the escalation, the steps to achieve these tasks are provided here.

Trusting a third-party digital certificate:
There are security implications that you are accepting by trusting the third-party certificate:
  • You accept that any code signed by the third-party certificate will be trusted to interact with McAfee processes, files, registry, and all other McAfee-protected objects.
  • You accept that file activity generated by processes signed with the third-party certificate may not be scanned.
  • You understand that any product/code releases from the same vendor if using the same digital certificate will automatically inherit the same allowances.
  • You understand that any product/code releases from the same vendor using a different digital certificate will also have to be trusted to achieve the same results.
Knowing and accepting the above, it is a relatively simple matter to provide the above .cer file to Technical Support and in return receive an executable package that will automatically add the certificate to the McAfee Trust certificate store. The executable package will need to be run on any/all systems where the third-party digital certificate must be trusted. If an ePO-deployable version of the package is not already provided, one can be built using the ePO Endpoint Deployment Kit (EEDK) tool available from the McAfee Community: https://community.mcafee.com/community/business/toolexchange.

Identifying the third party:
This is perhaps the most difficult undertaking of the whole process because the method to use for collecting this information depends on whether you are experiencing symptoms that affect product behavior. Refer to the sections below on collecting data when there are "No adverse symptoms" and when there are "Adverse symptoms:"
  • No adverse symptoms:
    1. Use Process Explorer, available from Microsoft.
    2. Run the tool as an Administrator.
    3. Select the McAfee process that has the third-party DLL inside it.
    4. Use the DLL view (or CTRL+D) and identify the third-party DLL, noting its location on disk (if there are multiple locations, note them all).
    5. Use Windows Explorer to locate that DLL on disk, and access its Properties.
    6. See Obtaining the .cer File below.
  • Adverse symptoms:
    • When the adverse symptoms occur during a Windows session, there are two data points to collect; they should ideally be collected in parallel (that is, run both tools at the same time and capture the issue once).
      1. Collect Process Monitor data:
        1. Use Process Monitor, available from Microsoft.
        2. Run the tool as an Administrator.
        3. Launch the McAfee process that is behaving unexpectedly.
        4. After it has reproduced the anomalous behavior, save the Procmon capture.
          IMPORTANT: Save All Events and use the Native PML format.
        5. Provide this information to Technical Support to help with identifying the third-party DLL being loaded by the McAfee process.
      2. Collect AMTrace data:
        1. Run AMTrace.exe (see KB86691) from an Administrator command prompt, providing the option to start immediately.
        2. Launch the McAfee process that is behaving unexpectedly.
        3. After it has reproduced the anomalous behavior, run AMTrace again, now providing the option to stop.
        4. Provide the created log to Technical Support to help with identifying the untrusted DLL.
    • When the adverse symptoms occur only Windows boot/startup, there are two data points to collect; they can be collected in serial or parallel:
      1. Collect Process Monitor data:
        1. Use Process Monitor, available from Microsoft.
        2. Run the tool as an Administrator.
        3. From the Options menu select Enable Boot Logging.
        4. Reboot the system.
        5. Log on and run Process Monitor again.
        6. Save the Procmon boot logging capture.
          IMPORTANT: Save All Events and use the Native PML format.
        7. Provide this information to Technical Support to help with identifying the third-party DLL being loaded by the McAfee process.
      2. Collect AMTrace data.
        1. Run AMTrace.exe (see KB86691) from an Administrator command prompt, providing the option to start on boot.
        2. Reboot the system.
        3. After it has reproduced the anomalous behavior, run AMTrace again, now providing the option to stop.
        4. Provide the created log to Technical Support to help with identifying the untrusted DLL.
Obtaining the .cer File:
  1. Access the properties of the EXE or DLL file (right-click and select Properties).
  2. Click the Digital Certificates tab.
NOTE: If this tab is missing, the file is not digitally signed and there is no option for us to trust this vendor's code – you must contact that vendor and request they provide signed code.
  1. Select a certificate from the Signature list that you want to trust.
  2. Click Details.
  3. Click View Certificate.
  4. Click the Details tab.
  5. Click Copy to File.
  6. In the Certificate Export Wizard, click Next.
  7. Click Next again to accept creating a .cer file.
  8. Specify a file name and location (for example, C:\Temp\My3rdParty.cer).
  9. Click Next and Finish.

Rate this document

Did this article resolve your issue?

Please provide any comments below

Beta Translate with

Select a desired language below to translate this page.

Glossary of Technical Terms

 Highlight Glossary Terms

Please take a moment to browse our Glossary of Technical Terms.