Minimum data collection steps for Endpoint Security issues

This article provides basic information about the Minimum Data Collection steps for troubleshooting common ENS issues.

Make sure that all logs are collected from the same system that experiences the issue, and that all logs are collected at the same time. The logging data time stamps can be used to troubleshoot the problem.

Mismatched logs from different systems, or logs collected at different times, can't be used for troubleshooting. Such logs might result in having to recollect all Minimum Data Collection logs.

IMPORTANT: The following files are needed for Technical Support:

Minimum Escalation Requirements (MER) files with debug logging for ENS are needed for all issues. For information about debug logging, see Verify whether ENS debug logging is enabled. For information about MER files, see: KB59385 - How to use MER tools with supported McAfee products. Debug logging must be enabled for the Real Protect RC.log to be generated.

Collect other files and logs for each issue below:

Slow boot or startup
Slow logon
Slow application startup (reproducible or random)
Slow application performance (reproducible or random)
Slow system performance (reproducible or random)
System hang or deadlock
System bug check (blue screen)
Application hang or deadlock (not responding and does not recover)
Application crash
Memory leak (user or kernel mode)
Issues related to Device Guard or Credential Guard
One or more ENS components fail to install
The status of ENS is: Endpoint Security Platform is not running!
Third-party DLL injection
Issues related to Threat Intelligence Exchange (TIE)
Issues related to ENS Firewall

Depending on the issue, you might need the following tools:

Tools

AMTrace is an internal tool to collect logging data from AMCore (last updated October 11, 2018). To use AMTrace to collect logging data from AMCore:
1. Prepare AMTrace:
  1. Download the zip package ENSDataCollect.zip from the Attachment section of this article.
  2. Extract the contents to the Desktop.
2. Run AMTrace:
  1. Click Start, type cmd.exe in the Search bar, right-click cmd.exe from the list, and click Run as administrator.
  2. When you are ready to start a trace, use the command option below, that the relevant data collection section requires.
    
    NOTE: The following are the paths to the AMTrace.exe file locations:
    - C:\Users\username\Desktop\ENSDataCollect\AMTracex86
    - C:\Users\username\Desktop\ENSDataCollect\AMTracex64
    AMTrace command options:
    - To use the AMTrace onboot option, run the following command:
      
      AMTrace.exe -b onboot -m 4GB
      
      This command instructs the tool to begin a trace at the next boot.
      NOTES:
      
      The "GB" is case sensitive. This example limits the log size to 4 GB. 10 MB is the minimum accepted value, and 2 GB is the default, if not specified.
      
      This option does not support the alternative logging modes described below.
    - To use AMTrace with the now option, run the following command:
      
      AMTrace.exe -b now -m 4GB
      
      IMPORTANT: The AMTrace now uses the rollover option by default.
      
      This command instructs the tool to begin a trace immediately, and to limit the log size to 4 GB per .etl file. When the log reaches 4 GB, a new log is created. Each log is appended with _1, _2, and so on, until you stop the trace, the user logs off, or you shut down the system.
      
      NOTE: The "GB" is case sensitive.
    - To use AMTrace without the rollover option:
      Choose an appropriate logging method for the issue you want to record:
      
      AMTrace.exe -b now -m 4GB -L stop
      AMTrace.exe -b now -m 4GB -L circular
      
      The stop mode creates a trace session that stops logging once reaching the size limit.
      The circular mode creates a trace session that logs to a single file. After reaching the maximum size, older events are overwritten.
      
      NOTE: The "L" and "GB" are case sensitive.
  3. Stop the trace and save the log. Use the following command:
    
    AMTrace -e
  4. When possible, AMTrace tries to automatically rename the resulting ETL files to include the start time and stop time of the logging in the file name. For example, amtrace_20200704.010203-010305.etl would indicate that logging began 2020-07-04 (July 4, 2020) at 1:02:03, and continued until 1:03:05.
    If AMTrace is unable to rename the file when logging stops, it is still possible to rename the file manually with another AMTrace command:
    
    AMTrace.exe --datestamp *.etl
    
    This command accepts wildcards (* or ?) to reference multiple characters or a single character respectively. This command renames the specified file or files to include the start and stop times in the file name. It does not affect files that already have the datestamp added.
To confirm whether an AMTrace is in progress, run the following command to list any active traces:

AMTrace -q

For a demonstration of how to collect AMTrace data with this procedure, watch the following video:
GFlags allows you to enable and disable advanced internal system diagnostic and troubleshooting features. You can run GFlags from a command prompt window or use its graphical user interface dialog box. It is most often used to turn on indicators that other tools track, count, and log.
MER collects event logs, file version details, files, process details, and registry details from McAfee products installed on your computer. Technical Support uses data collected to resolve problems. For more information, see: KB59385 - How to use MER tools with supported McAfee products.
Process Monitor is a tool from Microsoft that monitors and displays all file system activity on a Windows operating system in real time. Use the tool in system administration, computer forensics, and application debugging. How to use Process Monitor:
1. Prepare Process Monitor:
  1. Download Process Monitor from the Process Monitor downloads page.
  2. Extract Procmon.exe to the Desktop.
2. Run Process Monitor:
  1. When you are ready to start Process Monitor, use the option below, that the relevant data collection section requires.
    - To immediately start Process Monitor:
      1. Run Procmon.exe and it automatically starts to capture process information.
      2. To stop Process Monitor, press Ctrl+E or click File and deselect Capture Events. Press Ctrl+E again to resume data collection.
      3. To save the log, click File, Save... (select All Events and use the native PML format).
    - To enable the Process Monitor boot logging option if needed by the relevant data collection section:
      1. Open the Process Monitor console.
      2. Click Options.
      3. Click Enable Boot Logging.
      4. Click OK on the pop-up window. The next time a reboot occurs, a boot trace log is created.
      5. To save the log, run Process Monitor again and click File, Save... (select All Events and use the native PML format).
For a demonstration of how to collect Process Monitor data with this procedure, watch the following video:
Procdump is a command-line utility used to monitor an application for CPU spikes. It generates crash dumps during a spike that an administrator or developer can use to determine the cause of the spike.
Perfmon is a tool that administrators can use to examine how programs, running on their computers, affect the computer's performance. Use the tool in real time to analyze how running programs affect system performance. You can also use this tool to collect log file information for system performance data analysis later.
PoolMon displays data that the operating system collects about memory allocations from the system paged and nonpaged kernel pools, and memory pools used for Terminal Services sessions. The data is grouped by pool allocation tag. Microsoft Technical Support uses that information to find kernel mode memory leaks.
VMware converter is a free utility from VMware that helps convert Windows and Linux-based physical systems to VMware virtual machines. You can also use it to convert third-party image formats such as backup images and other virtual machines to VMware virtual machines. Use this tool to create virtual machines to provide to Technical Support for troubleshooting.
WinDbg is a Microsoft distributed debugger for the Windows operating system. Use this tool to debug user mode applications, device drivers, and the operating system itself in kernel mode. It has a graphical user interface and is more powerful than Visual Studio Debugger.

Windows Performance Recorder (WPR) is a performance recording tool from Microsoft that is based on Event Tracing for Windows (ETW). It records system events that you can then analyze by using Windows Performance Analyzer (WPA). How to use WPR:

Click Start, type cmd.exe in the Search bar, right-click cmd.exe from the list, and click Run as administrator.
Type wprui.exe and press Enter to start WPR.

NOTE: If the program does not run or is not found, you must first install it. WPR is part of the Windows Performance Toolkit. It is available from the Windows SDK or the Windows Assessment and Deployment Kit:
- For Windows SDK, see the Windows SDK downloads page.
- For Windows Assessment and Deployment Kit, see the Windows Assessment and Deployment Kit downloads page.

Choose to use a Performance Scenario and other settings, as recommended in the following table:

Performance Issue	Performance Scenario	Detail Level	Logging Mode	Profiles to Include	Number of Iterations
Slow boot or logon	Boot	See below	File	First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity	At least 1
High CPU usage	General	See below	File	First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity	N/A
Application is slow or unresponsive	General	See below	File	First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity	N/A

WPR data is collected to allow for deeper analysis of the problem. The "Minifilter I/O Activity" option is one of the most important and is located under the "Scenario Analysis" section.

The use of WPR places extra strain on the system, which can change or mask the original problem you want to investigate. Collect two data sets with different detail levels. Use the Light setting to show the issue, and the Verbose setting to allow a data set suitable for deeper analysis. Capture at least 30 seconds.

When possible, collect a WPR log without the issue while you perform the same task, for comparative purposes. A data set without ENS present is needed to establish the benchmark for expected performance.

For a demonstration of how to collect WPR data with this procedure, watch the following video:

The following sections describe what data to collect for each type of issue:

Verify whether ENS debug logging is enabled

For instructions, see: KB91797 - Enable debug logging to troubleshoot Endpoint Security issues.

Slow boot, startup, or logon

Perform the steps in this section if the symptoms are any of the following:

Slow boot or startup
Slow logon

Data collection steps for AMTrace and Process Monitor:

Start AMTrace with the onboot option.
Start Process Monitor and enable the boot logging option.
Reboot the system.
Reproduce the issue.
Log on to the system.
Stop AMTrace and save the log.
Open Process Monitor and save the boot log.

Data collection steps for Windows Performance Recorder:

Run WPR.
Configure the Boot Performance Scenario.
Start the capture.
Reboot the system.
Reproduce the issue.
Log on to the system.
Allow the WPR: Boot Trace to finish.
Capture the saved ETL files.

Slow performance (reproducible)

Perform the steps in this section if the symptoms are reproducible and are any of the following:

Slow application startup
Slow application performance
Slow system performance

Data collection steps for AMTrace and Process Monitor:

Start Process Monitor.
Start AMTrace with the now option.
Reproduce the issue.
Stop AMTrace and save the log.
Stop Process Monitor and save the log.

Data collection steps for Windows Performance Recorder:

Run WPR.
Configure the General Performance Scenario.
Start the trace.
Reproduce the issue.
Stop the trace.
Capture the saved ETL file.

Slow performance (random)

Perform the steps in this section if the symptoms occur randomly and are any of the following:

Slow application startup
Slow application performance
Slow system performance

Data collection steps for AMTrace:

Start AMTrace with the rollover option.
When the issue occurs, stop AMTrace and save the log.

Data collection steps for Windows Performance Recorder (WPR):

Run WPR.
Configure the General Performance Scenario with Memory as the Logging Mode.
Start the trace.
Reproduce the issue.
Save the trace, as soon as possible after you reproduce the issue.
Capture the saved ETL file.

System hang, deadlock, or bug check (blue screen)

Perform the steps in this section if the symptoms are any of the following:

System hang or deadlock
System bug check (blue screen)

Data collection steps for a system hang or deadlock:

Configure the system to create a full memory.dmp. See KB56023 - Create a memory dump for analysis by Technical Support.
Configure the system to allow for a keyboard crash. See this Microsoft article to force a keyboard crash.
Create the dump file when the issue occurs. Generally, the longer you can wait before you generate the dump file, the easier it is to identify the hang condition in the dump.

Data collection steps for a system bug check:

Configure the system to create a full memory.dmp. See KB56023 - Create a memory dump for analysis by Technical Support.
Collect the full dump file when the system bug check (blue screen) occurs.

Data collection steps for FLTMC output:

Open an administrative command prompt.
Type fltmc.
Collect the output from the fltmc command.

Application hang, deadlock, or crash

Perform the steps in this section if the symptoms are any of the following:

Application hang or deadlock (not responding and does not recover)
Application crash

Data collection steps for an application hang or deadlock:

Download ProcDump from the ProcDump downloads page.
Extract ProcDump to the Desktop.
Open an administrative command prompt, and change directory to C:\Users\username\Desktop\Procdump.
Run the following command: procdump -ma <process name>
Collect the created dump file, which is in the Procdump folder.

Data collection steps for an application crash:

If the crashing process is an ENS process, disable ENS Self-Protection.
Download ProcDump from the ProcDump downloads page.
Extract ProcDump to the Desktop.
Open an administrative command prompt, and change directory to C:\Users\username\Desktop\Procdump.
Run the following command: procdump -ma -e <process name>

NOTE: The -e switch instructs ProcDump to generate a dump the next time the process crashes.
Wait for the process to crash again.
Collect the created dump file, which is in the Procdump folder.
Re-enable ENS Self-Protection.

Memory leak (user or kernel)

Perform the steps in this section if the symptoms involve a User Mode or Application memory leak. Collect three (3) User Mode or Application crash dumps for analysis.

Download ProcDump from the ProcDump downloads page.
Extract ProcDump to the Desktop.
Identify the process name that is leaking memory.
Enable a stack trace on the leaking process. See KB91252 - How to enable a stack trace using the gflags.exe utility.
Wait for the suspect process to show high memory usage.
Open an administrative command prompt, and change directory to C:\Users\username\Desktop\Procdump.
Run the following command: procdump -ma <process name>
Collect the created dump file, which is in the Procdump folder.
Repeat the steps and collect three (3) User Mode or Application crash dumps for analysis.
Disable the stack trace on the process once all crash dump files are collected. See KB91252 - How to enable a stack trace using the gflags.exe utility.

NOTE: Memory leak data collection needs to be performed over time, just as the memory leak will exhibit its behavior over time. For the best data collection results, it is recommended to reboot the system, and then start the following data collection. This sequence allows the data set to show the existence of the memory leak, over time, as it manifests on the system.

Perform the steps in this section if there is a suspected kernel memory leak involving a McAfee process.

Familiarize yourself with Poolmon and Perfmon usage and configuration described in KB74951 - How to troubleshoot high memory use on systems.
Configure the system to create a full memory.dmp. See KB56023 - Create a memory dump for analysis by Technical Support.
Configure the system to allow for a keyboard crash. See this Microsoft article to force a keyboard crash.
Reboot the system reported to show a memory leak.
Use the configuration for Poolmon and Perfmon outlined in KB74951 - How to troubleshoot high memory use on systems. Start the Poolmon and Perfmon data collection.
Wait for the system to show high memory usage.
Stop Poolmon and Perfmon, and collect the resulting data.
Force the system to perform a bug check while the high memory usage is still exhibited.
Collect the memory dump.

Issues related to Device Guard or Credential Guard

Perform the steps in this section if the symptoms involve Device Guard or Credential Guard.

Collect the appropriate ENS data for the experienced symptom, as outlined in this article.
Also, collect an ETW (Event Tracing for Windows) trace with the following command, executed in an administrative command prompt:

@echo off
ECHO These commands enable tracing:
@echo on
logman create trace "base_DeviceGuard" -ow -o c:base_DeviceGuard.etl -p "Microsoft-Windows-DeviceGuard" 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 -ets
@echo off
echo
ECHO Reproduce your issue and enter any key to stop tracing
@echo on
pause
logman stop "base_DeviceGuard" -ets
@echo off
echo Tracing has been captured and saved successfully at c:base_DeviceGuard.etl
pause

ENS components fail to install

Perform the data collection steps in this section if one or more components of ENS fail to install.

NOTE: Make sure that you collect the data during a local installation of ENS. Troubleshoot each module as a separate product.

Download and unzip the standalone package from the Product Downloads site.
Start Process Monitor.
Start AMTrace with the rollover option.
Re-create the issue. Run the local installation (setupEP.exe) as administrator and select the single module you are troubleshooting.
Stop AMTrace and save the log.
Stop Process Monitor and save the log.
Collect a Minimum Escalation Requirements (MER) file (run as Administrator).

ENS is not running or third-party DLL injection

Perform the steps in this section if the symptoms are any of the following:

The status of ENS is: Endpoint Security Platform is not running!
Third-party DLL injection

Data collection steps:

Start Process Monitor.
Start AMTrace with the now option.
Open and close the ENS Console to re-create the issue.
Stop AMTrace and save the log.
Stop Process Monitor and save the log.
Collect a MER file (run as Administrator).
Export and collect a copy of the assigned Endpoint Security Common Options policy.

Issues that involve the Threat Intelligence Exchange (TIE) module

Perform the steps in this section if the symptoms are related to TIE:

Collect the appropriate data based on the symptoms outlined in this article.
Also, collect the TIE Server log on the TIE Server appliance at /var/McAfee/tieserver/logs/tieserver.log.

Issues that involve the ENS Firewall module

Perform the steps in KB90662 - Troubleshoot an application/network traffic when using ENS Firewall.

Back to top

Knowledge Center

Minimum data collection steps for Endpoint Security issues

Environment

Summary

Solution

Related Information

Attachment

Affected Products

Languages:

Glossary of Technical Terms

Title

Title

Knowledge Center

Minimum data collection steps for Endpoint Security issues

Environment

Summary

Solution

Related Information

Attachment

Affected Products

Languages:

Glossary of Technical Terms

Choose your region

Title

Title