Minimum data collection steps for Endpoint Security issues
Technical Articles ID:
KB86691
Last Modified: 9/17/2020
Last Modified: 9/17/2020
Environment
McAfee Endpoint Security (ENS) Adaptive Threat Protection (ATP) 10.x
McAfee ENS Firewall 10.x
McAfee ENS Platform 10.x
McAfee ENS Threat Prevention 10.x
McAfee ENS Web Control 10.x
McAfee ENS Firewall 10.x
McAfee ENS Platform 10.x
McAfee ENS Threat Prevention 10.x
McAfee ENS Web Control 10.x
Summary
This article provides basic information about the Minimum Data Collection steps for troubleshooting common ENS issues.
Make sure that all logs are collected from the same system that experiences the issue, and that all logs are collected at the same time. The logging data time stamps can be used to troubleshoot the problem.
Mismatched logs from different systems, or logs collected at different times, can't be used for troubleshooting. Such logs might result in having to recollect all Minimum Data Collection logs.
Depending on the issue, the following tools might be needed:
How to use AMTrace to collect logging data from AMCore:
AMTrace -q
For a demonstration of how to collect AMTrace data with this procedure, watch the following video:
Back to top
How to use Windows Performance Recorder (WPR):
Runwprui.exe :
For a demonstration of how to collect WPR data with this procedure, watch the following video:
Back to top
How to use Process Monitor:
For a demonstration of how to collect Process Monitor data with this procedure, watch the following video:
For instructions to enable and verify ENS debug logging, see KB91797.
Perform the steps in this section if the symptoms are any of the following:
Perform the steps in this section if the symptoms are any of the following:
Perform the steps in this section if the symptoms are any of the following:
Perform the steps in this section if there is a suspected kernel memory leak involving a McAfee process.
Perform the steps in this section if the symptoms involve Device Guard or Credential Guard.
Perform the data collection steps in this section if one or more components of ENS fail to install.
NOTE: Make sure that you collect the data during a local installation of ENS. Troubleshoot each module as a separate product.
Perform the steps in this section if the symptoms are any of the following:
Perform the steps in this section if the symptoms are related to TIE:
Perform the steps in KB90662 - How to troubleshoot an application or network traffic when using Endpoint Security Firewall.
Back to top
Make sure that all logs are collected from the same system that experiences the issue, and that all logs are collected at the same time. The logging data time stamps can be used to troubleshoot the problem.
Mismatched logs from different systems, or logs collected at different times, can't be used for troubleshooting. Such logs might result in having to recollect all Minimum Data Collection logs.
IMPORTANT: The following files are needed for Technical Support:
- Minimum Escalation Requirements (MER) files with debug logging for ENS are needed for all issues. For information about debug logging, see Verify whether ENS debug logging is enabled. For information about how to download the MER files for each McAfee product, see KB59385. Debug logging must be enabled for the Real Protect
RC.log to be generated.
- Slow boot or startup
- Slow logon
- Slow application startup (reproducible or random)
- Slow application performance (reproducible or random)
- Slow system performance (reproducible or random)
- System hang or deadlock
- System BugCheck (blue screen)
- Application hang or deadlock (not responding and does not recover)
- Application crash
- Memory leak (user or kernel mode)
- Issues related to Device Guard or Credential Guard
- One or more ENS components fail to install
- The status of ENS is: Endpoint Security Platform is not running!
- Third-party DLL injection
- Issues related to Threat Intelligence Exchange (TIE)
- Issues related to ENS Firewall
Depending on the issue, the following tools might be needed:
- AMTrace - an internal tool to collect logging data from AMCore (last updated October 11, 2018)
- Windows Performance Recorder - a tool from Microsoft
- Process Monitor - a tool from Microsoft
How to use AMTrace to collect logging data from AMCore:
- Prepare AMTrace:
- Download the zip package
ENSDataCollect.zip from the Attachment section of this article. - Extract the contents to the Desktop.
- Download the zip package
- Run AMTrace:
- Click Start, type
cmd.exe in the Search bar, right-clickcmd.exe from the list, and click Run as administrator. - When you are ready to start a trace, use the command option below, that the relevant data collection section requires.
NOTE: The following are the paths to theAMTrace.exe file locations:C:\Users\username\Desktop\ENSDataCollect\AMTracex86 C:\Users\username\Desktop\ENSDataCollect\AMTracex64
- To use the AMTrace
onboot option, run the following command:AMTrace.exe -b onboot -m 4GB This command instructs the tool to begin a trace at the next boot.
NOTES:- The "GB" is case sensitive. This example limits the log size to 4 GB. 10 MB is the minimum accepted value, and 2 GB is the default, if not specified.
- This option does not support the alternative logging modes described below.
- To use AMTrace with the now option, run the following command:
AMTrace.exe -b now -m 4GB
IMPORTANT: AMTrace now uses the rollover option by default.This command instructs the tool to begin a trace immediately, and to limit the log size to 4 GB per.etl file. When the log reaches 4 GB, a new log is created. Each log is appended with _1, _2, and so on, until you stop the trace, the user logs off, or you shut down the system.
NOTE: The "GB" is case sensitive. - To use AMTrace without the rollover option:
Choose an appropriate logging method for the issue you want to record:
AMTrace.exe -b now -m 4GB -L stop
AMTrace.exe -b now -m 4GB -L circular
The stop mode creates a trace session that stops logging once reaching the size limit.
The circular mode creates a trace session that logs to a single file. After reaching the maximum size, older events are overwritten.
NOTE: The "L" and "GB" are case sensitive.
- Stop the trace and save the log. Use the following command:
AMTrace -e
- When possible, AMTrace tries to automatically rename the resulting ETL files to include the start time and stop time of the logging in the file name. For example,
amtrace_20200704.010203-010305.etl would indicate that logging began 2020-07-04 (July 4, 2020) at 1:02:03, and continued until 1:03:05.
If AMTrace is unable to rename the file when logging stops, it is still possible to rename the file manually with another AMTrace command:
AMTrace.exe --datestamp *.etl
This command accepts wildcards (* or ?) to reference multiple characters or a single character respectively. This command renames the specified file or files to include the start and stop times in the file name. It does not affect files that already have the datestamp added.
- Click Start, type
For a demonstration of how to collect AMTrace data with this procedure, watch the following video:
Back to top
How to use Windows Performance Recorder (WPR):
Run
- Click Start, type
cmd.exe in the Search bar, right-clickcmd.exe from the list, and click Run as administrator. - Type
wprui.exe and press Enter to start WPR.NOTE: If the program does not run or is not found, you must first install it. WPR is part of the Windows Performance Toolkit. It is available from the Windows SDK or the Windows Assessment and Deployment Kit:- For Windows SDK, see https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk.
- For Windows Assessment and Deployment Kit, see https://msdn.microsoft.com/en-us/windows/hardware/commercialize/test/wpt/index.
- Choose to use a Performance Scenario and other settings, as recommended in the following table:
Performance IssuePerformance ScenarioDetail LevelLogging Mode
Profiles to Include Number of Iterations Slow boot or logonBootSee belowFileFirst-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity At least 1 High CPU usageGeneralSee belowFileFirst-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity N/A Application is slow or unresponsiveGeneral See below File First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity N/A WPR data is collected to allow for deeper analysis of the problem. The "Minifilter I/O Activity" option is one of the most important and is located under the "Scenario Analysis" section.
The use of WPR places extra strain on the system, which can change or mask the original problem you want to investigate. Collect two data sets with different detail levels. Use the Light setting to show the issue, and the Verbose setting to allow a data set suitable for deeper analysis. Capture at least 30 seconds.
When possible, collect a WPR log without the issue while you perform the same task, for comparative purposes. A data set without ENS present is needed to establish the benchmark for expected performance.
For a demonstration of how to collect WPR data with this procedure, watch the following video:
Back to top
How to use Process Monitor:
- Prepare Process Monitor:
- Download Process Monitor from: https://technet.microsoft.com/en-us/library/bb896645.aspx.
- Extract
Procmon.exe to the Desktop.
- Run Process Monitor:
- When you are ready to start Process Monitor, use the option below, that the relevant data collection section requires.
- To immediately start Process Monitor:
- Run
Procmon.exe and it automatically starts to capture process information. - To stop Process Monitor, press Ctrl+E or click File and deselect Capture Events. Press Ctrl+E again to resume data collection.
- To save the log, click File, Save... (select All Events and use the native PML format).
- Run
- To enable the Process Monitor boot logging option if required by the relevant data collection section:
- Open the Process Monitor console.
- Click Options.
- Click Enable Boot Logging.
- Click OK on the pop-up window. The next time a reboot occurs, a boot trace log is created.
- To save the log, run Process Monitor again and click File, Save... (select All Events and use the native PML format).
- To immediately start Process Monitor:
- When you are ready to start Process Monitor, use the option below, that the relevant data collection section requires.
For a demonstration of how to collect Process Monitor data with this procedure, watch the following video:
- Slow boot or startup
- Slow logon
- Start AMTrace with the
onboot option. - Start Process Monitor and enable the boot logging option.
- Reboot the system.
- Reproduce the issue.
- Log on to the system.
- Stop AMTrace and save the log.
- Open Process Monitor and save the boot log.
- Run WPR.
- Configure the Boot Performance Scenario.
- Start the capture.
- Reboot the system.
- Reproduce the issue.
- Log on to the system.
- Allow the WPR: Boot Trace to finish.
- Capture the saved ETL files.
Perform the steps in this section if the symptoms are reproducible and are any of the following:
- Slow application startup
- Slow application performance
- Slow system performance
- Start Process Monitor.
- Start AMTrace with the now option.
- Reproduce the issue.
- Stop AMTrace and save the log.
- Stop Process Monitor and save the log.
- Run WPR.
- Configure the General Performance Scenario.
- Start the trace.
- Reproduce the issue.
- Stop the trace.
- Capture the saved ETL file.
Perform the steps in this section if the symptoms occur randomly and are any of the following:
- Slow application startup
- Slow application performance
- Slow system performance
- Start AMTrace with the rollover option.
- When the issue occurs, stop AMTrace and save the log.
- Run WPR.
- Configure the General Performance Scenario with Memory as the Logging Mode.
- Start the trace.
- Reproduce the issue.
- Save the trace, as soon as possible after you reproduce the issue.
- Capture the saved ETL file.
- System hang or deadlock
- System BugCheck (blue screen)
-
Configure the system to create a full memory.dmp. See KB56023.
- Configure the system to allow for a keyboard crash. See https://msdn.microsoft.com/en-us/library/windows/hardware/ff545499%28v=vs.85%29.aspx.
- Create the dump file when the issue occurs. Generally, the longer you can wait before you generate the dump file, the easier it is to identify the hang condition in the dump.
- Configure the system to create a full memory.dmp. See KB56023.
- Collect the full dump file when the system BugCheck (blue screen) occurs.
- Open an administrative command prompt.
- Type
fltmc . - Collect the output from the
fltmc command.
- Application hang or deadlock (not responding and does not recover)
- Application crash
- Download ProcDump from: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.
- Extract ProcDump to the Desktop.
- Open an administrative command prompt, and change directory to
C:\Users\username\Desktop\Procdump . - Run the following command:
procdump -ma <process name> - Collect the created dump file, which is located in the
Procdump folder.
- Download
ProcDump from: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump. - Extract
ProcDump to the Desktop. - Open an administrative command prompt, and change directory to
C:\Users\username\Desktop\Procdump . - Run the following command:
procdump -ma -e <process name>NOTE: The -e switch instructsProcDump to generate a dump the next time the process crashes. - Wait for the process to crash again.
- Collect the created dump file, which is located in the
Procdump folder.
Perform the steps in this section if the symptoms involve a User Mode or Application memory leak. Collect three (3) User Mode or Application crash dumps for analysis.
- Download ProcDump from: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.
- Extract ProcDump to the Desktop.
- Identify the process name that is leaking memory.
- Enable a stack trace on the leaking process. See KB91252.
- Wait for the suspect process to show high memory usage.
- Open an administrative command prompt, and change directory to
C:\Users\username\Desktop\Procdump . - Run the following command:
procdump -ma <process name> - Collect the created dump file, which is located in the
Procdump folder. - Repeat the steps and collect three (3) User Mode or Application crash dumps for analysis.
- Disable the stack trace on the process once all crash dump files are collected. See KB91252.
Perform the steps in this section if there is a suspected kernel memory leak involving a McAfee process.
- Familiarize yourself with
Poolmon andPerfmon usage and configuration described in KB74951. - Configure the system to create a full memory.dmp. See KB56023.
- Configure the system to allow for a keyboard crash. See https://msdn.microsoft.com/en-us/library/windows/hardware/ff545499%28v=vs.85%29.aspx.
- Reboot the system reported to show a memory leak.
- Using the configuration for
Poolmon andPerfmon outlined in KB74951, start thePoolmon andPerfmon data collection. - Wait for the system to show high memory usage.
- Stop
Poolmon andPerfmon , and collect the resulting data. - Force the system to perform a BugCheck while the high memory usage is still exhibited.
- Collect the memory dump.
- Collect the appropriate ENS data for the experienced symptom, as outlined in this article.
- Also, collect an ETW (Event Tracing for Windows) trace with the following command, executed in an administrative command prompt:
@echo off
ECHO These commands enable tracing:
@echo on
logman create trace "base_DeviceGuard" -ow -o c:base_DeviceGuard.etl -p "Microsoft-Windows-DeviceGuard" 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 -ets
@echo off
echo
ECHO Reproduce your issue and enter any key to stop tracing
@echo on
pause
logman stop "base_DeviceGuard" -ets
@echo off
echo Tracing has been captured and saved successfully at c:base_DeviceGuard.etl
pause
NOTE: Make sure that you collect the data during a local installation of ENS. Troubleshoot each module as a separate product.
- Download and unzip the standalone package from the Product Downloads site at: https://secure.mcafee.com/apps/downloads/my-products/login.aspx?region=us.
- Start Process Monitor.
- Start AMTrace with the rollover option.
- Re-create the issue. Run the local installation (
setupEP.exe ) as administrator and select the single module you are troubleshooting. - Stop AMTrace and save the log.
- Stop Process Monitor and save the log.
- Collect a Minimum Escalation Requirements (MER) file (run as Administrator).
- The status of ENS is: Endpoint Security Platform is not running!
- Third-party DLL injection
- Start Process Monitor.
- Start AMTrace with the now option.
- Open and close the ENS Console to re-create the issue.
- Stop AMTrace and save the log.
- Stop Process Monitor and save the log.
- Collect a MER file (run as Administrator).
- Export and collect a copy of the assigned Endpoint Security Common Options policy.
- Collect the appropriate data based on the symptoms outlined in this article.
- Also, collect the TIE Server log on the TIE Server appliance at /var/McAfee/tieserver/logs/tieserver.log.
Back to top
Solution
To contact Technical Support, log on to the ServicePortal and go to the Create a Service Request page at https://support.mcafee.com/ServicePortal/faces/serviceRequests/createSR:
- If you are a registered user, type your User Id and Password, and then click Log In.
- If you are not a registered user, click Register and complete the required fields. Your password and logon instructions will be emailed to you.
Related Information
McAfee Self-Service Supportability (SSO) Orchestrator:
SSS Orchestrator is a data collection tool. It brings the above mentioned supportability tools under a single orchestrator. The tool invokes the right tool at the right time, and eases the data collection effort. This tool captures the context in which the data collection happens, which helps report proper telemetry data. For more information, see KB92519 - Self-Service Supportability Orchestrator data collection tool.
SSS Orchestrator is a data collection tool. It brings the above mentioned supportability tools under a single orchestrator. The tool invokes the right tool at the right time, and eases the data collection effort. This tool captures the context in which the data collection happens, which helps report proper telemetry data. For more information, see KB92519 - Self-Service Supportability Orchestrator data collection tool.
Attachment
Affected Products
Diagnostic Data Collection
Endpoint Security Adaptive Threat Protection
Endpoint Security Firewall 10.7.x
Endpoint Security Firewall 10.6.x
Endpoint Security Threat Prevention 10.7.x
Endpoint Security Threat Prevention 10.6.x
Endpoint Security Web Control 10.7.x
Endpoint Security Web Control 10.6.x
Troubleshooting
Endpoint Security Adaptive Threat Protection
Endpoint Security Firewall 10.7.x
Endpoint Security Firewall 10.6.x
Endpoint Security Threat Prevention 10.7.x
Endpoint Security Threat Prevention 10.6.x
Endpoint Security Web Control 10.7.x
Endpoint Security Web Control 10.6.x
Troubleshooting
Languages:
This article is available in the following languages:
GermanEnglish United States
Spanish Spain
French
Italian
Japanese
Portuguese Brasileiro
Chinese Simplified
