Loading...

Knowledge Center

Minimum Data Collection steps for troubleshooting Endpoint Security issues
Technical Articles ID:   KB86691
Last Modified:  11/18/2019
Rated:

Environment

McAfee Endpoint Security (ENS) Firewall 10.x
McAfee ENS Platform 10.x
McAfee ENS Threat Prevention 10.x
McAfee ENS Web Control 10.x

Summary

This article provides basic information about the Minimum Data Collection steps for troubleshooting common ENS issues.

Ensure that all logs are collected from the same system that experiences the issue, and that all logs are collected at the same time. The logging data time stamps can be used to troubleshoot the problem.

Mismatched logs from different systems, or logs collected at different times, can't be used for troubleshooting. Such logs might result in having to recollect all Minimum Data Collection logs.
 
IMPORTANT: The following files are required for Technical Support:
  • Minimum Escalation Requirements (MER) files with debug logging for ENS are required for all issues. For information about debug logging, see Verify whether ENS debug logging is enabled. For information about how to download the MER files for each McAfee product, see KB59385.
Collect other files and logs for each issue below: The following sections describe what data to collect for each type of issue.

Depending on the issue, the following tools might be required: See the following steps for instructions to prepare and use the tools.

  How to use AMTrace to collect logging data from AMCore:
  1. Prepare AMTrace:
    1. Download the zip package ENSDataCollect.zip from the Attachment section of this article.
    2. Extract the contents to the Desktop.
       
  2. Run AMTrace:
    1. Click Start, type cmd.exe in the Search bar, right-click cmd.exe from the list, and select Run as administrator. 
    2. When you are ready to start a trace, use the command option below, that the relevant data collection section requires.

      NOTE: The following are the paths to the AMTrace.exe file locations:
      • C:\Users\username\Desktop\ENSDataCollect\AMTracex86
      • C:\Users\username\Desktop\ENSDataCollect\AMTracex64
         
      AMTrace command options:
       
      • To use the AMTrace onboot option, run the following command:
         
        AMTrace.exe -b onboot -m 4GB
         
        This command instructs the tool to begin a trace at the next boot.
         
        NOTES: 
        • The "GB" is case sensitive. This example limits the log size to 4 GB. 10 MB is the minimum accepted value, and 2 GB is the default if not specified.
        • This option does not support the alternative logging modes described below.
         
      • To use AMTrace with the now option, run the following command:

        AMTrace.exe -b now -m 4GB

        IMPORTANT: AMTrace now uses the rollover option by default. 
         
        This command instructs the tool to begin a trace immediately, and to limit the log size to 4 GB per .etl file. When the log reaches 4 GB, a new log is created. Each log is appended with _1 or _2, and so on, until you stop the trace, the user logs off, or you shut down the system.

        NOTE: The "GB" is case sensitive. 
         
      • To use AMTrace without the rollover option:
        Choose an appropriate logging method for the issue you want to record:

        AMTrace.exe -b now -m 4GB -L stop
        AMTrace.exe -b now -m 4GB -L circular

        The stop mode creates a trace session that stops logging once reaching the size limit.
        The circular mode creates a trace session that logs to a single file. After reaching the maximum size, older events are overwritten.
         
        NOTE: The "L" and "GB" are case sensitive.
         
    3. Stop the trace and save the log. Use the following command:
       
      AMTrace -e
       
    4. When possible, AMTrace tries to automatically rename the resulting ETL files to include the start time and stop time of the logging in the file name. For example, amtrace_20200704.010203-010305.etl would indicate that logging began 2020-07-04 (July 4, 2020) at 1:02:03, and continued until 1:03:05.
      If AMTrace is unable to rename the file when logging stops, it is still possible to rename the file manually with another AMTrace command:

      AMTrace.exe --datestamp *.etl

      This command accepts wildcards (* or ?) to reference multiple characters or a single character respectively. This command renames the specified file or files to include the start and stop times in the file name. It does not affect files that already have the datestamp added.
To confirm whether an AMTrace is in progress, run the following command to list any active traces:

AMTrace -q

For a demonstration of how to collect AMTrace data with this procedure, watch the following video:



Back to top

How to use Windows Performance Recorder (WPR):
Run wprui.exe:
  1. Click Start, type cmd.exe in the Search bar, right-click cmd.exe from the list, and select Run as administrator. 
  2. Type wprui.exe and press Enter to start WPR.
     
    NOTE: If the program does not run or is not found, you must first install it. WPR is part of the Windows Performance Toolkit. It is available from the Windows SDK or the Windows Assessment and Deployment Kit:
  3. Choose to use a Performance Scenario and other settings, as recommended in the following table:
     
    Performance Issue
    Performance Scenario
    Detail Level
    Logging Mode
    		 Profiles to Include Number of Iterations
    Slow boot or logon
    Boot
    See below
    File
    		 First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity At least 1
    High CPU usage
    General
    See below
    File
    		 First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity N/A
    Application is slow or unresponsive
    		 General See below File First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity N/A
     
    WPR data is collected to allow for deeper analysis of the problem. The "Minifilter I/O Activity" option is one of the most important and is located under the "Scenario Analysis" section.

    The use of WPR places extra strain on the system, which can change or mask the original problem you want to investigate. Collect two data sets with different detail levels. Use the Light setting to show the issue, and the Verbose setting to allow a data set suitable for deeper analysis. Capture at least 30 seconds.

    When possible, collect a WPR log without the issue while you perform the same task, for comparative purposes. A data set without ENS present is needed to establish the benchmark for expected performance.  

For a demonstration of how to collect WPR data with this procedure, watch the following video:



Back to top

How to use Process Monitor:
  1. Prepare Process Monitor:
    1. Download Process Monitor from: https://technet.microsoft.com/en-us/library/bb896645.aspx.
    2. Extract Procmon.exe to the Desktop.
       
  2. Run Process Monitor:
    1. When you are ready to start Process Monitor, use the option below, that the relevant data collection section requires.
      • To immediately start Process Monitor:
        1. Run Procmon.exe and it automatically starts to capture process information.
        2. To stop Process Monitor, press Ctrl+E or click File and deselect Capture Events. Press Ctrl+E again to resume data collection.
        3. To save the log, click File, Save... (select All Events and use the native PML format).
           
      • To enable the Process Monitor boot logging option if required by the relevant data collection section:
        1. Open the Process Monitor console.
        2. Click Options.
        3. Click Enable Boot Logging.
        4. Click OK on the pop-up window. The next time a reboot occurs, a boot trace log is created.
        5. To save the log, run Process Monitor again and click File, Save... (select All Events and use the native PML format).

For a demonstration of how to collect Process Monitor data with this procedure, watch the following video:

Perform the steps in this section if the symptoms are any of the following:
  • Slow boot or startup
  • Slow logon
Data collection steps for AMTrace and Process Monitor:
  1. Start AMTrace with the onboot option.
  2. Start Process Monitor and enable the boot logging option.
  3. Reboot the system.
  4. Reproduce the issue.
  5. Log on to the system.
  6. Stop AMTrace and save the log.
  7. Open Process Monitor and save the boot log.
Data collection steps for Windows Performance Recorder:
  1. Run WPR.
  2. Configure the Boot Performance Scenario.
  3. Start the capture.
  4. Reboot the system.
  5. Reproduce the issue.
  6. Log on to the system.
  7. Allow the WPR: Boot Trace to finish.
  8. Capture the saved ETL files.
Perform the steps in this section if the symptoms are reproducible and are any of the following:
  • Slow application startup
  • Slow application performance
  • Slow system performance
Data collection steps for AMTrace and Process Monitor:
  1. Start Process Monitor.
  2. Start AMTrace with the now option.
  3. Reproduce the issue.
  4. Stop AMTrace and save the log.
  5. Stop Process Monitor and save the log.
Data collection steps for Windows Performance Recorder:
  1. Run WPR.
  2. Configure the General Performance Scenario.
  3. Start the trace.
  4. Reproduce the issue.
  5. Stop the trace.
  6. Capture the saved ETL file.
Back to top
Perform the steps in this section if the symptoms occur randomly and are any of the following:
  • Slow application startup
  • Slow application performance
  • Slow system performance
Data collection steps for AMTrace:
  1. Start AMTrace with the rollover option.
  2. When the issue occurs, stop AMTrace and save the log.
Data collection steps for Windows Performance Recorder (WPR):
  1. Run WPR.
  2. Configure the General Performance Scenario with Memory as the Logging Mode.
  3. Start the trace.
  4. Reproduce the issue.
  5. Save the trace, as soon as possible after you reproduce the issue.
  6. Capture the saved ETL file.
Back to top
Perform the steps in this section if the symptoms are any of the following:
  • System hang or deadlock
  • System BugCheck (blue screen)
Data collection steps for a system hang or deadlock:
  1. Configure the system to create a full memory.dmp. See KB56023.
  2. Configure the system to allow for a keyboard crash. See https://msdn.microsoft.com/en-us/library/windows/hardware/ff545499%28v=vs.85%29.aspx.
  3. Create the dump file when the issue occurs. Generally, the longer you can wait before you generate the dump file, the easier it is to identify the hang condition in the dump.
Data collection steps for a system BugCheck:
  1. Configure the system to create a full memory.dmp. See KB56023.
  2. Collect the full dump file when the system BugCheck (blue screen) occurs.
Data collection steps for FLTMC output:
  1. Open an administrative command prompt.
  2. Type fltmc.
  3. Collect the output from the fltmc command.
Perform the steps in this section if the symptoms are any of the following:
  • Application hang or deadlock (not responding and does not recover)
  • Application crash
Data collection steps for an application hang or deadlock:
  1. Download ProcDump from: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.
  2. Extract ProcDump to the Desktop.
  3. Open an administrative command prompt, and change directory to C:\Users\username\Desktop\Procdump.
  4. Run the following command: procdump -ma <process name>
  5. Collect the created dump file, which is located in the Procdump folder.
Data collection steps for an application crash:
  1. Download ProcDump from: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.
  2. Extract ProcDump to the Desktop.
  3. Open an administrative command prompt, and change directory to C:\Users\username\Desktop\Procdump.
  4. Run the following command: procdump -ma  -e <process name>
     
    NOTE: The -e switch instructs ProcDump to generate a dump the next time the process crashes.
     
  5. Wait for the process to crash again.
  6. Collect the created dump file, which is located in the Procdump folder.
Perform the steps in this section if the symptoms involve a User Mode or Application memory leak. Collect three (3) User Mode or Application crash dumps for analysis.
  1. Download ProcDump from: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.
  2. Extract ProcDump to the Desktop.
  3. Identify the process name that is leaking memory.
  4. Enable a stack trace on the leaking process. See KB91252.
  5. Wait for the suspect process to show high memory usage.
  6. Open an administrative command prompt, and change directory to C:\Users\username\Desktop\Procdump.
  7. Run the following command: procdump -ma <process name>
  8. Collect the created dump file, which is located in the Procdump folder.
  9. Repeat the steps and collect three (3) User Mode or Application crash dumps for analysis.
  10. Disable the stack trace on the process once all crash dump files are collected. See KB91252.
NOTE: Memory leak data collection needs to be performed over time, just as the memory leak will exhibit its behavior over time. For the best data collection results, it is recommended to reboot the system, and then start the following data collection. This sequence allows the data set to show the existence of the memory leak, over time, as it manifests on the system.

Perform the steps in this section if there is a suspected kernel memory leak involving a McAfee process.
  1. Familiarize yourself with Poolmon and Perfmon usage and configuration described in KB74951.
  2. Configure the system to create a full memory.dmp. See KB56023.
  3. Configure the system to allow for a keyboard crash. See https://msdn.microsoft.com/en-us/library/windows/hardware/ff545499%28v=vs.85%29.aspx.
  4. Reboot the system reported to show a memory leak.
  5. Using the configuration for Poolmon and Perfmon outlined in KB74951, start the Poolmon and Perfmon data collection.
  6. Wait for the system to show high memory usage.
  7. Stop Poolmon and Perfmon, and collect the resulting data.
  8. Force the system to perform a BugCheck while the high memory usage is still exhibited.
  9. Collect the memory dump.
  Back to top
Perform the steps in this section if the symptoms involve Device Guard or Credential Guard.
  1. Collect the appropriate ENS data for the experienced symptom, as outlined in this article.
  2. Also, collect an ETW (Event Tracing for Windows) trace with the following command, executed in an administrative command prompt:
     
    @echo off
    ECHO These commands enable tracing:
    @echo on
    logman create trace "base_DeviceGuard" -ow -o c:base_DeviceGuard.etl -p "Microsoft-Windows-DeviceGuard" 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 -ets
    @echo off
    echo
    ECHO Reproduce your issue and enter any key to stop tracing
    @echo on
    pause
    logman stop "base_DeviceGuard" -ets
    @echo off
    echo Tracing has been captured and saved successfully at c:base_DeviceGuard.etl
    pause
Back to top
Perform the data collection steps in this section if one or more components of ENS fail to install.

NOTE: Ensure that you collect the data during a local installation of ENS. Troubleshoot each module as a separate product.
  1. Download and unzip the standalone package from the Product Downloads site at: https://secure.mcafee.com/apps/downloads/my-products/login.aspx?region=us.
  2. Start Process Monitor.
  3. Start AMTrace with the rollover option.
  4. Re-create the issue. Run the local installation (setupEP.exe) as administrator and select the single module you are troubleshooting.
  5. Stop AMTrace and save the log.
  6. Stop Process Monitor and save the log.
  7. Collect a Minimum Escalation Requirements (MER) file (run as Administrator).
Back to top
Perform the steps in this section if the symptoms are any of the following:
  • The status of ENS is: Endpoint Security Platform is not running!
  • Third-party DLL injection
Data collection steps:
  1. Start Process Monitor.
  2. Start AMTrace with the now option.
  3. Open and close the ENS Console to re-create the issue.
  4. Stop AMTrace and save the log.
  5. Stop Process Monitor and save the log.
  6. Collect a MER file (run as Administrator).
  7. Export and collect a copy of the assigned Endpoint Security Common Options policy.
Back to top
Perform the steps in this section if the symptoms are related to TIE:
  1. Collect the appropriate data based on the symptoms outlined in this article.
  2. Also, collect the TIE Server log on the TIE Server appliance at /var/McAfee/tieserver/logs/tieserver.log.
Back to top
Perform the steps in KB90662 - How to troubleshoot an application or network traffic when using Endpoint Security Firewall.

Back to top

Solution

To contact Technical Support, log on to the ServicePortal and go to the Create a Service Request page at https://support.mcafee.com/ServicePortal/faces/serviceRequests/createSR:
  • If you are a registered user, type your User Id and Password, and then click Log In.
  • If you are not a registered user, click Register and complete the required fields. Your password and logon instructions will be emailed to you.

Attachment

ENSDataCollect.zip
321K • < 1 minute @ broadband


Rate this document

Glossary of Technical Terms


 Highlight Glossary Terms

Please take a moment to browse our Glossary of Technical Terms.