1. Read the NSP Sensor CLI Guide for your software version. See docs.mcafee.com for more of these guides. 
2. Determine the answers to the following questions:
   1. When was the latency first noticed and were any unusual events seen on your network?
      To determine if any issues occurred at the same time as the latency, review your firewall, router, and switch logs. Resolve these issues. Then monitor your Sensor and confirm that the latency is resolved.
       
   2. Did you modify your network or Sensor configuration just before or during the time the latency was seen?
      If you reverse these changes, is the issue resolved?
       
   3. Determine what Sensor software is running on the Sensor. Is it the latest version? See KB55448 for version information.
      If the Sensor software is not the latest release, upgrade it to see if the issue resolves.
       
   4. How much network traffic is the Sensor scanning, and is it within the capacity of the Sensor?
      
      NOTE: A latency issue is often related to dropped packet issues. Often, when latency is seen on a Sensor, dropped packets also occur.

If you have reverted any changes, determined there were no exceptional network events taking place, and upgraded your Sensor software, perform the following steps:

1. Open a command-line session to the Sensor through Hyperterminal, PuTTY, or another application.
   
   NOTE: Start session logging in the connection application. If the issue is still present when your session has finished, save the logs. See your product documentation for steps to save the logs. 
    
2. If the Sensor is running software version 8.3 or later, gather the output of the show feature status command.
   For more information, see the Sensor CLI Guide for your software version.
    
3. Begin latency debugging by isolating the port pair that shows the latency. Type the following commands, either Sensor-wide or on the target interface, and press Enter after each one:
   
   show intfport
   
   For example, show intfport 1A/1B/etc. For more information about the show inftport command, see KB54660.
   
   show inlinepktdropstats
   
   For example, show inlinepktdropstats 1A/1B/etc. For more information about the show inlinepktdropstats command, see KB69806.
   
   Sample output below:
   intruShell@IPS-5200> show inlinepktdropstats g1/1
   IP Checksum Error Drop Count             : 0
   TCP Checksum Error Drop Count            : 0
   UDP Checksum Error Drop Count            : 0
   ICMP Checksum Error Drop Count           : 0
   ICMPv6 Checksum Error Drop Count         : 0
   ACL Drop Count                           : 0
   Out-Of-Context/Bad Packet Drop Count     : 0
   Cold Start Drop Count                    : 0
   Off/HdrLen Error Drop Count              : 0
   Attack Packet Drop Count                 : 0
   IP Reassembly Timeout Drop Count         : 0
   IPv6 Reassembly Timeout Drop Count       : 0
   TCP Out-Of-Order Timeout Drop Count      : 0
   TCP Protocol Error Drop Count            : 0
   UDP Protocol Error Drop Count            : 0
   ICMP Protocol Error Drop Count           : 0
   ICMPv6 Protocol Error Drop Count         : 0
   IP Protocol Error Drop Count             : 0
   IPv6 Protocol Error Drop Count           : 0
   System Out-of-Resource Drop Count        : 0
   Host Quarantine IPv4 Packet Drop Count   : 0
   Host Quarantine IPv6 Packet Drop Count   : 0
   Conn Limiting/L7ddos Packet Drop Count   : 0
   DoS Attack Packets Dropped               : 0
   Stateless ACL Drop Count                 : 0
   Total CRC Error Packets Dropped          : 0
   Total Other Layer-2 Error Packets Dropped: 0
   Total IP Spoofed Packets dropped         : 0
   Total IP No Credit Packets dropped       : 0
   
    
4. View the output of the commands in the previous step.
   1. If you see any CRC or other errors:
      1. Ensure that Speed/Duplex settings are configured the same on Sensor and Peer Devices.
         IMPORTANT:  Do not configure one device manually and set the other to auto negotiate; they must be configured identically.
          
      2. Change the connecting cables to ensure that the issue is not a bad cable. Use known good cables where possible.
          
   2. If you see a positive Host Quarantine Packet Drop Count:
      Check for Quarantined IPs in Manager and see if the affected traffic is one of the Quarantined hosts.
       
   3. If you see a high Conn Limiting/L7ddos Packet Drop value:
      Check your connection limiting policies and WebServer DoS under Inspection options policies.
       
   4. If you see a high DoS Attack Packets Dropped value:
      Check Attack logs to see if any DoS related attacks are triggering.
       
   5. If you see a Stateless ACL Drop Count:
      Check your Firewall Policies for ACL Rules as applicable
       
5. Put the Sensor into Layer2 mode. Type the following command and press Enter:
   
   layer2 mode assert
    
6. Reproduce the issue:
   • If you put the Sensor into Layer2 mode and it does not resolve your issue:
     The Sensor software is not causing the latency. When you put the Sensor in Layer2 mode, the Sensor acts like a hub and is invisible to the network. Because the Sensor software is eliminated as the cause, the issue is either with the sensor hardware, such as interfaces, cables, fail-open kits, or elsewhere.
     
     Try to use other known good interfaces, cables, or fail-open kits, or physically bypass the Sensor. If these actions do not resolve the issue, McAfee recommends that you use a network analyzer to inspect network traffic and troubleshoot the issue.
      
   • If the issue is resolved:
     1. Take the Sensor out of Layer2 mode:
        Type layer2 mode deassert and press Enter.
         
     2. Make sure that the Sensor is out of Layer2:
        Type status and press Enter.
        The correct status is:
        Layer 2 Status          : normal (IDS/IPS/NAC).
         
     3. Continue with the troubleshooting.
         
7. Set the Sensor to Loadbalance traffic before it is sent to the front-end processor:
   1. Type loadbalance pre-fe and press Enter.
   2. Make sure that the Sensor is load-balanced to pre-fe:
      Type loadbalance status and press Enter.
      The correct status is:
      Primary : Pre-FE
       
8. Set the Sensor to load balance traffic post the front-end processor:
   1. Type loadbalance post-fe and press Enter.
   2. Make sure that the Sensor is load-balanced to post-fe:
      Type loadbalance status and press Enter.
      The correct status is:
      Primary : Post-FE
       
9. Set Sensor load balancing to normal mode for further troubleshooting:
   1. Type loadbalance normal and press Enter.
   2. Make sure that Sensor load balancing is set to normal:
      Type loadbalance status and press Enter.
      The correct status is:
      Primary : (null)
       
10. Turn off Layer3 packet processing on the Sensor using the Sensor debug mode:
    1. Type the following command and press Enter:
       
       debug
        
    2. Type the following command and press Enter:
       
       set l3 off
        
11. Reproduce the issue and examine the outcome:
    If turning off Turn off Layer3 resolves your issue, it is related to Layer3 packet processing on the Sensor. Note this detail and include it when you open a service request. 
     
12. Turn Layer3 packet processing back on and continue troubleshooting. Type the following command and press Enter:
    
    set l3 on
     
13. Check whether IP fragmentation (layer3 feature) is the cause of the issue by disabling it:
    1. Type debug and press Enter.
    2. Type set ipfrag off and press Enter.
       • If turning off IP fragmentation resolves your issue, check your network for fragmented packets.
       • If no change, turn IP fragmentation back on and continue further troubleshooting:
         Type set ipfrag on and press Enter.
          
14. Turn off Layer7 packet processing and attack detection on the Sensor:
     
    1. Type the following command and press Enter:
       
       set l7 off
        
    2. Test if the latency and other issues are resolved or their impact reduced:
       • If the latency/issues are reduced or resolved, a specific Layer7 packet within the traffic causes the issue. If a certain protocol is suspected to be the cause, use ACLs to isolate Sensor Latency.
         
         Create an ACL that removes the suspected protocol from inspection. When an ACL is defined for a specific protocol, the Sensor ignores the protocol and the protocol is not inspected internally.
         
         See the IPS Configuration Guide for your software version for further information about ACL creation. 
         NOTE: You must re-enable Layer7 for ACLs to work. Type set l7 on and press Enter.
          
         • If a certain protocol is found to be the cause of the latency:
           1. Note the protocol and the originating server/host and destination.
           2. Gather a capture of this traffic if possible and include these details when you open your service request.
           3. Turn on Layer7 packet processing and attack detection, type the following command and press Enter:
              
              set l7 on
               
           4. Move to the next numbered step and continue troubleshooting.
               
         • If not found to be the cause of the latency:
           1. Record the ACLs you configured and protocols eliminated.
           2. Turn on Layer7 packet processing and attack detection, type the following command and press Enter:
              
              set l7 on
               
           3. Move to the next numbered step and continue troubleshooting
              
               
       • If the latency/issues are not reduced:
         Turn on Layer7 packet processing and attack detection on and continue troubleshooting. Type the following command and press Enter:
         
         set l7 on
          
15. Gather memory, Sensor load consumption, latency monitor, and performance debug level information, and details related to TCP, UDP, ICMP, and fragmented traffic.
     
    NOTE: Before you perform the following steps, enable the session logs in the package used to operate your command-line session (Hyperterminal or PuTTY).
     
    1. Type the following commands, and press Enter after each to gather the information:
       
       latency-monitor enable action alert-only or latency-monitor enable action layer2-forward 
       NOTE: Layer2 mode must be on for the layer2-forward action to take effect. You must run the command layer2 mode on to turn on Layer2 mode.
        
       latency-monitor status
        
       sensor perf-debug  
       NOTE: Make sure to set the time long enough to complete the following steps. For example, to enable perf-debug for 15 minutes: sensor perf-debug 15.
        
       sensor perf-debug status
        
       set sensor-load on
        
    2. Perform the following commands in order:
       
       IMPORTANT: You must run the entire set at least three times to gather sufficient data over time.
        
       show sensor-load
         
       sensor perf-debug show  
       NOTE: You must run the command within the time frame you specified when you executed sensor perf-debug.
        
       sensor-datapath-stat-analysis show
        
       datapathstat core all parameter all
        
       show tcpipstats
        
       show flows
        
       show mem-usage
        
       show intfport (for example, show intfport 1A/1B/etc)
        
       show inlinepktdropstats (for example, show inlinepktdropstats 1A/1B/etc)
        
       show statistics l4
        
       show statistics tcp
        
       show statistics udp
        
       show statistics icmp
        
       show statistics ipfrag
       
       bcmcli "ports"
       
       bcmcli "show counters"
       
       bcmcli "show counters all"
       
       bcmcli "ps"
       
       bcmcli "vlan show"
       
       bcmcli "trunk show"
       
       l7dpstat
       
       show datapath processunits
       
       show all datapath error-counters
       
       show statistics l4
       
       l7show
       
       show attack count
       
       show statistics alerts
       
       rspstat
       
       show fe stat
        
        
    3. After you run the above commands at least three times, run the following commands once:
        
       sensor perf-debug upload-protoStats
        
       set sensor-load off
       sensor perf-debug off
        
16. Exit debug mode. Type disable and press Enter.
     
17. Collect the SSH session logs and tracelogs. For information about how to collect a diagnostics trace from the Sensor (Trace.log), see KB55549.
    Supply both logs to Technical Support when you open a Service Request, and include the following details:
     
    • Did disabling layer 2, layer 3, or layer 7 processing have an effect?
    • Do you know if any protocols could be responsible for the latency?
    • What applications are affected by the latency?
    • Were any changes made to the affected network, applications, or Sensor around the time the latency started?
    • What previous software version was installed on the Sensor?
    • How long has the current software version been installed without causing latency?