How to troubleshoot Sensor latency issues
Technical Articles ID:
KB70861
Last Modified: 3/16/2022
Last Modified: 3/16/2022
Environment
Network Security Sensor Appliance
Summary
Before you begin:
- Read the NSP Sensor CLI Guide for your software version. See the Product Documentation site for this guide.
- Determine the answers to the following questions:
- When was the latency first noticed and were any unusual events seen on your network?
To determine if any issues occurred at the same time as the latency, review your firewall, router, and switch logs. Resolve these issues. Then, monitor your Sensor and confirm that the latency is resolved.
- Did you modify your network or Sensor configuration just before or during the time the latency was seen?
If you reverse these changes, is the issue resolved?
- Determine what Sensor software is running on the Sensor. Is it the latest version? For version information, see KB55448 - Current Network Security Platform release information
If the Sensor software isn’t the latest release, upgrade it to see if the issue resolves.
- How much network traffic is the Sensor scanning, and is it within the capacity of the Sensor?
NOTE: A latency issue is often related to dropped packet issues. Often, when latency is seen on a Sensor, dropped packets also occur.
- When was the latency first noticed and were any unusual events seen on your network?
- Open a command-line session to the Sensor through Hyperterminal, PuTTY, or another application.
NOTE: Start session logging in the connection application. If the issue is still present when your session has finished, save the logs. See your product documentation for steps to save the logs.
- If the Sensor is running software version 8.3 or later, gather the output of the show feature status command.
For more information, see the Sensor CLI Guide for your software version.
- Begin latency debugging by isolating the port pair that shows the latency. Type the following commands, either Sensor-wide or on the target interface, and press Enter after each one:
show intfport
For example,show intfport 1A/1B/etc . For more information about theshow inftport command, see KB54660 - Sensor command: show inftport
show inlinepktdropstats
For example,show inlinepktdropstats 1A/1B/etc . For more information about theshow inlinepktdropstats command, see KB69806 - Sensor Troubleshooting: Show inlinepktdropstats command
Sample output below:
intruShell@IPS-5200> show inlinepktdropstats g1/1
IP Checksum Error Drop Count : 0
TCP Checksum Error Drop Count : 0
UDP Checksum Error Drop Count : 0
ICMP Checksum Error Drop Count : 0
ICMPv6 Checksum Error Drop Count : 0
ACL Drop Count : 0
Out-Of-Context/Bad Packet Drop Count : 0
Cold Start Drop Count : 0
Off/HdrLen Error Drop Count : 0
Attack Packet Drop Count : 0
IP Reassembly Timeout Drop Count : 0
IPv6 Reassembly Timeout Drop Count : 0
TCP Out-Of-Order Timeout Drop Count : 0
TCP Protocol Error Drop Count : 0
UDP Protocol Error Drop Count : 0
ICMP Protocol Error Drop Count : 0
ICMPv6 Protocol Error Drop Count : 0
IP Protocol Error Drop Count : 0
IPv6 Protocol Error Drop Count : 0
System Out-of-Resource Drop Count : 0
Host Quarantine IPv4 Packet Drop Count : 0
Host Quarantine IPv6 Packet Drop Count : 0
Conn Limiting/L7ddos Packet Drop Count : 0
DoS Attack Packets Dropped : 0
Stateless ACL Drop Count : 0
Total CRC Error Packets Dropped : 0
Total Other Layer-2 Error Packets Dropped: 0
Total IP Spoofed Packets dropped : 0
Total IP No Credit Packets dropped : 0
- View the output of the commands in the previous step.
- If you see any CRC or other errors:
- Make sure that the Speed/Duplex settings are configured identically on Sensor and Peer Devices.
IMPORTANT: Don't configure one device manually and set the other to auto negotiate; they must be configured identically.
- Change the connecting cables to make sure that the issue isn't a bad cable. Use known good cables where possible.
- Make sure that the Speed/Duplex settings are configured identically on Sensor and Peer Devices.
- If you see a positive Host Quarantine Packet Drop Count:
Check for Quarantined IPs in Manager and see if the affected traffic is one of the Quarantined hosts.
- If you see a high
Conn Limiting/L7ddos Packet Drop value:
Check your connection limiting policies and WebServer DoS under Inspection options policies.
- If you see a high DoS Attack Packets Dropped value:
Check Attack logs to see if any DoS related attacks are triggering.
- If you see a Stateless ACL Drop Count:
Check your Firewall Policies for ACL Rules as applicable.
- If you see any CRC or other errors:
- Put the Sensor into Layer2 mode. Type the following command and press Enter:
layer2 mode assert
- Reproduce the issue:
- If you put the Sensor into Layer2 mode and it doesn't resolve your issue:
The Sensor software isn't causing the latency. When you put the Sensor in Layer2 mode, the Sensor acts like a hub and is invisible to the network. Because the Sensor software is eliminated as the cause, the issue is either with the sensor hardware or elsewhere. (Sensor hardware can include interfaces, cables, or fail-open kits.)
Try to use other known good interfaces, cables, or fail-open kits, or physically bypass the Sensor. If these actions don't resolve the issue, we recommend that you use a network analyzer to inspect network traffic and troubleshoot the issue.
- If the issue is resolved:
- Take the Sensor out of Layer2 mode:
Typelayer2 mode deassert and press Enter.
- Make sure that the Sensor is out of Layer2:
Typestatus and press Enter.
The correct status is:
Layer 2 Status : normal (IDS/IPS/NAC).
- Continue with the troubleshooting.
- Take the Sensor out of Layer2 mode:
- If you put the Sensor into Layer2 mode and it doesn't resolve your issue:
- Set the Sensor to Load balance traffic before it's sent to the front-end processor:
- Type
loadbalance pre-fe and press Enter. - Make sure that the Sensor is load-balanced to
pre-fe :
Typeloadbalance status and press Enter.
The correct status is:
Primary : Pre-FE
- Type
- Set the Sensor to load balance traffic post the front-end processor:
- Type
loadbalance post-fe and press Enter. - Make sure that the Sensor is load-balanced to
post-fe :
Typeloadbalance status and press Enter.
The correct status is:
Primary : Post-FE
- Type
- Set Sensor load balancing to the normal mode for further troubleshooting:
- Type
loadbalance normal and press Enter. - Make sure that Sensor load balancing is set to normal:
Typeloadbalance status and press Enter.
The correct status is:
Primary : (null)
- Type
- Turn off Layer3 packet processing on the Sensor using the Sensor debug mode:
- Type the following command and press Enter:
debug
- Type the following command and press Enter:
set l3 off
- Type the following command and press Enter:
- Reproduce the issue and examine the outcome:
If turning off Turn off Layer3 resolves your issue, it's related to Layer3 packet processing on the Sensor. Note this detail and include it when you open a service request.
- Turn Layer3 packet processing back on and continue troubleshooting. Type the following command and press Enter:
set l3 on - Check whether IP fragmentation (layer3 feature) is the cause of the issue by disabling it:
- Type
debug and press Enter. - Type
set ipfrag off and press Enter.- If turning off IP fragmentation resolves your issue, check your network for fragmented packets.
- If there's no change, turn IP fragmentation back on and continue further troubleshooting:
Typeset ipfrag on and press Enter.
- Type
- Turn off Layer7 packet processing and attack detection on the Sensor:
- Type the following command and press Enter:
set l7 off - Test if the latency and other issues are resolved or their impact is reduced:
- If the latency or issues are reduced or resolved, a specific Layer7 packet within the traffic causes the issue. If a certain protocol is suspected to be the cause, use ACLs to isolate Sensor Latency.
Create an ACL that removes the suspected protocol from inspection. When an ACL is defined for a specific protocol, the Sensor ignores the protocol and the protocol isn’t inspected internally.
See the IPS Configuration Guide for your software version for further information about ACL creation.
NOTE: You must re-enable Layer7 for ACLs to work. Typeset l7 on and press Enter.
- If a certain protocol is found to be the cause of the latency:
- Note the protocol and the originating server/host and destination.
- Gather a capture of this traffic if possible and include these details when you open your service request.
- Turn on Layer7 packet processing and attack detection, and type the following command and press Enter:
set l7 on - Move to the next numbered step and continue troubleshooting.
- If not found to be the cause of the latency:
- Record the ACLs you configured and protocols eliminated.
- Turn on Layer7 packet processing and attack detection, and type the following command and press Enter:
set l7 on - Move to the next numbered step and continue troubleshooting.
- If the latency/issues aren't reduced:
Turn on Layer7 packet processing and attack detection and continue troubleshooting. Type the following command and press Enter:
set l7 on
- If a certain protocol is found to be the cause of the latency:
- If the latency or issues are reduced or resolved, a specific Layer7 packet within the traffic causes the issue. If a certain protocol is suspected to be the cause, use ACLs to isolate Sensor Latency.
- Type the following command and press Enter:
- Gather memory, Sensor load consumption, latency monitor, and performance debug level information, and details related to TCP, UDP, ICMP, and fragmented traffic.
NOTE: Before you perform the following steps, enable the session logs in the package used to operate your command-line session (Hyperterminal or PuTTY).
- Type the following commands, and press Enter after each to gather the information:
latency-monitor enable action alert-only or latency-monitor enable action layer2-forward
NOTE: Layer2 mode must be on for the layer2-forward action to take effect. You must run the commandlayer2 mode on to turn on Layer2 mode.latency-monitor status sensor perf-debug
NOTE: Make sure to set the time long enough to complete the following steps. For example, to enable perf-debug for 15 minutes:sensor perf-debug 15 .sensor perf-debug status set sensor-load on - Perform the following commands in order:
IMPORTANT: You must run the entire set at least three times to gather sufficient data over time.show sensor-load sensor perf-debug show
NOTE: You must run the command within the time frame you specified when you executedsensor perf-debug .sensor-datapath-stat-analysis show datapathstat core all parameter all show tcpipstats show flows show mem-usage show intfport (for example,show intfport 1A/1B/etc )show inlinepktdropstats (for example,show inlinepktdropstats 1A/1B/etc )show statistics l4 show statistics tcp show statistics udp show statistics icmp show statistics ipfrag
bcmcli "ports"
bcmcli "show counters"
bcmcli "show counters all"
bcmcli "ps"
bcmcli "vlan show"
bcmcli "trunk show"
l7dpstat
show datapath processunits
show all datapath error-counters
show statistics l4
l7show
show attack count
show statistics alerts
rspstat
show fe stat - After you run the above commands at least three times, run the following commands once:
sensor perf-debug upload-protoStats
set sensor-load offsensor perf-debug off
- Type the following commands, and press Enter after each to gather the information:
- Exit debug mode. Type
disable and press Enter. - Collect the SSH session logs and
tracelogs . For information about how to collect a diagnostics trace from the Sensor (Trace.log), see KB55549 - How to collect a diagnostics trace from the Network Security Platform Sensor (Trace.log)
Supply both logs to Technical Support when you open a Service Request, and include the following details:- Did disabling layer 2, layer 3, or layer 7 processing have an effect?
- Do you know if any protocols might be responsible for the latency?
- What applications are affected by the latency?
- Were any changes made to the affected network, applications, or Sensor around the time the latency started?
- What previous software version was installed on the Sensor?
- How long has the current software version been installed without causing latency?
To contact Technical Support, go to the Create a Service Request page and log on to the ServicePortal.
- If you are a registered user, type your User ID and Password, and then click Log In.
- If you are not a registered user, click Register and complete the fields to have your password and instructions emailed to you.
Related Information
For product documents, go to the Product Documentation portal.
Affected Products
Languages:
This article is available in the following languages:
GermanEnglish United States
Spanish Spain
French
Italian
Japanese
Portuguese Brasileiro
Chinese Simplified
