Cisco ASA Packet Drop Troubleshooting


As a firewall, the Cisco ASA drops packets. That’s great until it drops packets that you want to permit, and you have no idea what is going on. Fortunately, the ASA supports different tools to show you why and what packets it drops.

In this lesson, we’ll cover the following tools:

Here is the topology I use:

H1 Asa1 H2 Packet Drop Topology

We use a simple topology:

  • Two “host” devices:
    • These are routers with ip routing disabled, and they use ASA1 as their default gateway.
    • An HTTP server is enabled on both devices.
  • ASA1 runs ASAv Version 9.9(2) and has two interfaces:
    • INSIDE: security level 100
    • OUTSIDE: security level 0

In this topology, H1 will be able to initiate a connection to H2. H2 won’t be able to initiate a connection to H1 because we go from a low-security level (0) to a high-security level (100).

hostname ASA1
!
interface GigabitEthernet0/0
 nameif INSIDE
 security-level 100
 ip address 192.168.1.254 255.255.255.0 
!
interface GigabitEthernet0/1
 nameif OUTSIDE
 security-level 0
 ip address 192.168.2.254 255.255.255.0
!
class-map inspection_default
 match default-inspection-traffic
!
policy-map global_policy
 class inspection_default
  inspect ip-options 
  inspect netbios 
  inspect rtsp 
  inspect sunrpc 
  inspect tftp 
  inspect xdmcp 
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect esmtp 
  inspect sqlnet 
  inspect sip  
  inspect skinny  
!
service-policy global_policy global
!
: end
hostname H1
!
no ip routing
!         
interface GigabitEthernet0/1
 ip address 192.168.1.1 255.255.255.0
!
ip default-gateway 192.168.1.254
!
ip http server
no ip http secure-server
!
end
hostname H2
!
no ip routing
!         
interface GigabitEthernet0/1
 ip address 192.168.2.2 255.255.255.0
!
ip default-gateway 192.168.2.254
!
ip http server
no ip http secure-server
!
end

Let’s get started!

1.Connection State

The ASA keeps track of all existing connections and their state. Let’s see what this looks like. First, we connect from H1 to H2:

H1#telnet 192.168.2.2 80
Trying 192.168.2.2, 80 ... Open

Now look at the connections with the show conn command:

ASA1# show conn 
1 in use, 1 most used

TCP OUTSIDE  192.168.2.2:80 INSIDE  192.168.1.1:50195, idle 0:00:00, bytes 0, flags U

You can see the flags if you add the detail parameter:

ASA1# show conn detail 
0 in use, 1 most used
Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
       B - initial SYN from outside, b - TCP state-bypass or nailed,
       C - CTIQBE media, c - cluster centralized,
       D - DNS, d - dump, E - outside back connection, e - semi-distributed,
       F - outside FIN, f - inside FIN,
       G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
       i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
       k - Skinny media, L - LISP triggered flow owner mobility
       l - local director/backup stub flow
       M - SMTP data, m - SIP media, n - GUP
       N - inspected by Snort
       O - outbound data, o - offloaded,
       P - inside back connection,
       Q - Diameter, q - SQL*Net data,
       R - outside acknowledged FIN,
       R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
       s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, u - STUN,
       V - VPN orphan, v - M3UA W - WAAS,
       w - secondary domain backup,
       X - inspected by service module,
       x - per session, Y - director stub flow, y - backup stub flow,
       Z - Scansafe redirection, z - forwarding stub flow

Let’s try to connect from H2 to H1:

H2#telnet 192.168.1.1 80
Trying 192.168.1.1, 80 ... 
% Connection timed out; remote host not responding

This connection fails, so it doesn’t show up in the connection overview:

ASA1# show conn
0 in use, 1 most used

At least we know something is going on.

2.Interface drops

The ASA keeps track of drops on the interface. Here’s where you find this:

ASA1# show interface GigabitEthernet 0/1 | include packets dropped
        10 packets dropped

We see the ASA drops packets on the interface, but we have no idea what. You can use clear interface to reset this counter.

3.Syslog

Syslog is one of the best tools to figure out what is going on with your packet drops. The ASA has over 2000 unique syslog messages. You can send syslog messages to different locations. For debugging, the console works fine. Let’s see how we can use this.

First, you need to enable logging up to the debug level:

ASA1(config)# logging enable
ASA1(config)# logging buffered debugging

Optionally, you can increase the buffer size so you can store more syslog messages. For this example, I’m disabling the timestamps so that the messages are easier to look at:

ASA1(config)# logging buffer-size 1000000
ASA1(config)# no logging timestamp

Let’s try to connect from H2 to H1 on TCP port 80:

H2#telnet 192.168.1.1 80
Trying 192.168.1.1, 80 ... 
% Connection timed out; remote host not responding

Now check out your log with the show logging command:

ASA1(config)# show logging
Syslog logging: enabled
    Facility: 20
    Timestamp logging: disabled
    Hide Username logging: enabled
    Standby logging: disabled
    Debug-trace logging: disabled
    Console logging: disabled
    Monitor logging: disabled
    Buffer logging: level debugging, 11 messages logged
    Trap logging: disabled
    Permit-hostdown logging: disabled
    History logging: disabled
    Device ID: disabled
    Mail logging: disabled
    ASDM logging: disabled
%ASA-5-111008: User 'enable_15' executed the 'logging buffer-size 1000000' command.
%ASA-5-111010: User 'enable_15', running 'CLI' from IP 0.0.0.0, executed 'logging buffer-size 1000000'
%ASA-5-111008: User 'enable_15' executed the 'no logging timestamp' command.
%ASA-5-111010: User 'enable_15', running 'CLI' from IP 0.0.0.0, executed 'no logging timestamp'
%ASA-7-111009: User 'enable_15' executed cmd: show logging
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE

Above, we see the syslog messages and the code. If you want to know what each message means, you have to check the Cisco ASA Series Syslog Messages. In this case, we see message code 106001. Here is the explanation for this message, taken directly from Cisco.com:

Explanation An attempt was made to connect to an inside address is denied by the security policy that is defined for the specified traffic type. The IP address displayed is the real IP address instead of the IP address that appears through NAT. Possible tcp_flags values correspond to the flags in the TCP header that were present when the connection was denied. For example, a TCP packet arrived for which no connection state exists in the ASA, and it was dropped. The tcp_flags in this packet are FIN and ACK.

When there is much traffic going on, you’ll need to filter these messages. You can either use include to filter the message:

ASA1(config)# show logging | include 106001
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE

Or get rid of what you don’t need with exclude:

ASA1(config)# show logging | exclude 111008|111009|111010|302010
Syslog logging: enabled
    Facility: 20
    Timestamp logging: disabled
    Hide Username logging: enabled
    Standby logging: disabled
    Debug-trace logging: disabled
    Console logging: disabled
    Monitor logging: disabled
    Buffer logging: level debugging, 19 messages logged
    Trap logging: disabled
    Permit-hostdown logging: disabled
    History logging: disabled
    Device ID: disabled
    Mail logging: disabled
    ASDM logging: disabled
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE
%ASA-2-106001: Inbound TCP connection denied from 192.168.2.2/13279 to 192.168.1.1/80 flags SYN  on interface OUTSIDE

These syslog messages are handy to figure out what is going on.

4.ASP drops

Another useful tool is to check the Accelerated Security Path (ASP) drops with the show asp drop command. This command gives an overview of packets that the ASA drops with a reason. Let’s have a look:

ASA1# show asp drop

Frame drop:
  Flow is denied by configured rule (acl-drop)                                 3

Last clearing: 12:12:46 UTC Apr 24 2020 by enable_15

Flow drop:

Last clearing: 12:12:46 UTC Apr 24 2020 by enable_15

Above, we see 3 hits because of “acl-drop”. You can reset this counter with the clear asp drop command if needed. There is a huge list of reasons, which you can find on the Cisco “show asp drop” command reference page.

5.Packet Capture

We can also capture packets to take a closer look. There are two options:

  • Capture ASP dropped packets
  • Capture any packets you want.

Let’s check both options.

5.1.ASP Drops Capture

The show asp drop command tells us why something is dropped with a counter, but that’s it. It doesn’t tell us exactly what is dropped. Let’s capture some packets so we can see them. We do this with the capture command:

ASA1(config)# capture ASP_DROPS type asp-drop acl-drop

The command above supports some extra parameters. For example, you could capture only specific protocol numbers (AH, ESP, GRE, etc.) or add an access-list. If you have a lot of traffic, you probably want a specific capture, but in my case, this is fine.

Let’s generate some more traffic from H2 to H1:

H2#telnet 192.168.1.1 80
Trying 192.168.1.1, 80 ... 
% Connection timed out; remote host not responding

Here are the packets we captured:

ASA1(config)# show capture ASP_DROPS

4 packets captured

   1: 12:16:18.018919       192.168.2.2.30833 > 192.168.1.1.80: S 178607441:178607441(0) win 4128  Drop-reason: (acl-drop) Flow is denied by configured rule

   2: 12:16:20.021666       192.168.2.2.30833 > 192.168.1.1.80: S 178607441:178607441(0) win 4128  Drop-reason: (acl-drop) Flow is denied by configured rule

   3: 12:16:24.028456       192.168.2.2.30833 > 192.168.1.1.80: S 178607441:178607441(0) win 4128  Drop-reason: (acl-drop) Flow is denied by configured rule

   4: 12:16:32.034986       192.168.2.2.30833 > 192.168.1.1.80: S 178607441:178607441(0) win 4128  Drop-reason: (acl-drop) Flow is denied by configured rule

4 packets shown

The above output is interesting. This output tells us which packets get dropped with the reason. When you are finished, don’t forget to get rid of the capture:

ASA1(config)# no capture ASP_DROPS

5.2.Regular capture

The capture above is useful, but it can also be useful to do a “regular” capture to see all packets. You can combine this with an access-list too. For example, let’s say I want to see all traffic between H1 and H2. I can create an access-list for this traffic and combine it with a capture:

ASA1(config)# access-list H1_H2 permit ip host 192.168.1.1 host 192.168.2.2
ASA1(config)# access-list H1_H2 permit ip host 192.168.2.2 host 192.168.1.1

ASA1(config)# capture MY_CAPTURE interface OUTSIDE access-list H1_H2

Now generate some traffic:

H2#telnet 192.168.1.1 80
Trying 192.168.1.1, 80 ... 
% Connection timed out; remote host not responding

Now we can see if our capture actually captured something:

ASA1(config)# show capture
capture MY_CAPTURE type raw-data access-list H1_H2 interface OUTSIDE [Capturing - 296 bytes]

This seems to be the case. Let’s check the packets:

ASA1(config)# show capture MY_CAPTURE

4 packets captured

   1: 12:21:05.793080       192.168.2.2.55073 > 192.168.1.1.80: S 567620225:567620225(0) win 4128  
   2: 12:21:07.796620       192.168.2.2.55073 > 192.168.1.1.80: S 567620225:567620225(0) win 4128  
   3: 12:21:11.804859       192.168.2.2.55073 > 192.168.1.1.80: S 567620225:567620225(0) win 4128  
   4: 12:21:19.811878       192.168.2.2.55073 > 192.168.1.1.80: S 567620225:567620225(0) win 4128  
4 packets shown

The output above is different from the ASP drop capture and could prove useful when you try to troubleshoot a problem.

6.Packet Tracer

Packet tracer is useful to see what actions the ASA applies to a packet. There are two options:

  • Crafted packet: You can enter any source or destination IP address and port numbers.
  • Captured packet: You can use a captured packet as the input.

Let’s try both options.

6.1.Crafted Packet

Let’s try the packet-tracer command to see how the ASA deals with traffic from H2 to H1. I set a random source port and destination TCP port 80:

ASA1(config)# packet-tracer input OUTSIDE tcp 192.168.2.2 54321 192.168.1.1 80

Phase: 1
Type: CAPTURE
Subtype: 
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.1.1 using egress ifc  INSIDE

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: ACCESS-LIST
Subtype: 
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: OUTSIDE
input-status: up
input-line-status: up
output-interface: INSIDE
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

This output shows us everything the ASA does with this packet. For example, it also shows when it applies NAT, whether the packet gets encrypted with IPSec, etc.

6.2.Captured Packet

We can also use a captured packet as the input for packet-tracer. Let’s try one more capture. This time we add the trace parameter:

ASA1(config)# capture MY_CAPTURE interface OUTSIDE access-list H1_H2 trace

And generate some traffic:

H2#telnet 192.168.1.1 80

Let’s see if we captured anything:

ASA1(config)# show capture MY_CAPTURE

1 packet captured

   1: 13:05:03.345624       192.168.2.2.48877 > 192.168.1.1.80: S 2760169659:2760169659(0) win 4128  
1 packet shown

We captured one packet. Above, you can see the packet number. Let’s see what actions the ASA applies to this packet:

ASA1(config)# show capture MY_CAPTURE trace packet-number 1

4 packets captured

   1: 13:05:03.345624       192.168.2.2.48877 > 192.168.1.1.80: S 2760169659:2760169659(0) win 4128  
Phase: 1
Type: CAPTURE
Subtype: 
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW 
Config:
Additional Information:
found next-hop 192.168.1.1 using egress ifc  INSIDE

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: ACCESS-LIST
Subtype: 
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: OUTSIDE
input-status: up
input-line-status: up
output-interface: INSIDE
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule


1 packet shown

We now see the packet-tracer output for the captured packet.

7.Service-Policy

Another reason why your ASA drops packets could be the service-policy you use. With the show service-policy command, we can see what packets match the inspection, and what packets are dropped. Here is an example:

ASA1# show service-policy

Global policy: 
  Service-policy: global_policy
    Class-map: inspection_default
      Inspect: ip-options _default_ip_options_map, packet 0, lock fail 0, drop 0, reset-drop 0, 5-min-pkt-rate 0 pkts/sec, v6-fail-close 0 sctp-drop-override 0
      Inspect: netbios, packet 0, lock fail 0, drop 0, reset-drop 0, 5-min-pkt-rate 0 pkts/sec, v6-fail-close 0 sctp-drop-override 0
[output omitted]

Above, you see that the ASA didn’t drop any of the traffic that it inspected. You can also use the flow parameter to see how the ASA inspects specific traffic. For example, between H1 and H2:

ASA1# show service-policy flow tcp host 192.168.1.1 host 192.168.2.2 eq 80

Global policy: 
  Service-policy: global_policy
    Class-map: class-default
      Match: any
      Action:

This ASA uses the default service-policy, but if you made any changes to your service-policies or class-maps, this is something you might want to check.

8.Conclusion

We covered different options you can use to troubleshoot packet drops on your Cisco ASA:

  • Connection State: Use the show conn command to see current connections and their state.
  • Interface drops: Use the show interface command to see if your ASA drops any packets on the interface.
  • Syslog: See exactly what happens with your packets.
  • ASP drops: See the reason and a counter of packet drops.
  • Packet captures: Capture the packets to take a closer look.
  • Packet tracer: See what actions the ASA applies to your packets.
  • Service-policy: Look at the ASA inspection rules and packet drops.

I hope you enjoyed this lesson. If you have any questions feel free to leave a comment!

Comments

Post a Comment

Popular posts from this blog

show asp drop Command Usage

The  show asp drop  command shows the packets or connections dropped by the accelerated security path, which might help you troubleshoot a problem. See the general operations configuration guide for more information about the accelerated security path. This information is used for debugging purposes only, and the information output is subject to change. Consult Cisco TAC to help you debug your system with this command. The following sections include each drop reason name and description, including recommendations: rame Drop Reasons   ---------------------------------------------------------------- Name: natt-keepalive NAT-T keepalive message: This counter will increment when the appliance receives an IPSec NAT-T keepalive message. NAT-T keepalive messages are sent from the IPSec peer to the appliance to keep NAT/PAT flow information current in network devices between the NAT-T IPSec peer and the appliance.   Recommendation: If you have configured IPSec NAT-T on your ...
Read more

1.Cisco ASA Clock Configuration

The Cisco ASA firewall has a battery on the motherboard that saves the clock settings. Even when it’s is powered off, the clock will be stored. There are two important reasons why you want to make sure that your ASA has the correct date/time: In case of a security breach you want to track log files for events. With an incorrect timestamp, your log files are useless. PKI (Public Key Infrastructure) that we use for digital certificates to authenticate remote users (IPSEC or SSL VPN) requires the correct date/time. The most simple method is to configure the date/time manually, you can do it like this: ASA1(config)# clock set 13:15:00 Dec 19 2014 Just use the  clock set  command and enter the correct time/date. You can verify it like this: ASA1# show clock 13:15:15.709 UTC Fri Dec 19 2014 As you can see, the default timezone is UTC. If you are in another timezone like me then you have to change this: ASA1(config)# clock timezone CET +1 Use the  clock timezone  command t...
Read more