Firewall Manager -- High ping latency -- High cpu ksoftirqd -- cpu 100% -- packet loss -- High iptables rule number

[francis.brosnan]

1. Introduction

This article describes how to solve an scenario where a firewall based on iptables, running with #core-admin-en and using Core-Admin FirewallManager and IpBlocker tool, where firewall shows poor performance.

2. Symtoms

1. Linux server running FirewallManager and Ipblocker tool, based on iptables that shows ksoftirqd processes consuming high or 100% cpu.

2. Checking “dropped” indicators at network card interface, they keep on growing:

   >> ifconfig -a | grep dropped
   RX packets:55169628049 errors:0 dropped:30255399 overruns:0 frame:0
   TX packets:43640984422 errors:0 dropped:0 overruns:0 carrier:0
   RX packets:0 errors:0 dropped:0 overruns:0 frame:0
   TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
   RX packets:43184969750 errors:0 dropped:121487666 overruns:0 frame:0
   TX packets:51691961677 errors:0 dropped:0 overruns:0 carrier:0
   RX packets:2126364061 errors:0 dropped:4388060 overruns:0 frame:0
   TX packets:3374447258 errors:0 dropped:0 overruns:0 carrier:0
   RX packets:62047417 errors:0 dropped:0 overruns:0 frame:0
   TX packets:62047417 errors:0 dropped:0 overruns:0 carrier:0

3. EthTool shows no buffer rx available by increasing in a constant manner its count reporting:

   >> ethtool -S eth0 | grep buffer
   rx_no_buffer_count: 144683278
   >> ethtool -S eth0 | grep buffer
   rx_no_buffer_count: 144683377

4. You have a high number of iptables rules, above 20000:

   >> iptables -S | wc -l
   49560

5. You also have a high number of crad-ip-blocker.pyc rules that are causing that high number of iptables rules:

   >> crad-ip-blocker.pyc -l | grep 15612

6. You see a lot of interruptions at /proc/interrupts, especially associated to a network card (IR-PCI-MSI-edge ethX) and Local timer interrupts:

   >> cat /proc/interrupts

7. At your iptables log, you have a high denied traffic log reporting:

   >> tail -f /var/log/kern.log

3. Initial analisys

In such case, if most of these indicators are met, it is possible your system is under high connection denied preasure and your iptables rule chain is taking too long for each incoming packet that is not able to cope with current traffic.

4. About droppped indicator at network card (ethtool or ifconfig)

Generally speaking, you should have no or very low dropped indicators at any of your network cards while running:

>> ifconfig -a | grep dropped

Even in the case you have an iptables firewall and/or a QoS (tc based) service.

To have dropped indication at your network card means driver/kernel/card is not able to handle that traffic because it is malformed, there is not enough buffer or some internal processing is lagging behind causing traffic to be queued, and finally dropped.

It is important to understand this dropped indication has nothing to do with iptables DROP or tc drop (due to buckets and/or policy).

The first (network dropped indication) happens very early before getting into iptables/tc. The second (iptables dropping) happens after and, in any case, if iptables DROPS some traffic, it is not accounted at network card level using dropped indication as we have seen (remember: ifconfig -a | grep dropped).

5. A healthty iptables system must not have dropped at network card level

A Linux system, even under high rate traffic denied must hot have dropped indication at network card/driver/kernel level (ifconfig -a | dropped).

Even in the case you have iptables and QoS (tc) dropping packets because policy is not met.

6. How to solve this issue

1. First, reduce the number of blocking rules you have installed with IpBlocker.
   Select a list of rules that should be installed only for a couple of days and the removed.

   Possibly you installed them with permanent indication. In general, it is not recommended to do permanent blocking: it is best to block for some days to avoid having your firewall loaded with a bunch of rules not really needed.

   To remove ip blocker rules with a command line run:

   # take a backup
   >> crad-ip-blocker.pyc -l > list-of-blocking-rules.txt

   # now remove rules
   >> for I in crad-ip-blocker.pyc -l | cut -f1 -d:
   do
   crad-ip-blocker.pyc -r $I
   done

2. After that, check how many rules you have, to see if you have reduced:

   >> iptables -S | wc -l

3. At this point if everything went ok, you should be recovering to normal firewall processing.

   If not, check iptables -S rules to see if they are caused by Firewall Manager rules or another system/scripting installing blocking rules. You have to reduce the number of iptables rules.

4. If you have solved the issue, update your syn_floood_detect_checker to warn you when you get above 10.000 iptables rules so you can do some housekeeping before getting into trouble:

   4.a) Click on actions:

   

   4.b) Show machine’s checkers:

   

   4.c) Click on syn_flood_detect checker

   

   4.d) Click on Configure and setup new limit (Max rules limit):

   

7. What does not work

a) Increasing RX buffer size at network card will not solve the issue. Indeed it will solve not lossing packets but will increase latency because the problem is caused at packet processing for dropping.

>> ethtool -G eth0 rx 4096

b) Installing or restarting irqbalance