Outgoing packets that fail to match INVALID, ESTABLISHED, RELATED or NEW?

[Whit Blauvelt]

Hi,

With this setup:

- Public MX on the firewall
- Everything INVALID tossed and logged
- Everything ESTABLISHED, RELATED allowed
- Everything NEW from the MX IP as outbound OUTPUT allowed
- Everything else in OUTPUT dropped and logged

There are packets getting down to the OUTPUT drop and log rule that are
outward from Postfix on port 25 to remote addresses on high ports. This is
obviously Postfix replying to incoming connections, some of them as
legitimate as the Apache mailing list at apache.org. It's not happening for
most incoming mail connections, or always for the ones it does, but it's
been more than a few.

So how are these replies falling through the cracks of the available states?
Thy're not INVALID. And if they're not matching ESTABLISHED, RELATED
(perhaps because they've timed out since the transaction was started), they
ought to qualify as NEW, right?

I've set up a stateless rule just for outgoing traffic with --sport 25 from
the MX IP. Doesn't seem I should have needed to though. Is there a known
flaw in Netfilter's stateful inspection that requires such a workaround?

While I'm mostly seeing this with smtp, I also see it on an outgoing nmap
scan originated from this box - where some of nmap's packets (to various
standard service ports) ended up blocked after failing to match the INVALID,
ESTABLISHED, RELATED or NEW rules. This happens consistently in a certain
part of the nmap sequence.

Should this ever happen? Is conntrack failing to mark any state on some
locally-originated packets? Are other people finding a need to set up a
stateless rule to allow some classes of locally-originated traffic out? Is
this a bug I should file against conntrack, or just a known loophole in the
matching process?

Thanks,
Whit

------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion

[Vadim Kurland ✎]

not enough data to say for sure.

On Jul 24, 2009, at 8:31 AM, Whit Blauvelt wrote:

> Hi,
>
> With this setup:
>
> - Public MX on the firewall
> - Everything INVALID tossed and logged
> - Everything ESTABLISHED, RELATED allowed
> - Everything NEW from the MX IP as outbound OUTPUT allowed
> - Everything else in OUTPUT dropped and logged
>
> There are packets getting down to the OUTPUT drop and log rule that  
> are
> outward from Postfix on port 25 to remote addresses on high ports.  
> This is
> obviously Postfix replying to incoming connections, some of them as
> legitimate as the Apache mailing list at apache.org. It's not  
> happening for
> most incoming mail connections, or always for the ones it does, but  
> it's
> been more than a few.
>

could be final packets of the closing tcp session sequence (they  
should have FIN flag set). These sometimes slip through conntrack  
state tracking for some reason. Everything should just work though.  
Can't say for sure without seeing log record.

> So how are these replies falling through the cracks of the available  
> states?
> Thy're not INVALID. And if they're not matching ESTABLISHED, RELATED
> (perhaps because they've timed out since the transaction was  
> started), they
> ought to qualify as NEW, right?
>
> I've set up a stateless rule just for outgoing traffic with --sport  
> 25 from
> the MX IP. Doesn't seem I should have needed to though. Is there a  
> known
> flaw in Netfilter's stateful inspection that requires such a  
> workaround?
>

You should not need this rule. Do things break if the rule is not  
there ?

> While I'm mostly seeing this with smtp, I also see it on an outgoing  
> nmap
> scan originated from this box - where some of nmap's packets (to  
> various
> standard service ports) ended up blocked after failing to match the  
> INVALID,
> ESTABLISHED, RELATED or NEW rules. This happens consistently in a  
> certain
> part of the nmap sequence.
>
> Should this ever happen? Is conntrack failing to mark any state on  
> some
> locally-originated packets? Are other people finding a need to set  
> up a
> stateless rule to allow some classes of locally-originated traffic  
> out? Is
> this a bug I should file against conntrack, or just a known loophole  
> in the
> matching process?

can;t say without seeing your rules. Others never complained about  
this, so likely it is something with your rules.


Vadim Kurland ✍
vadim@...







------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion

[Whit Blauvelt]

On Fri, Jul 24, 2009 at 09:06:43AM -0700, Vadim Kurland ✎ wrote:

> could be final packets of the closing tcp session sequence (they should
> have FIN flag set). These sometimes slip through conntrack state tracking
> for some reason. Everything should just work though. Can't say for sure
> without seeing log record.

The outgoing-from-port-25 logging looks like:

Jul 24 10:23:54 whit kernel: [1999026.789061] OUTPUT deny: IN= OUT=eth0 SRC=123.456.789.77 DST=140.211.11.3 LEN=67 TOS=0x00
   PREC=0x00 TTL=64 ID=21349 DF PROTO=TCP SPT=25 DPT=58805 WINDOW=130 RES=0x00 ACK PSH FIN URGP=0

So we do see FIN there as you suggest. FIN is in all the smtp stuff getting
caught. Not sure if anything breaks with this.

The failed nmap logging looks like, as this happens for an ftp port probe,
but it's a general failure for a group of probes to common services:

Jul 23 10:51:33 whit kernel: [1914286.014808] OUTPUT denyIN= OUT=eth0 SRC=123.456.789.77 DST=11.22.33.44 LEN=60 TOS=0x00
  PREC=0x00 TTL=52 ID=27543 DF PROTO=TCP SPT=43332 DPT=21 WINDOW=128 RES=0x00 URGP=0

So those are not FIN. I'm not aware of anything with smtp being broken. But
nmap certainly is for part of its core functions. Since it's hard to imagine
other firewall-building sysadmins not running nmap to test the results, if
nobody's ever reported this problem, maybe there is a bug specific to my
conntrack version?

Here's the rules. There's the ESTABLISHED, RELATED up top. The INVALID right
after it. The original OUTPUT NEW rule is rule 9. The kludge to skip the
state match for tcp from port 24 is rule 8.

#!/bin/sh

... [standard stuff]

prolog_commands() {
echo "Running prolog script"

}

epilog_commands() {
echo "Running epilog script"

}

run_epilog_and_exit() {
  epilog_commands
  exit $1
}

prolog_commands


MODULES_DIR="/lib/modules/`uname -r`/kernel/net/"
MODULES=`find $MODULES_DIR -name '*conntrack*'|sed  -e 's/^.*\///' -e 's/\([^\.]\)\..*/\1/'`
for module in $MODULES; do
  if $LSMOD | grep ${module} >/dev/null; then continue; fi
  $MODPROBE ${module} ||  exit 1
done


# Using 0 address table files


# Configure interfaces
getaddr tun0  i_tun0
getaddr6 tun0  i_tun0_v6

# Add virtual addresses for NAT rules


log 'Activating firewall script generated Fri Jul 24 12:58:03 2009  by root'



# ================ IPv4


# ================ Table 'filter', automatic rules
$IPTABLES -P OUTPUT  DROP
$IPTABLES -P INPUT   DROP
$IPTABLES -P FORWARD DROP

cat /proc/net/ip_tables_names | while read table; do
  $IPTABLES -t $table -L -n | while read c chain rest; do
      if test "X$c" = "XChain" ; then
        $IPTABLES -t $table -F $chain
      fi
  done
  $IPTABLES -t $table -X
done


$IPTABLES -A INPUT   -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A OUTPUT  -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# backup ssh access
#
$IPTABLES -A INPUT  -p tcp -m tcp  -s 123.456.789.77/255.255.255.255  --dport 22  -m state --state NEW,ESTABLISHED -j ACCEPT
$IPTABLES -A OUTPUT  -p tcp -m tcp  -d 123.456.789.77/255.255.255.255  --sport 22  -m state --state ESTABLISHED,RELATED -j ACCEPT

# drop packets that do not match any valid state
#
$IPTABLES -N drop_invalid
$IPTABLES -A OUTPUT   -m state --state INVALID  -j drop_invalid
$IPTABLES -A INPUT    -m state --state INVALID  -j drop_invalid
$IPTABLES -A FORWARD  -m state --state INVALID  -j drop_invalid
$IPTABLES -A drop_invalid -j LOG --log-level debug --log-prefix "RULE -1 -- DENY "
$IPTABLES -A drop_invalid  -j DROP

# ================ Table 'filter', rule set Policy
# Policy compiler errors and warnings:
#
#
# Rule 0 (lo)
#
echo "Rule 0 (lo)"
#
#
#
$IPTABLES -A INPUT  -i lo  -m state --state NEW  -j ACCEPT
$IPTABLES -A OUTPUT  -o lo  -m state --state NEW  -j ACCEPT
#
# Rule 1 (eth0)
#
echo "Rule 1 (eth0)"
#
# anti-spoof
#
$IPTABLES -N In_RULE_1
$IPTABLES -A FORWARD  -i eth0  -s 123.456.789.77  -j In_RULE_1
$IPTABLES -A FORWARD  -i eth0  -s 192.168.2.6  -j In_RULE_1
test -n "$i_tun0" && $IPTABLES -A FORWARD  -i eth0  -s $i_tun0  -j In_RULE_1
$IPTABLES -A FORWARD  -i eth0  -s 192.168.2.0/24  -j In_RULE_1
$IPTABLES -A In_RULE_1  -j LOG  --log-level info --log-prefix "Spoof: "
$IPTABLES -A In_RULE_1  -j DROP
#
# Rule 2 (tun0)
#
echo "Rule 2 (tun0)"
#
#
#
$IPTABLES -N Cid4133X10842.0
test -n "$i_tun0" && $IPTABLES -A OUTPUT  -o tun0  -s $i_tun0  -m state --state NEW  -j Cid4133X10842.0
$IPTABLES -A Cid4133X10842.0  -d 10.20.30.0/24  -j ACCEPT
$IPTABLES -A Cid4133X10842.0  -d 192.168.1.0/24  -j ACCEPT
$IPTABLES -A Cid4133X10842.0  -d 10.50.30.0/24  -j ACCEPT
#
# Rule 3 (eth1)
#
echo "Rule 3 (eth1)"
#
#
#
$IPTABLES -A INPUT  -i eth1  -s 192.168.2.0/24  -d 192.168.2.6  -m state --state NEW  -j ACCEPT
$IPTABLES -A INPUT  -i eth1  -s 192.168.2.0/24  -d 123.456.789.77  -m state --state NEW  -j ACCEPT
$IPTABLES -A FORWARD  -i eth1  -s 192.168.2.0/24  -d 123.456.789.65/28  -m state --state NEW  -j ACCEPT
#
# Rule 4 (eth1)
#
echo "Rule 4 (eth1)"
#
#
#
$IPTABLES -A OUTPUT  -o eth1  -s 192.168.2.6  -d 192.168.2.0/24  -m state --state NEW  -j ACCEPT
#
# Rule 5 (eth0)
#
echo "Rule 5 (eth0)"
#
#
#
$IPTABLES -N Cid4172X10842.0
$IPTABLES -A INPUT  -i eth0  -d 123.456.789.77  -m state --state NEW  -j Cid4172X10842.0
$IPTABLES -A Cid4172X10842.0 -p icmp  -m icmp  --icmp-type 8/0   -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p icmp  -m icmp  --icmp-type 11/0   -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p icmp  -m icmp  --icmp-type 11/1   -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p icmp  -m icmp  --icmp-type 0/0   -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p icmp  -m icmp  --icmp-type 3  -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p tcp -m tcp  -m multiport  --dports 80,53,25  -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p udp -m udp  --dport 33434:33524  -j ACCEPT
$IPTABLES -A Cid4172X10842.0 -p udp -m udp  -m multiport  --dports 53,123  -j ACCEPT
#
# Rule 6 (global)
#
echo "Rule 6 (global)"
#
#
#
$IPTABLES -N In_RULE_6
$IPTABLES -A INPUT  -i + -p tcp -m tcp  -d 123.456.789.77  --dport 113  -j In_RULE_6
$IPTABLES -A In_RULE_6   -j LOG  --log-level info --log-prefix "auth: "
$IPTABLES -A In_RULE_6 -p tcp -m tcp  -j REJECT  --reject-with tcp-reset
#
# Rule 7 (eth0)
#
echo "Rule 7 (eth0)"
#
#
#
$IPTABLES -N In_RULE_7
$IPTABLES -A INPUT  -i eth0 -p tcp -m tcp  -d 123.456.789.77  --dport 22  -m state --state NEW  -m limit --limit 3/hour --limit-burst 3 -j In_RULE_7
$IPTABLES -A In_RULE_7  -j LOG  --log-level info --log-prefix "ssh: "
$IPTABLES -A In_RULE_7  -j ACCEPT
#
# Rule 8 (eth0)
#
echo "Rule 8 (eth0)"
#
#
#
$IPTABLES -A OUTPUT  -o eth0 -p tcp -m tcp  -s 123.456.789.77  --sport 25  -j ACCEPT
#
# Rule 9 (eth0)
#
echo "Rule 9 (eth0)"
#
#
#
$IPTABLES -A OUTPUT  -o eth0  -s 123.456.789.77  -m state --state NEW  -j ACCEPT
#
# Rule 10 (global)
#
echo "Rule 10 (global)"
#
# input
#
#
$IPTABLES -N RULE_10
$IPTABLES -A INPUT  -j RULE_10
$IPTABLES -A RULE_10  -j LOG  --log-level info --log-prefix "INPUT deny: "
$IPTABLES -A RULE_10  -j DROP
#
# Rule 11 (global)
#
echo "Rule 11 (global)"
#
# output
#
$IPTABLES -N RULE_11
$IPTABLES -A OUTPUT  -j RULE_11
$IPTABLES -A RULE_11  -j LOG  --log-level info --log-prefix "OUTPUT deny: "
$IPTABLES -A RULE_11  -j DROP
#
# Rule 12 (global)
#
echo "Rule 12 (global)"
#
# forward
#
$IPTABLES -N RULE_12
$IPTABLES -A FORWARD  -j RULE_12
$IPTABLES -A RULE_12  -j LOG  --log-level info --log-prefix "FORWARD deny: "
$IPTABLES -A RULE_12  -j DROP

epilog_commands

echo 1 > /proc/sys/net/ipv4/ip_forward

---

Whit

------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion

[Vadim Kurland ✎]

On Jul 24, 2009, at 10:11 AM, Whit Blauvelt wrote:

> On Fri, Jul 24, 2009 at 09:06:43AM -0700, Vadim Kurland ✎ wrote:
>
>> could be final packets of the closing tcp session sequence (they  
>> should
>> have FIN flag set). These sometimes slip through conntrack state  
>> tracking
>> for some reason. Everything should just work though. Can't say for  
>> sure
>> without seeing log record.
>
> The outgoing-from-port-25 logging looks like:
>
> Jul 24 10:23:54 whit kernel: [1999026.789061] OUTPUT deny: IN=  
> OUT=eth0 SRC=123.456.789.77 DST=140.211.11.3 LEN=67 TOS=0x00
>   PREC=0x00 TTL=64 ID=21349 DF PROTO=TCP SPT=25 DPT=58805 WINDOW=130  
> RES=0x00 ACK PSH FIN URGP=0
>
> So we do see FIN there as you suggest. FIN is in all the smtp stuff  
> getting
> caught. Not sure if anything breaks with this.
>

as I guessed, this is the final packet of the 4-way closing sequence  
of tcp session. Things will "just work" if you ignore this. I wasn't  
able to find explanation why conntrack misses these packets.

> The failed nmap logging looks like, as this happens for an ftp port  
> probe,
> but it's a general failure for a group of probes to common services:
>
> Jul 23 10:51:33 whit kernel: [1914286.014808] OUTPUT denyIN=  
> OUT=eth0 SRC=123.456.789.77 DST=11.22.33.44 LEN=60 TOS=0x00
>  PREC=0x00 TTL=52 ID=27543 DF PROTO=TCP SPT=43332 DPT=21 WINDOW=128  
> RES=0x00 URGP=0
>
> So those are not FIN. I'm not aware of anything with smtp being  
> broken. But
> nmap certainly is for part of its core functions. Since it's hard to  
> imagine
> other firewall-building sysadmins not running nmap to test the  
> results, if
> nobody's ever reported this problem, maybe there is a bug specific  
> to my
> conntrack version?

looks like blocked nmap packet had no tcp flags set at all, which is  
invalid configuration. Such packet will not match NEW state and will  
be blocked by the firewall.

--vk


------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion

[Whit Blauvelt]

On Fri, Jul 24, 2009 at 10:21:02AM -0700, Vadim Kurland ✎ wrote:

> looks like blocked nmap packet had no tcp flags set at all, which is  
> invalid configuration. Such packet will not match NEW state and will be
> blocked by the firewall.

Doesn't surprise me that nmap would try something invalid. This suggests
something's being missed by

$IPTABLES -A OUTPUT   -m state --state INVALID  -j drop_invalid

since the nmap packets aren't blocked until the final OUTPUT denying rule.
I'd even tried deleting the INVALID matching for OUTPUT, just in case the
problem was in logging them as INVALID rather than matching them as such,
but they still didn't get blocked until the final catch-all OUTPUT drop/log
rule.

So I guess the takeaway is (1) don't worry about it for smtp but (2) the
rule for OUTPUT has to be set to stateless for nmap to fully work, and (3)
there are invalid packets that INVALID fails to match.

Thanks for the help on this.

Whit

------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion

[Whit Blauvelt]

On Fri, Jul 24, 2009 at 01:36:42PM -0400, Whit Blauvelt wrote:

> (3) there are invalid packets that INVALID fails to match.

Correction. INVALID will match them. That log sample for nmap came from a
time when I'd deleted the INVALID match rule. My bad.

Whit

------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion

[Vadim Kurland ✎]

On Jul 24, 2009, at 10:36 AM, Whit Blauvelt wrote:

> On Fri, Jul 24, 2009 at 10:21:02AM -0700, Vadim Kurland ✎ wrote:
>
>> looks like blocked nmap packet had no tcp flags set at all, which is
>> invalid configuration. Such packet will not match NEW state and  
>> will be
>> blocked by the firewall.
>
> Doesn't surprise me that nmap would try something invalid. This  
> suggests
> something's being missed by
>
> $IPTABLES -A OUTPUT   -m state --state INVALID  -j drop_invalid
>
> since the nmap packets aren't blocked until the final OUTPUT denying  
> rule.
> I'd even tried deleting the INVALID matching for OUTPUT, just in  
> case the
> problem was in logging them as INVALID rather than matching them as  
> such,
> but they still didn't get blocked until the final catch-all OUTPUT  
> drop/log
> rule.
>
> So I guess the takeaway is (1) don't worry about it for smtp but (2)  
> the
> rule for OUTPUT has to be set to stateless for nmap to fully work,  
> and (3)
> there are invalid packets that INVALID fails to match.
>

definitions of the state INVALID vary. The filtering HOWTO says:

http://www.netfilter.org/documentation/HOWTO//packet-filtering-HOWTO.html

INVALID
A packet which could not be identified for some reason: this includes  
running out of memory and ICMP errors which don't correspond to any  
known connection. Generally these packets should be dropped.


the man page iptables(8) says

Possible states are INVALID meaning that the packet is associated with  
no known connection  ...


If the first definition is correct, then nmap packet as shown should  
not match state INVALID.

Either way, the nmap packet that got blocked was probably part of the  
OS fingerprint probing or TCP NULL scan (-sN) which is certainly an  
unusual activity that does not happen during normal operation of  
regular applications. You need to add a rule if you need this kind of  
probing to work, otherwise I would not worry about it.

--vk




------------------------------------------------------------------------------
_______________________________________________
Fwbuilder-discussion mailing list
Fwbuilder-discussion@...
https://lists.sourceforge.net/lists/listinfo/fwbuilder-discussion