Web lists-archives.com

Re: Network issue




Dear Kevin,

and just one more thing.
I am beginner with iptables, but your POSTROUTING doesn't seem good as well.

You made a POSTROUTING in the case of VM-s what you manage (they can see the internet), but there is no source change on IP packets which goes in the direction of your VM-s. Your current rule works only in two direction, if any source from any direction is in the 10.0.0.0/24 LAN.

I think you should add there a second postrouting rule.
And for POSTROUTING you should use -i and -o flags instead of -s, because I am not sure if iptables understands which IP address should be the source when it makes the masquerade.

But first of all, I think you should check the hardware flow control setting.

- Tamas Fekete


2018-08-29 21:24 GMT+02:00 Fekete Tamás <fektom@xxxxxxxxx>:
Dear all,

the explanation of Reco seems quite good for me.
If you use load generator on your mobile device it could be the reason.

What happens if you use a normal download for a large file? Do you get the same result?

If yes, I advise to look at your hardware configuration like flow control. This line doesn't seem good for me (mind the text in bold):

"Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None"

See this link:

Let's see what happens if you reconfigure your network device.

- Tamas Fekete

2018-08-29 20:58 GMT+02:00 Reco <recoverym4n@xxxxxxxxx>:
        Hi.

Please do not top post. This is a mailing list, not a corporate e-mail
spamfest.

On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> Hi, I've never used OVH. How certain are you that the e1000 network driver
> is the correct one?
> Under VMWare/ESX the network driver choice can be crucial, for example.

1) You cannot run Xen in VSphere/ESXi.

2) No sane public provider will use VSphere/ESXi for hosting, the costs
can dim a budget of a small country.

3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
reality. SR-IOV is a way to go.


Now, to the issue at hand.

> On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <kevin.dagneaux@xxxxxxxxxxxx>
> wrote:

Please note that you seem to have "link down" first:

> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down

and 'malfunctioning' netfilter rules next.

> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0

Observe 'RST' flag at each and every 'DROPPED' message.
I find it highly unlikely that whatever PHP load test you did it
involved sending mass amounts of TCP Reset packets.

It seems that the following scenario has much higher probability:

1) You fired your load generator application.

2) Your hosting provided immediately got a signal of your typical DOS
attack (not to be confused with DDOS) coming from that seems to be a
typical mobile phone.

3) DDOS protection kicked in:

a) Isolating your server from the network to stop DOS.

b) Sending forged TCP RST to your server to break existing connections
*and* termninate unneeded Apache workers (or whatever you have there).

c) Banning the initiator of DOS (i.e. you on mobile network) temporarily.

4) Real network outage of your server was 2 seconds (time between "link
down" and "link up").

Reco

>
>
> > Hi,
> >
> > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of them
> > in run Apache.
> > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed an
> > HTML5/PHP speed test application.
> > When i use this app, i've no problem in general, but, when a make a speed
> > test from a source who have more bandwith than the server (the server is
> > limited at 100Mb/s by OVH and i make the test from a 4G+ network where i've
> > ~150Mb/s of bandwith), in this case, the DOM0 lost his network connection
> > (like the ethernet cable is unplugged) until i reboot the server.
> >
> > When i check the syslog of DOM0, i see that iptables drop incomming packet
> > on port 80 instead of routing them to the VM.
> >
> > This is my iptables script i use on DOM0 :
> >
> > #!/bin/bash
> >
> > IPT="/sbin/iptables"
> >
> >
> > ###########################################################################################
> > # Filter
> >
> > ## Remise par defaut des regles
> > $IPT -t filter -P INPUT   ACCEPT
> > $IPT -t filter -P FORWARD ACCEPT
> > $IPT -t filter -P OUTPUT  ACCEPT
> >
> > ## On purge les tables
> > $IPT -t filter -F
> >
> > ## On autorise lo
> > $IPT -t filter -A INPUT -i lo -j ACCEPT
> >
> > ## On ouvre les ports nécéssaires au DOM0
> > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22      -j
> > ACCEPT                                         ## SSH
> > $IPT -t filter -A INPUT -m udp -p udp --dport 53      -j
> > ACCEPT                                         ## DNS
> > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j
> > ACCEPT                                         ## Ping
> > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT
> >
> > ## On accepte si la connexion est déjà établie
> > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
> >
> > ## On log ce qui n'a pas été matché par les règles précédente
> > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets "
> >
> > ## On bloque tout le reste
> > $IPT -t filter -P INPUT DROP
> >
> >
> > ############################################################################################
> > # Nat
> >
> > ## Remise par defaut des regles
> > $IPT -t nat -P PREROUTING  ACCEPT
> > $IPT -t nat -P POSTROUTING ACCEPT
> > $IPT -t nat -P INPUT       ACCEPT
> > $IPT -t nat -P OUTPUT      ACCEPT
> >
> > ## On purge
> > $IPT -t nat -F
> >
> > ### Routage des ports entrants pour la VM "mails"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT --to
> > 10.0.0.30:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25    -j DNAT --to
> > 10.0.0.30:25                       ## SMTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587   -j DNAT --to
> > 10.0.0.30:587                      ## SMTP SUBMISSION
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465   -j DNAT --to
> > 10.0.0.30:465                      ## SMTP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143   -j DNAT --to
> > 10.0.0.30:143                      ## IMAP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993   -j DNAT --to
> > 10.0.0.30:993                      ## IMAP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930  -j DNAT --to
> > 10.0.0.30:9930                     ## IMAP SSL
> >
> > ### Routage des ports entrants pour la VM "sql"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT --to
> > 10.0.0.20:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306  -j DNAT --to
> > 10.0.0.20:3306                     ## MariaDB
> >
> > ### Routage des ports entrants pour la VM "files"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT --to
> > 10.0.0.10:22                       ## SSH
> >
> > ### Routage des ports entrant pour la VM "web"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT --to
> > 10.0.0.40:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80    -j DNAT --to
> > 10.0.0.40:80                       ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443   -j DNAT --to
> > 10.0.0.40:443                      ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "monitor"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT --to
> > 10.0.0.50:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT --to
> > 10.0.0.50:80                 ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT --to
> > 10.0.0.50:443                        ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "comm"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT --to
> > 10.0.0.60:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222  -j DNAT --to
> > 10.0.0.60:5222                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269  -j DNAT --to
> > 10.0.0.60:5269                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280  -j DNAT --to
> > 10.0.0.60:5280                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281  -j DNAT --to
> > 10.0.0.60:5281                     ## Jabber
> >
> > ### Routage des ports entrants pour la VM "secure"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT --to
> > 10.0.0.70:22                       ## SSH
> >
> > ### Routage des ports entrants pour la VM "net"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT --to
> > 10.0.0.80:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388  -j DNAT --to
> > 10.0.0.80:8388                     ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388  -j DNAT --to
> > 10.0.0.80:8388                     ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT --to
> > 10.0.0.80:1194                     ## OpenVPN
> >
> > ### Autorise les VMs a accéder a internet
> > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j  MASQUERADE
> >
> > And this is what i have in syslogs :
> >
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> >
> > *** I've several hundred of similare line where only the ID change, then :
> > ***
> >
> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
> > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier
> > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier
> > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full
> > Duplex, Flow Control: None
> > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100 speed:
> > disabling TSO
> >
> > The log show that the network is UP but the server is offline.
> > In the VM's logs i don't have abnormal things.
> >
> > OVH have changed the motherboard and the ethernet cable but the problem
> > still persist, the syslog show that it's probably a software problem.
> >
> > Do you have idea of what happen or a way to help me to debug this problem?
> >
> > Thank you in advance.
> >
> > Kevin
> > --
> >
> > DAGNEAUX Kevin
> > Service informatique
> > 03 29 36 88 85
> > kevin.dagneaux@xxxxxxxxxxxx
> >