Web lists-archives.com

Re: Network issue




Dear all,

the explanation of Reco seems quite good for me.
If you use load generator on your mobile device it could be the reason.

What happens if you use a normal download for a large file? Do you get the same result?

If yes, I advise to look at your hardware configuration like flow control. This line doesn't seem good for me (mind the text in bold):

"Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None"

See this link:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/tuning_and_optimizing_red_hat_enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-adjusting_network_settings-flow_control_for_e1000_network_interface_cards

Let's see what happens if you reconfigure your network device.

- Tamas Fekete

2018-08-29 20:58 GMT+02:00 Reco <recoverym4n@xxxxxxxxx>:
        Hi.

Please do not top post. This is a mailing list, not a corporate e-mail
spamfest.

On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> Hi, I've never used OVH. How certain are you that the e1000 network driver
> is the correct one?
> Under VMWare/ESX the network driver choice can be crucial, for example.

1) You cannot run Xen in VSphere/ESXi.

2) No sane public provider will use VSphere/ESXi for hosting, the costs
can dim a budget of a small country.

3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
reality. SR-IOV is a way to go.


Now, to the issue at hand.

> On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <kevin.dagneaux@xxxxxxxxxxxx>
> wrote:

Please note that you seem to have "link down" first:

> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down

and 'malfunctioning' netfilter rules next.

> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0

Observe 'RST' flag at each and every 'DROPPED' message.
I find it highly unlikely that whatever PHP load test you did it
involved sending mass amounts of TCP Reset packets.

It seems that the following scenario has much higher probability:

1) You fired your load generator application.

2) Your hosting provided immediately got a signal of your typical DOS
attack (not to be confused with DDOS) coming from that seems to be a
typical mobile phone.

3) DDOS protection kicked in:

a) Isolating your server from the network to stop DOS.

b) Sending forged TCP RST to your server to break existing connections
*and* termninate unneeded Apache workers (or whatever you have there).

c) Banning the initiator of DOS (i.e. you on mobile network) temporarily.

4) Real network outage of your server was 2 seconds (time between "link
down" and "link up").

Reco

>
>
> > Hi,
> >
> > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of them
> > in run Apache.
> > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed an
> > HTML5/PHP speed test application.
> > When i use this app, i've no problem in general, but, when a make a speed
> > test from a source who have more bandwith than the server (the server is
> > limited at 100Mb/s by OVH and i make the test from a 4G+ network where i've
> > ~150Mb/s of bandwith), in this case, the DOM0 lost his network connection
> > (like the ethernet cable is unplugged) until i reboot the server.
> >
> > When i check the syslog of DOM0, i see that iptables drop incomming packet
> > on port 80 instead of routing them to the VM.
> >
> > This is my iptables script i use on DOM0 :
> >
> > #!/bin/bash
> >
> > IPT="/sbin/iptables"
> >
> >
> > ###########################################################################################
> > # Filter
> >
> > ## Remise par defaut des regles
> > $IPT -t filter -P INPUT   ACCEPT
> > $IPT -t filter -P FORWARD ACCEPT
> > $IPT -t filter -P OUTPUT  ACCEPT
> >
> > ## On purge les tables
> > $IPT -t filter -F
> >
> > ## On autorise lo
> > $IPT -t filter -A INPUT -i lo -j ACCEPT
> >
> > ## On ouvre les ports nécéssaires au DOM0
> > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22      -j
> > ACCEPT                                         ## SSH
> > $IPT -t filter -A INPUT -m udp -p udp --dport 53      -j
> > ACCEPT                                         ## DNS
> > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j
> > ACCEPT                                         ## Ping
> > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT
> >
> > ## On accepte si la connexion est déjà établie
> > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
> >
> > ## On log ce qui n'a pas été matché par les règles précédente
> > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets "
> >
> > ## On bloque tout le reste
> > $IPT -t filter -P INPUT DROP
> >
> >
> > ############################################################################################
> > # Nat
> >
> > ## Remise par defaut des regles
> > $IPT -t nat -P PREROUTING  ACCEPT
> > $IPT -t nat -P POSTROUTING ACCEPT
> > $IPT -t nat -P INPUT       ACCEPT
> > $IPT -t nat -P OUTPUT      ACCEPT
> >
> > ## On purge
> > $IPT -t nat -F
> >
> > ### Routage des ports entrants pour la VM "mails"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT --to
> > 10.0.0.30:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25    -j DNAT --to
> > 10.0.0.30:25                       ## SMTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587   -j DNAT --to
> > 10.0.0.30:587                      ## SMTP SUBMISSION
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465   -j DNAT --to
> > 10.0.0.30:465                      ## SMTP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143   -j DNAT --to
> > 10.0.0.30:143                      ## IMAP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993   -j DNAT --to
> > 10.0.0.30:993                      ## IMAP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930  -j DNAT --to
> > 10.0.0.30:9930                     ## IMAP SSL
> >
> > ### Routage des ports entrants pour la VM "sql"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT --to
> > 10.0.0.20:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306  -j DNAT --to
> > 10.0.0.20:3306                     ## MariaDB
> >
> > ### Routage des ports entrants pour la VM "files"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT --to
> > 10.0.0.10:22                       ## SSH
> >
> > ### Routage des ports entrant pour la VM "web"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT --to
> > 10.0.0.40:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80    -j DNAT --to
> > 10.0.0.40:80                       ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443   -j DNAT --to
> > 10.0.0.40:443                      ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "monitor"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT --to
> > 10.0.0.50:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT --to
> > 10.0.0.50:80                 ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT --to
> > 10.0.0.50:443                        ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "comm"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT --to
> > 10.0.0.60:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222  -j DNAT --to
> > 10.0.0.60:5222                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269  -j DNAT --to
> > 10.0.0.60:5269                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280  -j DNAT --to
> > 10.0.0.60:5280                     ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281  -j DNAT --to
> > 10.0.0.60:5281                     ## Jabber
> >
> > ### Routage des ports entrants pour la VM "secure"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT --to
> > 10.0.0.70:22                       ## SSH
> >
> > ### Routage des ports entrants pour la VM "net"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT --to
> > 10.0.0.80:22                       ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388  -j DNAT --to
> > 10.0.0.80:8388                     ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388  -j DNAT --to
> > 10.0.0.80:8388                     ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT --to
> > 10.0.0.80:1194                     ## OpenVPN
> >
> > ### Autorise les VMs a accéder a internet
> > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j  MASQUERADE
> >
> > And this is what i have in syslogs :
> >
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=""> > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> >
> > *** I've several hundred of similare line where only the ID change, then :
> > ***
> >
> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
> > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier
> > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier
> > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full
> > Duplex, Flow Control: None
> > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100 speed:
> > disabling TSO
> >
> > The log show that the network is UP but the server is offline.
> > In the VM's logs i don't have abnormal things.
> >
> > OVH have changed the motherboard and the ethernet cable but the problem
> > still persist, the syslog show that it's probably a software problem.
> >
> > Do you have idea of what happen or a way to help me to debug this problem?
> >
> > Thank you in advance.
> >
> > Kevin
> > --
> >
> > DAGNEAUX Kevin
> > Service informatique
> > 03 29 36 88 85
> > kevin.dagneaux@xxxxxxxxxxxx
> >