Web lists-archives.com

Inexplicable memory usage after move to Debian9




Hi Recently I've started moving a fleet of Debian 7, 32-bit machines over to Debian 9, 64-bit. This migration is done by creating a fresh Debian 9 image with the necessary services, moving over user data (some wars and the content of /home) and rebooting into the new OS.

Relevant services (ones we manage and use) are:

- Jetty- Puppet - SSH - AutoSSH - NewRelic Infrastructure Through Puppet, we enforce system configuration is pretty much identical, save for stuff like host names and SSH keys. Now, we notice that on some systems, the RAM usage is way higher than expected, to the point where system memory is exhausted and processes (are) terminate(d). Investigation into what is causing this, leaves me at a dead end. I can't figure out where the memory is being consumed. Even after quitting all services we manage (leaving a "clean" installation), RAM usage on the system hovers just over 600MB, half a gig over what the same exact image consumes just after boot. The only fix I found so far is to reboot the system. The systems have ~1.8GB of system memory available. We don't use swap. Enabling swap gives the system some breathing room. On one system I enabled a swap volume of 512MB, which the kernel fills up and leaves filled up indefinitely. This points me to unused memory being allocated by mistake. A memory leak in the kernel, maybe? Below I've posted the output of some of the things I checked (with all services online). I also searched the internet and reached a topic about slab allocation (1). However, that didn't seem to solve anything. Can anyone here point me to some more stuff I can check out or try to debug this? Thanks! Simon root@mysystem:~# uname -a Linux mysystem 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u2 (2018-02-21) x86_64 GNU/Linux root@mysystem:~# free -m total used free shared buff/cache available Mem: 1831 1091 238 16 501 520 Swap: 511 511 0 root@mysystem:~# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 524268 244480 46044 467248 1 2 2137 36 97 35 9 3 85 3 0 0 0 524268 244092 46052 467244 0 0 0 16 1125 2406 3 3 94 1 0 0 0 524268 244092 46060 467284 0 0 4 208 1230 3906 4 3 93 0 0 0 0 524268 244084 46068 467256 0 0 0 32 1003 1990 1 2 97 0 0 0 0 524268 244180 46068 467256 0 0 0 0 1099 2121 4 1 95 0 0 0 0 524268 244204 46076 467272 0 0 0 20 1000 1978 1 2 97 1 0 0 0 524268 244080 46076 467272 0 0 0 0 1135 2315 2 2 96 0 0 0 0 524268 244080 46084 467264 0 0 0 16 1079 2103 1 3 96 1 0 0 0 524268 244080 46092 467272 0 0 0 56 1002 1973 2 2 96 0 0 1 0 524268 244080 46100 467264 0 0 0 16 997 1979 1 2 97 1 0 0 0 524268 244144 46100 467268 0 0 0 0 988 1957 1 2 97 0 0 0 1 524268 244228 46108 467260 0 0 0 980 1292 2700 4 5 81 9 0 root@mysystem:~# smem PID User Command Swap USS PSS RSS 528 root /sbin/agetty -f /etc/issue. 148 4 4 8 554 myuser /usr/lib/autossh/autossh -o 80 24 40 184 220 root /lib/systemd/systemd-udevd 552 108 140 688 10665 root /usr/lib/autossh/autossh -o 0 104 143 648 432 root /usr/sbin/cron -f 164 120 147 500 434 root /usr/sbin/irqbalance --fore 252 156 172 508 12042 mail /usr/sbin/nullmailer-send - 120 144 219 1228 382 systemd-timesync /lib/systemd/systemd-timesy 456 152 301 1060 439 messagebus /usr/bin/dbus-daemon --syst 272 280 348 1116 430 root /lib/systemd/systemd-logind 380 192 448 1224 557 myuser /usr/bin/ssh -o StrictHostK 564 360 515 1472 216 root /sbin/lvmetad -f 188 484 517 1028 10668 root /usr/bin/ssh -o StrictHostK 0 736 825 1112 14225 ntp /usr/sbin/ntpd -p /var/run/ 0 808 849 1404 435 root /usr/sbin/rsyslogd -n 716 896 980 2036 11612 root /usr/sbin/sshd -D 0 856 991 1560 500 root /sbin/dhclient -4 -v -pf /r 840 928 1010 2068 1330 root sudo -i 0 920 1375 3548 1234 myuser -bash 0 600 1407 3684 1331 root -bash 0 640 1447 3712 1233 myuser sshd: myuser@pts/0 0 300 1697 4568 1 root /sbin/init 524 1568 1935 3524 1227 root sshd: posios [priv] 0 1372 2984 6380 193 root /lib/systemd/systemd-journa 312 4304 4521 5828 8885 root /usr/bin/python /usr/bin/sm 0 9100 9452 11292 14574 root /usr/bin/newrelic-infra 1440 17348 17378 17828 520 root /opt/puppetlabs/puppet/bin/ 20636 40140 40168 40592 566 jetty /usr/lib/jvm/java-8-openjdk 493896 958124 958381 959804 [1] https://unix.stackexchange.com/questions/244735/why-are-slab-objects-not-reclaimed-automatically