Web lists-archives.com

Re: Speed Problem Copying Files




On 5/9/19 1:49 AM, Lothar Schilling wrote:
Hi everybody,

for years I have used CentOS for our server landscape. Now I decided to
give Debian a try. I just set up a Stretch 9.8 system supposed to become
our main backup server. So I set up a backup job wih rsync. But the
going is really very very slooooow. Trying to figure out what's happening:

   * iperf -c [host] => bandwith almost 1000 Mbit, that's fine.
   * dd if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct =>
     10737418240 Bytes (11 GB, 10 GiB) kopiert, 40,4992 s, 265 MB/s, so
     that's fine as well

But whenever I try to rsync, cp or scp - whether copy files on the local
hard disk only or over the network - speed goes down to 500 kB/s.

Filesystem is ext4.

I don't have any clue about what's going on. Any kind of help would be
appreciated, thank you!

Lothar


On 5/9/19 2:46 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:14 schrieb Jonas Smedegaard:
>> Is _only_ transfer speed affected?
>>
>> I am no expert in this, but imagine that if you rsync massive amounts
>> involving hardlinks then memory becomes a problem too.
>>
>> Perhaps run atop to monitor bottlenecks live
>>
>>
>>   - Jonas
>>
> It is most definitely not a memory or cpu problem. It's a HP Proliant
> with 32 GB RAM and a Intel Xeon CPU 2.40GHz with 4 cores. Also the
> problem stays the same if I just copy one large file.


On 5/9/19 3:26 AM, Lothar Schilling wrote:> Am 09.05.2019 um 11:51 schrieb Kevin DAGNEAUX:
>> Check if it's related to the disk speed (hdparm and/or iotop).
>>
>> Kevin
>>
>
> hdparm -tT /dev/sda
> /dev/sda:
>   Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>   Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
>
> iotop -o (for rsync and cp)
> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
> --info=progress2 /daten/testfile /daten/testfile2
>
> iotop -o (for dd)
> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct


On 5/9/19 5:43 AM, Lothar Schilling wrote:> Am 09.05.2019 um 13:27 schrieb Martin:
>> [..]
>>> hdparm -tT /dev/sda
>>> /dev/sda:
>>>   Timing cached reads:   13348 MB in  2.00 seconds = 6683.42 MB/sec
>>>   Timing buffered disk reads: 1014 MB in  3.00 seconds = 337.72 MB/sec
>>>
>>> iotop -o (for rsync and cp)
>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     476.15 K/s
>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     487.86 K/s
>>>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>>> 19531 be/4 root        0.00 B/s  476.15 K/s  0.00 % 99.24 % rsync
>>> --info=progress2 /daten/testfile /daten/testfile2
>>>
>>> iotop -o (for dd)
>>> Total DISK READ :       0.00 B/s | Total DISK WRITE :     297.68 M/s
>>> Actual DISK READ:       0.00 B/s | Actual DISK WRITE:     297.68 M/s
>>>    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
>>> 19557 be/4 root        0.00 B/s  297.68 M/s  0.00 % 99.99 % dd
>>> if=/dev/zero of=/daten/testfile bs=1G count=10 oflag=direct
>> Show us the 'dd if=/daten/testfile bs=1G oflag=direct of=/dev/null', please. >> If this is as slow as this ~480k/s above, check your disk's health status. Like with smartmontools or some disk-utility software.
>>
>> Martin
>>
> Fast enough...
>
> dd if=/daten/testfile bs=1G oflag=direct of=/daten/testfile2
> 10+0 Datensätze ein
> 10+0 Datensätze aus
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 72,7297 s, 148 MB/s
>
> dd if=/daten/testfile of=/dev/null
> 20971520+0 Datensätze ein
> 20971520+0 Datensätze aus
> 10737418240 Bytes (11 GB, 10 GiB) kopiert, 36,6887 s, 293 MB/s


rsync(1) can be fickle, especially when host operating systems and/or rsync versions mismatch. But, you also seem to have file system issues.


The above information could be more useful if it were presented in context -- e.g. complete console sessions -- meaningful prompts (date, time, username, hostname, working directory), exact commands input, and exact output printed.


Please use the Bash shell for the root account and the following line to /root/.bashrc:

    export PS1='\n\D{%Y-%m-%d %H:%M:%S} '${USER}'@\h \w\n\$ '


Then start a new shell, su(1) to root, change to a directory within the file system that is having issues, and run the following commands. Adjust the commands for your hosts and network, as required ('po' is my workstation, 'dpchrist' is my user account, 'samba' is my file server, and '/var/local/samba/dpchrist' is my storage directory on my file server):

2019-05-09 21:59:55 root@po /mnt/scratch
# cat /etc/debian_version
9.9

2019-05-09 22:00:00 root@po /mnt/scratch
# uname -a
Linux po 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux

2019-05-09 22:00:05 root@po /mnt/scratch
# lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda              8:0    0  55.9G  0 disk
|-sda1           8:1    0   953M  0 part  /boot
|-sda2           8:2    0   1.9G  0 part
| `-sda2_crypt 253:1    0   1.9G  0 crypt [SWAP]
|-sda3           8:3    0   9.3G  0 part
| `-sda3_crypt 253:0    0   9.3G  0 crypt /
`-sda4           8:4    0  38.2G  0 part
  `-sda4_crypt 253:2    0  38.2G  0 crypt /mnt/scratch
sr0             11:0    1  1024M  0 rom

2019-05-09 22:00:10 root@po /mnt/scratch
# mount | egrep 'sd[a-z]'
/dev/mapper/sda3_crypt on / type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/) /dev/sda1 on /boot type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/) /dev/mapper/sda4_crypt on /mnt/scratch type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)

2019-05-09 22:00:19 root@po /mnt/scratch
# df | egrep 'sd[a-z]'
/dev/mapper/sda3_crypt   9763840 5540800   4160128  58% /
/dev/sda1                 975872   80892    800244  10% /boot
/dev/mapper/sda4_crypt  40056832 1066752  38732224   3% /mnt/scratch

2019-05-09 22:00:27 root@po /mnt/scratch
# time dd if=/dev/urandom of=foo bs=1M count=1K conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.987 s, 67.2 MB/s

real	0m15.988s
user	0m0.000s
sys	0m3.976s

2019-05-09 22:02:13 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && time dd if=foo of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.49821 s, 430 MB/s

real	0m2.518s
user	0m0.000s
sys	0m0.252s

2019-05-09 22:27:32 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar && uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && ssh $uh rm -f $p/$g && rsync -a --stats $f $uh:$p/$g

Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,073,741,824 bytes
Total transferred file size: 1,073,741,824 bytes
Literal data: 1,073,741,824 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1,074,004,051
Total bytes received: 35

sent 1,074,004,051 bytes  received 35 bytes  113,053,061.68 bytes/sec
total size is 1,073,741,824  speedup is 1.00

2019-05-09 22:27:52 root@po /mnt/scratch
# echo 3 > /proc/sys/vm/drop_caches && f=foo && g=bar && uh=dpchrist@samba && p=/var/local/samba/dpchrist/test && rm -f $g && rsync -a --stats $uh:$p/$g $g

Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 1,073,741,824 bytes
Total transferred file size: 1,073,741,824 bytes
Literal data: 1,073,741,824 bytes
Matched data: 0 bytes
File list size: 43
File list generation time: 0.022 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 43
Total bytes received: 1,074,004,061

sent 43 bytes  received 1,074,004,061 bytes  102,286,105.14 bytes/sec
total size is 1,073,741,824  speedup is 1.00


Add in other commands as you see fit, such as whatever generated the output in your previous posts.


Please reply with your *complete* console session (you can remove all of the above in your reply).


David