From https://github.com/borgbackup/borg/pull/480 discussion:
Did you try 1024 (linux cache block size) or 4096 (internal sector size of bigger
hdds, also used in msgpack fallback.py as lower bound, see link)?
I've tested different values - 512 and 1024 are slightly better than 4096 in my case.
read_size = 1 ls -laR: 75.57 sec
read_size = 64 ls -laR: 27.81 sec
read_size = 512 ls -laR: 27.40 sec
read_size = 1024 ls -laR: 27.20 sec
read_size = 4096 ls -laR: 30.15 sec
read_size = 0 ls -laR: 442.96 sec (default)
OK, maybe we should go for 1024 then. That happens to be < MTU size, so in case someone works on NFS
(or other network FS) we will have less reads, less network packets, less latency.
Single-shot unpacker read buffer decreased from (default) 1Mb to 512b.
"ls -alR" on ~100k files backup mounted with fuse went from ~7min to 30 seconds.
this was making us require mock, which is really a test component and
shouldn't be part of the runtime dependencies. furthermore, it was
making the imports and the code more brittle: it may have been
possible that, through an environment variable, backups could be
corrupted because mock libraries would be configured instead of real
once, which is a risk we shouldn't be taking.
finally, this was used only to build docs, which we will build and
commit to git by hand with a fully working borg when relevant.
see #384.
Thomas Waldmann