Merge pull request #3461 from ThomasWaldmann/docs-lz4-b2-zstd

docs update for lz4 b2 zstd changes

README.rst

@@ -58,8 +58,13 @@ Main features
     authenticity is verified using HMAC-SHA256. Data is encrypted clientside.
 
 **Compression**
-    All data can be compressed by lz4 (super fast, low compression), zlib
+    All data can be optionally compressed:
-    (medium speed and compression) or lzma (low speed, high compression).
+
+    * lz4 (super fast, low compression)
+    * zstd (wide range from high speed and low compression to high compression
+      and lower speed)
+    * zlib (medium speed and compression)
+    * lzma (low speed, high compression)
 
 **Off-site backups**
     Borg can store data on any remote host accessible over SSH.  If Borg is

docs/deployment/central-backup-server.rst

@@ -171,8 +171,6 @@ Salt running on a Debian system.
         - openssl
         - libacl1-dev
         - libacl1
-        - liblz4-dev
-        - liblz4-1
         - build-essential
         - libfuse-dev
         - fuse

docs/faq.rst

@@ -788,7 +788,7 @@ Here's a (incomplete) list of some major changes:
 * more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_)
 * less chunk management overhead (less memory and disk usage for chunks index)
 * faster remote cache resync (useful when backing up multiple machines into same repo)
-* compression: no, lz4, zlib or lzma compression, adjustable compression levels
+* compression: no, lz4, zstd, zlib or lzma compression, adjustable compression levels
 * repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode)
 * simple sparse file support, great for virtual machine disk files
 * can read special files (e.g. block devices) or from stdin, write to stdout
@@ -840,8 +840,7 @@ The default in Borg is lz4, which is fast enough to not use significant CPU time
 in most cases, but can only achieve modest compression. It still compresses
 easily compressed data fairly well.
 
-zlib compression with all levels (1-9) as well as LZMA (1-6) are available
+Borg also offers zstd, zlib and lzma compression, choose wisely.
-as well, for cases where they are worth it.
 
 Which choice is the best option depends on a number of factors, like
 bandwidth to the repository, how well the data compresses, available CPU

docs/global.rst.inc

@@ -14,6 +14,7 @@
 .. _libacl: https://savannah.nongnu.org/projects/acl/
 .. _libattr: https://savannah.nongnu.org/projects/attr/
 .. _liblz4: https://github.com/Cyan4973/lz4
+.. _libzstd: https://github.com/facebook/zstd
 .. _libb2: https://github.com/BLAKE2/libb2
 .. _ZeroMQ: http://zeromq.org/
 .. _OpenSSL: https://www.openssl.org/

docs/installation.rst

@@ -141,12 +141,17 @@ following dependencies first:
   optional install.
 * OpenSSL_ >= 1.0.0, plus development headers.
 * libacl_ (which depends on libattr_), both plus development headers.
-* liblz4_ >= r129 (1.7.0), plus development headers.
 * ZeroMQ_ >= 4.0.0, plus development headers.
+* We have bundled code of the following packages, but borg by default (see
+  setup.py if you want to change that) prefers a shared library if it can
+  be found on the system (lib + dev headers) at build time:
+
+  - liblz4_ >= 1.7.0 (r129)
+  - libzstd_ >= 1.3.0
+  - libb2_
 * some Python dependencies, pip will automatically install them for you
 * optionally, the llfuse_ Python package is required if you wish to mount an
   archive as a FUSE filesystem. See setup.py about the version requirements.
-* optionally libb2_. If it is not found a bundled implementation is used instead.
 
 If you have troubles finding the right package names, have a look at the
 distribution specific sections below or the Vagrantfile in the git repository,
@@ -169,7 +174,6 @@ Install the dependencies with development headers::
     sudo apt-get install python3 python3-dev python3-pip python-virtualenv \
     libssl-dev openssl \
     libacl1-dev libacl1 \
-    liblz4-dev liblz4-1 \
     libzmq3-dev libzmq3 \
     build-essential
     sudo apt-get install libfuse-dev fuse pkg-config    # optional, for FUSE support
@@ -188,7 +192,6 @@ Install the dependencies with development headers::
     sudo dnf install python3 python3-devel python3-pip python3-virtualenv
     sudo dnf install openssl-devel openssl
     sudo dnf install libacl-devel libacl
-    sudo dnf install lz4-devel
     sudo dnf install gcc gcc-c++
     sudo dnf install redhat-rpm-config                 # not needed in Korora
     sudo dnf install fuse-devel fuse pkgconfig         # optional, for FUSE support
@@ -205,7 +208,7 @@ Install the dependencies automatically using zypper::
 Alternatively, you can enumerate all build dependencies in the command line::
 
     sudo zypper install python3 python3-devel \
-    libacl-devel liblz4-devel openssl-devel \
+    libacl-devel openssl-devel \
     python3-Cython python3-Sphinx python3-msgpack-python \
     python3-pytest python3-setuptools python3-setuptools_scm \
     python3-sphinx_rtd_theme python3-llfuse gcc gcc-c++
@@ -218,7 +221,7 @@ Mac OS X
 Assuming you have installed homebrew_, the following steps will install all the
 dependencies::
 
-    brew install python3 lz4 openssl
+    brew install python3 openssl
     brew install pkg-config                            # optional, for FUSE support
     pip3 install virtualenv
 
@@ -238,7 +241,7 @@ and commands to make FUSE work for using the mount command.
 
 ::
 
-     pkg install -y python3 openssl liblz4 fusefs-libs pkgconf
+     pkg install -y python3 openssl fusefs-libs pkgconf
      pkg install -y git
      python3.4 -m ensurepip # to install pip for Python3
      To use the mount command:
@@ -270,7 +273,6 @@ Use the Cygwin installer to install the dependencies::
     python3 python3-devel python3-setuptools
     binutils gcc-g++
     libopenssl openssl-devel
-    liblz4_1 liblz4-devel
     git make openssh
 
 You can then install ``pip`` and ``virtualenv``::

docs/internals/data-structures.rst

@@ -913,15 +913,16 @@ Borg supports the following compression methods:
 
 - none (no compression, pass through data 1:1)
 - lz4 (low compression, but super fast)
+- zstd (level 1-22 offering a wide range: level 1 is lower compression and high
+  speed, level 22 is higher compression and lower speed) - since borg 1.1.4
 - zlib (level 0-9, level 0 is no compression [but still adding zlib overhead],
   level 1 is low, level 9 is high compression)
 - lzma (level 0-9, level 0 is low, level 9 is high compression).
 
-Speed:  none > lz4 > zlib > lzma
+Speed:  none > lz4 > zlib > lzma, lz4 > zstd
-Compression: lzma > zlib > lz4 > none
+Compression: lzma > zlib > lz4 > none, zstd > lz4
 
-Be careful, higher zlib and especially lzma compression levels might take a
+Be careful, higher compression levels might use a lot of resources (CPU/memory).
-lot of resources (CPU and memory).
 
 The overall speed of course also depends on the speed of your target storage.
 If that is slow, using a higher compression level might yield better overall

docs/quickstart.rst

@@ -186,6 +186,17 @@ Backup compression
 The default is lz4 (very fast, but low compression ratio), but other methods are
 supported for different situations.
 
+You can use zstd for a wide range from high speed (and relatively low
+compression) using N=1 to high compression (and lower speed) using N=22.
+
+zstd is a modern compression algorithm and might be preferable over zlib and
+lzma, except if you need compatibility to older borg versions (< 1.1.4) that
+did not yet offer zstd.
+
+    $ borg create --compression zstd,N /path/to/repo::arch ~
+
+Other options are:
+
 If you have a fast repo storage and you want minimum CPU usage, no compression::
 
     $ borg create --compression none /path/to/repo::arch ~

src/borg/archiver.py

@@ -1968,7 +1968,13 @@ class Archiver:
             Do not compress.
 
         lz4
-            Use lz4 compression. High speed, low compression. (default)
+            Use lz4 compression. Very high speed, very low compression. (default)
+
+        zstd[,L]
+            Use zstd ("zstandard") compression, a modern wide-range algorithm.
+            If you do not explicitely give the compression level L (ranging from 1
+            to 22), it will use level 3.
+            Archives compressed with zstd are not compatible with borg < 1.1.4.
 
         zlib[,L]
             Use zlib ("gz") compression. Medium speed, medium compression.
@@ -1995,6 +2001,8 @@ class Archiver:
         Examples::
 
             borg create --compression lz4 REPO::ARCHIVE data
+            borg create --compression zstd REPO::ARCHIVE data
+            borg create --compression zstd,10 REPO::ARCHIVE data
             borg create --compression zlib REPO::ARCHIVE data
             borg create --compression zlib,1 REPO::ARCHIVE data
             borg create --compression auto,lzma,6 REPO::ARCHIVE data