mirror of https://github.com/borgbackup/borg.git
Merge pull request #3461 from ThomasWaldmann/docs-lz4-b2-zstd
docs update for lz4 b2 zstd changes
This commit is contained in:
commit
79b2f2f653
|
@ -58,8 +58,13 @@ Main features
|
||||||
authenticity is verified using HMAC-SHA256. Data is encrypted clientside.
|
authenticity is verified using HMAC-SHA256. Data is encrypted clientside.
|
||||||
|
|
||||||
**Compression**
|
**Compression**
|
||||||
All data can be compressed by lz4 (super fast, low compression), zlib
|
All data can be optionally compressed:
|
||||||
(medium speed and compression) or lzma (low speed, high compression).
|
|
||||||
|
* lz4 (super fast, low compression)
|
||||||
|
* zstd (wide range from high speed and low compression to high compression
|
||||||
|
and lower speed)
|
||||||
|
* zlib (medium speed and compression)
|
||||||
|
* lzma (low speed, high compression)
|
||||||
|
|
||||||
**Off-site backups**
|
**Off-site backups**
|
||||||
Borg can store data on any remote host accessible over SSH. If Borg is
|
Borg can store data on any remote host accessible over SSH. If Borg is
|
||||||
|
|
|
@ -171,8 +171,6 @@ Salt running on a Debian system.
|
||||||
- openssl
|
- openssl
|
||||||
- libacl1-dev
|
- libacl1-dev
|
||||||
- libacl1
|
- libacl1
|
||||||
- liblz4-dev
|
|
||||||
- liblz4-1
|
|
||||||
- build-essential
|
- build-essential
|
||||||
- libfuse-dev
|
- libfuse-dev
|
||||||
- fuse
|
- fuse
|
||||||
|
|
|
@ -788,7 +788,7 @@ Here's a (incomplete) list of some major changes:
|
||||||
* more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_)
|
* more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_)
|
||||||
* less chunk management overhead (less memory and disk usage for chunks index)
|
* less chunk management overhead (less memory and disk usage for chunks index)
|
||||||
* faster remote cache resync (useful when backing up multiple machines into same repo)
|
* faster remote cache resync (useful when backing up multiple machines into same repo)
|
||||||
* compression: no, lz4, zlib or lzma compression, adjustable compression levels
|
* compression: no, lz4, zstd, zlib or lzma compression, adjustable compression levels
|
||||||
* repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode)
|
* repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode)
|
||||||
* simple sparse file support, great for virtual machine disk files
|
* simple sparse file support, great for virtual machine disk files
|
||||||
* can read special files (e.g. block devices) or from stdin, write to stdout
|
* can read special files (e.g. block devices) or from stdin, write to stdout
|
||||||
|
@ -840,8 +840,7 @@ The default in Borg is lz4, which is fast enough to not use significant CPU time
|
||||||
in most cases, but can only achieve modest compression. It still compresses
|
in most cases, but can only achieve modest compression. It still compresses
|
||||||
easily compressed data fairly well.
|
easily compressed data fairly well.
|
||||||
|
|
||||||
zlib compression with all levels (1-9) as well as LZMA (1-6) are available
|
Borg also offers zstd, zlib and lzma compression, choose wisely.
|
||||||
as well, for cases where they are worth it.
|
|
||||||
|
|
||||||
Which choice is the best option depends on a number of factors, like
|
Which choice is the best option depends on a number of factors, like
|
||||||
bandwidth to the repository, how well the data compresses, available CPU
|
bandwidth to the repository, how well the data compresses, available CPU
|
||||||
|
|
|
@ -14,6 +14,7 @@
|
||||||
.. _libacl: https://savannah.nongnu.org/projects/acl/
|
.. _libacl: https://savannah.nongnu.org/projects/acl/
|
||||||
.. _libattr: https://savannah.nongnu.org/projects/attr/
|
.. _libattr: https://savannah.nongnu.org/projects/attr/
|
||||||
.. _liblz4: https://github.com/Cyan4973/lz4
|
.. _liblz4: https://github.com/Cyan4973/lz4
|
||||||
|
.. _libzstd: https://github.com/facebook/zstd
|
||||||
.. _libb2: https://github.com/BLAKE2/libb2
|
.. _libb2: https://github.com/BLAKE2/libb2
|
||||||
.. _ZeroMQ: http://zeromq.org/
|
.. _ZeroMQ: http://zeromq.org/
|
||||||
.. _OpenSSL: https://www.openssl.org/
|
.. _OpenSSL: https://www.openssl.org/
|
||||||
|
|
|
@ -141,12 +141,17 @@ following dependencies first:
|
||||||
optional install.
|
optional install.
|
||||||
* OpenSSL_ >= 1.0.0, plus development headers.
|
* OpenSSL_ >= 1.0.0, plus development headers.
|
||||||
* libacl_ (which depends on libattr_), both plus development headers.
|
* libacl_ (which depends on libattr_), both plus development headers.
|
||||||
* liblz4_ >= r129 (1.7.0), plus development headers.
|
|
||||||
* ZeroMQ_ >= 4.0.0, plus development headers.
|
* ZeroMQ_ >= 4.0.0, plus development headers.
|
||||||
|
* We have bundled code of the following packages, but borg by default (see
|
||||||
|
setup.py if you want to change that) prefers a shared library if it can
|
||||||
|
be found on the system (lib + dev headers) at build time:
|
||||||
|
|
||||||
|
- liblz4_ >= 1.7.0 (r129)
|
||||||
|
- libzstd_ >= 1.3.0
|
||||||
|
- libb2_
|
||||||
* some Python dependencies, pip will automatically install them for you
|
* some Python dependencies, pip will automatically install them for you
|
||||||
* optionally, the llfuse_ Python package is required if you wish to mount an
|
* optionally, the llfuse_ Python package is required if you wish to mount an
|
||||||
archive as a FUSE filesystem. See setup.py about the version requirements.
|
archive as a FUSE filesystem. See setup.py about the version requirements.
|
||||||
* optionally libb2_. If it is not found a bundled implementation is used instead.
|
|
||||||
|
|
||||||
If you have troubles finding the right package names, have a look at the
|
If you have troubles finding the right package names, have a look at the
|
||||||
distribution specific sections below or the Vagrantfile in the git repository,
|
distribution specific sections below or the Vagrantfile in the git repository,
|
||||||
|
@ -169,7 +174,6 @@ Install the dependencies with development headers::
|
||||||
sudo apt-get install python3 python3-dev python3-pip python-virtualenv \
|
sudo apt-get install python3 python3-dev python3-pip python-virtualenv \
|
||||||
libssl-dev openssl \
|
libssl-dev openssl \
|
||||||
libacl1-dev libacl1 \
|
libacl1-dev libacl1 \
|
||||||
liblz4-dev liblz4-1 \
|
|
||||||
libzmq3-dev libzmq3 \
|
libzmq3-dev libzmq3 \
|
||||||
build-essential
|
build-essential
|
||||||
sudo apt-get install libfuse-dev fuse pkg-config # optional, for FUSE support
|
sudo apt-get install libfuse-dev fuse pkg-config # optional, for FUSE support
|
||||||
|
@ -188,7 +192,6 @@ Install the dependencies with development headers::
|
||||||
sudo dnf install python3 python3-devel python3-pip python3-virtualenv
|
sudo dnf install python3 python3-devel python3-pip python3-virtualenv
|
||||||
sudo dnf install openssl-devel openssl
|
sudo dnf install openssl-devel openssl
|
||||||
sudo dnf install libacl-devel libacl
|
sudo dnf install libacl-devel libacl
|
||||||
sudo dnf install lz4-devel
|
|
||||||
sudo dnf install gcc gcc-c++
|
sudo dnf install gcc gcc-c++
|
||||||
sudo dnf install redhat-rpm-config # not needed in Korora
|
sudo dnf install redhat-rpm-config # not needed in Korora
|
||||||
sudo dnf install fuse-devel fuse pkgconfig # optional, for FUSE support
|
sudo dnf install fuse-devel fuse pkgconfig # optional, for FUSE support
|
||||||
|
@ -205,7 +208,7 @@ Install the dependencies automatically using zypper::
|
||||||
Alternatively, you can enumerate all build dependencies in the command line::
|
Alternatively, you can enumerate all build dependencies in the command line::
|
||||||
|
|
||||||
sudo zypper install python3 python3-devel \
|
sudo zypper install python3 python3-devel \
|
||||||
libacl-devel liblz4-devel openssl-devel \
|
libacl-devel openssl-devel \
|
||||||
python3-Cython python3-Sphinx python3-msgpack-python \
|
python3-Cython python3-Sphinx python3-msgpack-python \
|
||||||
python3-pytest python3-setuptools python3-setuptools_scm \
|
python3-pytest python3-setuptools python3-setuptools_scm \
|
||||||
python3-sphinx_rtd_theme python3-llfuse gcc gcc-c++
|
python3-sphinx_rtd_theme python3-llfuse gcc gcc-c++
|
||||||
|
@ -218,7 +221,7 @@ Mac OS X
|
||||||
Assuming you have installed homebrew_, the following steps will install all the
|
Assuming you have installed homebrew_, the following steps will install all the
|
||||||
dependencies::
|
dependencies::
|
||||||
|
|
||||||
brew install python3 lz4 openssl
|
brew install python3 openssl
|
||||||
brew install pkg-config # optional, for FUSE support
|
brew install pkg-config # optional, for FUSE support
|
||||||
pip3 install virtualenv
|
pip3 install virtualenv
|
||||||
|
|
||||||
|
@ -238,7 +241,7 @@ and commands to make FUSE work for using the mount command.
|
||||||
|
|
||||||
::
|
::
|
||||||
|
|
||||||
pkg install -y python3 openssl liblz4 fusefs-libs pkgconf
|
pkg install -y python3 openssl fusefs-libs pkgconf
|
||||||
pkg install -y git
|
pkg install -y git
|
||||||
python3.4 -m ensurepip # to install pip for Python3
|
python3.4 -m ensurepip # to install pip for Python3
|
||||||
To use the mount command:
|
To use the mount command:
|
||||||
|
@ -270,7 +273,6 @@ Use the Cygwin installer to install the dependencies::
|
||||||
python3 python3-devel python3-setuptools
|
python3 python3-devel python3-setuptools
|
||||||
binutils gcc-g++
|
binutils gcc-g++
|
||||||
libopenssl openssl-devel
|
libopenssl openssl-devel
|
||||||
liblz4_1 liblz4-devel
|
|
||||||
git make openssh
|
git make openssh
|
||||||
|
|
||||||
You can then install ``pip`` and ``virtualenv``::
|
You can then install ``pip`` and ``virtualenv``::
|
||||||
|
|
|
@ -913,15 +913,16 @@ Borg supports the following compression methods:
|
||||||
|
|
||||||
- none (no compression, pass through data 1:1)
|
- none (no compression, pass through data 1:1)
|
||||||
- lz4 (low compression, but super fast)
|
- lz4 (low compression, but super fast)
|
||||||
|
- zstd (level 1-22 offering a wide range: level 1 is lower compression and high
|
||||||
|
speed, level 22 is higher compression and lower speed) - since borg 1.1.4
|
||||||
- zlib (level 0-9, level 0 is no compression [but still adding zlib overhead],
|
- zlib (level 0-9, level 0 is no compression [but still adding zlib overhead],
|
||||||
level 1 is low, level 9 is high compression)
|
level 1 is low, level 9 is high compression)
|
||||||
- lzma (level 0-9, level 0 is low, level 9 is high compression).
|
- lzma (level 0-9, level 0 is low, level 9 is high compression).
|
||||||
|
|
||||||
Speed: none > lz4 > zlib > lzma
|
Speed: none > lz4 > zlib > lzma, lz4 > zstd
|
||||||
Compression: lzma > zlib > lz4 > none
|
Compression: lzma > zlib > lz4 > none, zstd > lz4
|
||||||
|
|
||||||
Be careful, higher zlib and especially lzma compression levels might take a
|
Be careful, higher compression levels might use a lot of resources (CPU/memory).
|
||||||
lot of resources (CPU and memory).
|
|
||||||
|
|
||||||
The overall speed of course also depends on the speed of your target storage.
|
The overall speed of course also depends on the speed of your target storage.
|
||||||
If that is slow, using a higher compression level might yield better overall
|
If that is slow, using a higher compression level might yield better overall
|
||||||
|
|
|
@ -186,6 +186,17 @@ Backup compression
|
||||||
The default is lz4 (very fast, but low compression ratio), but other methods are
|
The default is lz4 (very fast, but low compression ratio), but other methods are
|
||||||
supported for different situations.
|
supported for different situations.
|
||||||
|
|
||||||
|
You can use zstd for a wide range from high speed (and relatively low
|
||||||
|
compression) using N=1 to high compression (and lower speed) using N=22.
|
||||||
|
|
||||||
|
zstd is a modern compression algorithm and might be preferable over zlib and
|
||||||
|
lzma, except if you need compatibility to older borg versions (< 1.1.4) that
|
||||||
|
did not yet offer zstd.
|
||||||
|
|
||||||
|
$ borg create --compression zstd,N /path/to/repo::arch ~
|
||||||
|
|
||||||
|
Other options are:
|
||||||
|
|
||||||
If you have a fast repo storage and you want minimum CPU usage, no compression::
|
If you have a fast repo storage and you want minimum CPU usage, no compression::
|
||||||
|
|
||||||
$ borg create --compression none /path/to/repo::arch ~
|
$ borg create --compression none /path/to/repo::arch ~
|
||||||
|
|
|
@ -1968,7 +1968,13 @@ class Archiver:
|
||||||
Do not compress.
|
Do not compress.
|
||||||
|
|
||||||
lz4
|
lz4
|
||||||
Use lz4 compression. High speed, low compression. (default)
|
Use lz4 compression. Very high speed, very low compression. (default)
|
||||||
|
|
||||||
|
zstd[,L]
|
||||||
|
Use zstd ("zstandard") compression, a modern wide-range algorithm.
|
||||||
|
If you do not explicitely give the compression level L (ranging from 1
|
||||||
|
to 22), it will use level 3.
|
||||||
|
Archives compressed with zstd are not compatible with borg < 1.1.4.
|
||||||
|
|
||||||
zlib[,L]
|
zlib[,L]
|
||||||
Use zlib ("gz") compression. Medium speed, medium compression.
|
Use zlib ("gz") compression. Medium speed, medium compression.
|
||||||
|
@ -1995,6 +2001,8 @@ class Archiver:
|
||||||
Examples::
|
Examples::
|
||||||
|
|
||||||
borg create --compression lz4 REPO::ARCHIVE data
|
borg create --compression lz4 REPO::ARCHIVE data
|
||||||
|
borg create --compression zstd REPO::ARCHIVE data
|
||||||
|
borg create --compression zstd,10 REPO::ARCHIVE data
|
||||||
borg create --compression zlib REPO::ARCHIVE data
|
borg create --compression zlib REPO::ARCHIVE data
|
||||||
borg create --compression zlib,1 REPO::ARCHIVE data
|
borg create --compression zlib,1 REPO::ARCHIVE data
|
||||||
borg create --compression auto,lzma,6 REPO::ARCHIVE data
|
borg create --compression auto,lzma,6 REPO::ARCHIVE data
|
||||||
|
|
Loading…
Reference in New Issue