From 041151275009750e977061aabe5653c1d5873fce Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 1 Oct 2016 18:23:36 +0200 Subject: [PATCH 1/4] ran build_usage --- docs/usage/common-options.rst.inc | 2 + docs/usage/create.rst.inc | 13 +-- docs/usage/delete.rst.inc | 10 +++ docs/usage/help.rst.inc | 132 +++++++++++++++++++++++++----- docs/usage/info.rst.inc | 21 ++++- docs/usage/key_export.rst.inc | 43 ++++++++++ docs/usage/key_import.rst.inc | 32 ++++++++ docs/usage/list.rst.inc | 12 ++- 8 files changed, 229 insertions(+), 36 deletions(-) create mode 100644 docs/usage/key_export.rst.inc create mode 100644 docs/usage/key_import.rst.inc diff --git a/docs/usage/common-options.rst.inc b/docs/usage/common-options.rst.inc index 29bada9d8..fb235b52a 100644 --- a/docs/usage/common-options.rst.inc +++ b/docs/usage/common-options.rst.inc @@ -24,5 +24,7 @@ | set umask to M (local and remote, default: 0077) ``--remote-path PATH`` | set remote path to executable (default: "borg") + ``--remote-ratelimit rate`` + | set remote network upload rate limit in kiByte/s (default: 0=unlimited) ``--consider-part-files`` | treat part files like normal files (e.g. to list/extract them) \ No newline at end of file diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc index 7b31c18cd..ba2dfe217 100644 --- a/docs/usage/create.rst.inc +++ b/docs/usage/create.rst.inc @@ -61,18 +61,9 @@ Archive options ``--chunker-params CHUNK_MIN_EXP,CHUNK_MAX_EXP,HASH_MASK_BITS,HASH_WINDOW_SIZE`` | specify the chunker parameters. default: 19,23,21,4095 ``-C COMPRESSION``, ``--compression COMPRESSION`` - | select compression algorithm (and level): - | none == no compression (default), - | auto,C[,L] == built-in heuristic (try with lz4 whether the data is - | compressible) decides between none or C[,L] - with C[,L] - | being any valid compression algorithm (and optional level), - | lz4 == lz4, - | zlib == zlib (default level 6), - | zlib,0 .. zlib,9 == zlib (with level 0..9), - | lzma == lzma (default level 6), - | lzma,0 .. lzma,9 == lzma (with level 0..9). + | select compression algorithm, see the output of the "borg help compression" command for details. ``--compression-from COMPRESSIONCONFIG`` - | read compression patterns from COMPRESSIONCONFIG, one per line + | read compression patterns from COMPRESSIONCONFIG, see the output of the "borg help compression" command for details. Description ~~~~~~~~~~~ diff --git a/docs/usage/delete.rst.inc b/docs/usage/delete.rst.inc index 87451ec50..2a685e885 100644 --- a/docs/usage/delete.rst.inc +++ b/docs/usage/delete.rst.inc @@ -27,6 +27,16 @@ optional arguments `Common options`_ | +filters + ``-P``, ``--prefix`` + | only consider archive names starting with this prefix + ``--sort-by`` + | Comma-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp + ``--first N`` + | consider first N archives after other filters were applied + ``--last N`` + | consider last N archives after other filters were applied + Description ~~~~~~~~~~~ diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc index b079dd2d6..572b9f23e 100644 --- a/docs/usage/help.rst.inc +++ b/docs/usage/help.rst.inc @@ -99,40 +99,134 @@ borg help placeholders ~~~~~~~~~~~~~~~~~~~~~~ - Repository (or Archive) URLs, --prefix and --remote-path values support these - placeholders: +Repository (or Archive) URLs, --prefix and --remote-path values support these +placeholders: - {hostname} +{hostname} - The (short) hostname of the machine. + The (short) hostname of the machine. - {fqdn} +{fqdn} - The full name of the machine. + The full name of the machine. - {now} +{now} - The current local date and time. + The current local date and time, by default in ISO-8601 format. + You can also supply your own `format string `_, e.g. {now:%Y-%m-%d_%H:%M:%S} - {utcnow} +{utcnow} - The current UTC date and time. + The current UTC date and time, by default in ISO-8601 format. + You can also supply your own `format string `_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S} - {user} +{user} - The user name (or UID, if no name is available) of the user running borg. + The user name (or UID, if no name is available) of the user running borg. - {pid} +{pid} - The current process ID. + The current process ID. - {borgversion} +{borgversion} - The version of borg. + The version of borg. Examples:: - borg create /path/to/repo::{hostname}-{user}-{utcnow} ... - borg create /path/to/repo::{hostname}-{now:%Y-%m-%d_%H:%M:%S} ... - borg prune --prefix '{hostname}-' ... + borg create /path/to/repo::{hostname}-{user}-{utcnow} ... + borg create /path/to/repo::{hostname}-{now:%Y-%m-%d_%H:%M:%S} ... + borg prune --prefix '{hostname}-' ... + +.. _borg_compression: + +borg help compression +~~~~~~~~~~~~~~~~~~~~~ + + +Compression is off by default, if you want some, you have to specify what you want. + +Valid compression specifiers are: + +none + + Do not compress. (default) + +lz4 + + Use lz4 compression. High speed, low compression. + +zlib[,L] + + Use zlib ("gz") compression. Medium speed, medium compression. + If you do not explicitely give the compression level L (ranging from 0 + to 9), it will use level 6. + Giving level 0 (means "no compression", but still has zlib protocol + overhead) is usually pointless, you better use "none" compression. + +lzma[,L] + + Use lzma ("xz") compression. Low speed, high compression. + If you do not explicitely give the compression level L (ranging from 0 + to 9), it will use level 6. + Giving levels above 6 is pointless and counterproductive because it does + not compress better due to the buffer size used by borg - but it wastes + lots of CPU cycles and RAM. + +auto,C[,L] + + Use a built-in heuristic to decide per chunk whether to compress or not. + The heuristic tries with lz4 whether the data is compressible. + For incompressible data, it will not use compression (uses "none"). + For compressible data, it uses the given C[,L] compression - with C[,L] + being any valid compression specifier. + +The decision about which compression to use is done by borg like this: + +1. find a compression specifier (per file): + match the path/filename against all patterns in all --compression-from + files (if any). If a pattern matches, use the compression spec given for + that pattern. If no pattern matches (and also if you do not give any + --compression-from option), default to the compression spec given by + --compression. See docs/misc/compression.conf for an example config. + +2. if the found compression spec is not "auto", the decision is taken: + use the found compression spec. + +3. if the found compression spec is "auto", test compressibility of each + chunk using lz4. + If it is compressible, use the C,[L] compression spec given within the + "auto" specifier. If it is not compressible, use no compression. + +Examples:: + + borg create --compression lz4 REPO::ARCHIVE data + borg create --compression zlib REPO::ARCHIVE data + borg create --compression zlib,1 REPO::ARCHIVE data + borg create --compression auto,lzma,6 REPO::ARCHIVE data + borg create --compression-from compression.conf --compression auto,lzma ... + +compression.conf has entries like:: + + # example config file for --compression-from option + # + # Format of non-comment / non-empty lines: + # : + # compression-spec is same format as for --compression option + # path/filename pattern is same format as for --exclude option + none:*.gz + none:*.zip + none:*.mp3 + none:*.ogg + +General remarks: + +It is no problem to mix different compression methods in one repo, +deduplication is done on the source data chunks (not on the compressed +or encrypted data). + +If some specific chunk was once compressed and stored into the repo, creating +another backup that also uses this chunk will not change the stored chunk. +So if you use different compression specs for the backups, whichever stores a +chunk first determines its compression. See also borg recreate. diff --git a/docs/usage/info.rst.inc b/docs/usage/info.rst.inc index e4a27e7b5..be61637eb 100644 --- a/docs/usage/info.rst.inc +++ b/docs/usage/info.rst.inc @@ -15,13 +15,26 @@ positional arguments `Common options`_ | +filters + ``-P``, ``--prefix`` + | only consider archive names starting with this prefix + ``--sort-by`` + | Comma-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp + ``--first N`` + | consider first N archives after other filters were applied + ``--last N`` + | consider last N archives after other filters were applied + Description ~~~~~~~~~~~ This command displays detailed information about the specified archive or repository. -The "This archive" line refers exclusively to the given archive: -"Deduplicated size" is the size of the unique chunks stored only for the -given archive. +Please note that the deduplicated sizes of the individual archives do not add +up to the deduplicated size of the repository ("all archives"), because the two +are meaning different things: -The "All archives" line shows global statistics (all chunks). +This archive / deduplicated size = amount of data stored ONLY for this archive + = unique chunks of this archive. +All archives / deduplicated size = amount of data stored in the repo + = all chunks in the repository. diff --git a/docs/usage/key_export.rst.inc b/docs/usage/key_export.rst.inc new file mode 100644 index 000000000..f251d4307 --- /dev/null +++ b/docs/usage/key_export.rst.inc @@ -0,0 +1,43 @@ +.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit! + +.. _borg_key_export: + +borg key export +--------------- +:: + + borg key export REPOSITORY PATH + +positional arguments + REPOSITORY + + PATH + where to store the backup + +optional arguments + ``--paper`` + | Create an export suitable for printing and later type-in + +`Common options`_ + | + +Description +~~~~~~~~~~~ + +If repository encryption is used, the repository is inaccessible +without the key. This command allows to backup this essential key. + +There are two backup formats. The normal backup format is suitable for +digital storage as a file. The ``--paper`` backup format is optimized +for printing and typing in while importing, with per line checks to +reduce problems with manual input. + +For repositories using keyfile encryption the key is saved locally +on the system that is capable of doing backups. To guard against loss +of this key, the key needs to be backed up independently of the main +data backup. + +For repositories using the repokey encryption the key is saved in the +repository in the config file. A backup is thus not strictly needed, +but guards against the repository becoming inaccessible if the file +is damaged for some reason. diff --git a/docs/usage/key_import.rst.inc b/docs/usage/key_import.rst.inc new file mode 100644 index 000000000..c8c2a0a85 --- /dev/null +++ b/docs/usage/key_import.rst.inc @@ -0,0 +1,32 @@ +.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit! + +.. _borg_key_import: + +borg key import +--------------- +:: + + borg key import REPOSITORY PATH + +positional arguments + REPOSITORY + + PATH + path to the backup + +optional arguments + ``--paper`` + | interactively import from a backup done with --paper + +`Common options`_ + | + +Description +~~~~~~~~~~~ + +This command allows to restore a key previously backed up with the +export command. + +If the ``--paper`` option is given, the import will be an interactive +process in which each line is checked for plausibility before +proceeding to the next line. For this format PATH must not be given. diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc index 8e32df6ab..ce6bd8042 100644 --- a/docs/usage/list.rst.inc +++ b/docs/usage/list.rst.inc @@ -20,8 +20,6 @@ optional arguments ``--format``, ``--list-format`` | specify format for file listing | (default: "{mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NL}") - ``-P``, ``--prefix`` - | only consider archive names starting with this prefix ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN ``--exclude-from EXCLUDEFILE`` @@ -30,6 +28,16 @@ optional arguments `Common options`_ | +filters + ``-P``, ``--prefix`` + | only consider archive names starting with this prefix + ``--sort-by`` + | Comma-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp + ``--first N`` + | consider first N archives after other filters were applied + ``--last N`` + | consider last N archives after other filters were applied + Description ~~~~~~~~~~~ From f57feb121d10d18c3dbc02a9e887620f4c527138 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 1 Oct 2016 18:29:45 +0200 Subject: [PATCH 2/4] fix module names in api.rst --- docs/api.rst | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/api.rst b/docs/api.rst index ab634e864..7ce030192 100644 --- a/docs/api.rst +++ b/docs/api.rst @@ -2,94 +2,94 @@ API Documentation ================= -.. automodule:: src.borg.archiver +.. automodule:: borg.archiver :members: :undoc-members: -.. automodule:: src.borg.archive +.. automodule:: borg.archive :members: :undoc-members: -.. automodule:: src.borg.repository +.. automodule:: borg.repository :members: :undoc-members: -.. automodule:: src.borg.remote +.. automodule:: borg.remote :members: :undoc-members: -.. automodule:: src.borg.cache +.. automodule:: borg.cache :members: :undoc-members: -.. automodule:: src.borg.key +.. automodule:: borg.key :members: :undoc-members: -.. automodule:: src.borg.nonces +.. automodule:: borg.nonces :members: :undoc-members: -.. automodule:: src.borg.item +.. automodule:: borg.item :members: :undoc-members: -.. automodule:: src.borg.constants +.. automodule:: borg.constants :members: :undoc-members: -.. automodule:: src.borg.logger +.. automodule:: borg.logger :members: :undoc-members: -.. automodule:: src.borg.helpers +.. automodule:: borg.helpers :members: :undoc-members: -.. automodule:: src.borg.locking +.. automodule:: borg.locking :members: :undoc-members: -.. automodule:: src.borg.shellpattern +.. automodule:: borg.shellpattern :members: :undoc-members: -.. automodule:: src.borg.lrucache +.. automodule:: borg.lrucache :members: :undoc-members: -.. automodule:: src.borg.fuse +.. automodule:: borg.fuse :members: :undoc-members: -.. automodule:: src.borg.selftest +.. automodule:: borg.selftest :members: :undoc-members: -.. automodule:: src.borg.xattr +.. automodule:: borg.xattr :members: :undoc-members: -.. automodule:: src.borg.platform.base +.. automodule:: borg.platform.base :members: :undoc-members: -.. automodule:: src.borg.platform.linux +.. automodule:: borg.platform.linux :members: :undoc-members: -.. automodule:: src.borg.hashindex +.. automodule:: borg.hashindex :members: :undoc-members: -.. automodule:: src.borg.compress +.. automodule:: borg.compress :members: get_compressor, Compressor, CompressorBase :undoc-members: -.. automodule:: src.borg.chunker +.. automodule:: borg.chunker :members: :undoc-members: -.. automodule:: src.borg.crypto +.. automodule:: borg.crypto :members: :undoc-members: From e6241cce125d977a26fa86ba7f054019fddf2117 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 1 Oct 2016 18:44:52 +0200 Subject: [PATCH 3/4] update CHANGES --- docs/changes.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/changes.rst b/docs/changes.rst index f2510b145..dbeb110b8 100644 --- a/docs/changes.rst +++ b/docs/changes.rst @@ -50,8 +50,8 @@ The best check that everything is ok is to run a dry-run extraction:: borg extract -v --dry-run REPO::ARCHIVE -Version 1.1.0b2 (not released yet) ----------------------------------- +Version 1.1.0b2 (2016-10-01) +---------------------------- Bug fixes: @@ -71,13 +71,19 @@ New features: - borg help compression, #1582 - borg check: delete chunks with integrity errors, #1575, so they can be "repaired" immediately and maybe healed later. +- archives filters concept (refactoring/unifying older code) + + - covers --first/--last/--prefix/--sort-by options + - currently used for borg list/info/delete Other changes: - borg check --verify-data slightly tuned (use get_many()) -- Change {utcnow} and {now} to ISO-8601 format ("T" date/time separator) +- change {utcnow} and {now} to ISO-8601 format ("T" date/time separator) - repo check: log transaction IDs, improve object count mismatch diagnostic - Vagrantfile: use TW's fresh-bootloader pyinstaller branch +- fix module names in api.rst +- hashindex: bump api_version Version 1.1.0b1 (2016-08-28) From 5baf749b29c79f7c3df2ef5b37bdc6f9d4086c4f Mon Sep 17 00:00:00 2001 From: Marian Beermann Date: Sat, 1 Oct 2016 20:17:30 +0200 Subject: [PATCH 4/4] vagrant freebsd64: fix tox not finding any pythons --- Vagrantfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Vagrantfile b/Vagrantfile index c2bb41428..7980d9bd4 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -277,6 +277,7 @@ def run_tests(boxname) if which pyenv > /dev/null; then # for testing, use the earliest point releases of the supported python versions: pyenv global 3.4.0 3.5.0 + pyenv local 3.4.0 3.5.0 fi # otherwise: just use the system python if which fakeroot > /dev/null; then