From 56c599e6dc3780b7b539172f497997043c4f90c0 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sun, 12 Apr 2020 19:51:49 +0200 Subject: [PATCH] build_usage --- docs/usage/check.rst.inc | 71 ++++++++++++++++--------------- docs/usage/common-options.rst.inc | 1 + docs/usage/compact.rst.inc | 33 ++++++++------ docs/usage/create.rst.inc | 7 ++- docs/usage/delete.rst.inc | 10 +++-- docs/usage/extract.rst.inc | 7 ++- docs/usage/help.rst.inc | 10 +++++ docs/usage/list.rst.inc | 3 ++ docs/usage/mount.rst.inc | 3 ++ 9 files changed, 89 insertions(+), 56 deletions(-) diff --git a/docs/usage/check.rst.inc b/docs/usage/check.rst.inc index 5843fccc9..e6c2717a0 100644 --- a/docs/usage/check.rst.inc +++ b/docs/usage/check.rst.inc @@ -89,55 +89,56 @@ The check command verifies the consistency of a repository and the corresponding First, the underlying repository data files are checked: -- For all segments the segment magic (header) is checked -- For all objects stored in the segments, all metadata (e.g. crc and size) and +- For all segments, the segment magic header is checked. +- For all objects stored in the segments, all metadata (e.g. CRC and size) and all data is read. The read data is checked by size and CRC. Bit rot and other types of accidental damage can be detected this way. -- If we are in repair mode and a integrity error is detected for a segment, - we try to recover as many objects from the segment as possible. -- In repair mode, it makes sure that the index is consistent with the data - stored in the segments. -- If you use a remote repo server via ssh:, the repo check is executed on the - repo server without causing significant network traffic. +- In repair mode, if an integrity error is detected in a segment, try to recover + as many objects from the segment as possible. +- In repair mode, make sure that the index is consistent with the data stored in + the segments. +- If checking a remote repo via ``ssh:``, the repo check is executed on the server + without causing significant network traffic. - The repository check can be skipped using the ``--archives-only`` option. -- A repository check can be time consuming. Partial checks are possible with the ``--max-duration`` option. +- A repository check can be time consuming. Partial checks are possible with the + ``--max-duration`` option. Second, the consistency and correctness of the archive metadata is verified: - Is the repo manifest present? If not, it is rebuilt from archive metadata chunks (this requires reading and decrypting of all metadata and data). -- Check if archive metadata chunk is present. if not, remove archive from - manifest. +- Check if archive metadata chunk is present; if not, remove archive from manifest. - For all files (items) in the archive, for all chunks referenced by these - files, check if chunk is present. - If a chunk is not present and we are in repair mode, replace it with a same-size - replacement chunk of zeros. - If a previously lost chunk reappears (e.g. via a later backup) and we are in - repair mode, the all-zero replacement chunk will be replaced by the correct chunk. - This requires reading of archive and file metadata, but not data. -- If we are in repair mode and we checked all the archives: delete orphaned - chunks from the repo. -- if you use a remote repo server via ssh:, the archive check is executed on - the client machine (because if encryption is enabled, the checks will require - decryption and this is always done client-side, because key access will be - required). -- The archive checks can be time consuming, they can be skipped using the + files, check if chunk is present. In repair mode, if a chunk is not present, + replace it with a same-size replacement chunk of zeroes. If a previously lost + chunk reappears (e.g. via a later backup), in repair mode the all-zero replacement + chunk will be replaced by the correct chunk. This requires reading of archive and + file metadata, but not data. +- In repair mode, when all the archives were checked, orphaned chunks are deleted + from the repo. One cause of orphaned chunks are input file related errors (like + read errors) in the archive creation process. +- If checking a remote repo via ``ssh:``, the archive check is executed on the + client machine because it requires decryption, and this is always done client-side + as key access is needed. +- The archive checks can be time consuming; they can be skipped using the ``--repository-only`` option. -The ``--max-duration`` option can be used to split a long-running repository check into multiple partial checks. -After the given number of seconds the check is interrupted. The next partial check will continue where the -previous one stopped, until the complete repository has been checked. Example: Assuming a full check took 7 -hours, then running a daily check with --max-duration=3600 (1 hour) would result in one full check per week. +The ``--max-duration`` option can be used to split a long-running repository check +into multiple partial checks. After the given number of seconds the check is +interrupted. The next partial check will continue where the previous one stopped, +until the complete repository has been checked. Example: Assuming a full check took 7 +hours, then running a daily check with --max-duration=3600 (1 hour) resulted in one +full check per week. -Attention: Partial checks can only do way less checks than a full check (only the CRC32 checks on segment file -entries are done) and cannot be combined with ``--repair``. Partial checks may therefore be useful only with very -large repositories where a full check would take too long. Doing a full repository check aborts a partial check; -the next partial check will start from the beginning. +Attention: Partial checks can only do way less checking than a full check (only the +CRC32 checks on segment file entries are done), and cannot be combined with the +``--repair`` option. Partial checks may therefore be useful only with very large +repositories where a full check took too long. Doing a full repository check aborts a +partial check; the next partial check will restart from the beginning. The ``--verify-data`` option will perform a full integrity verification (as opposed to checking the CRC32 of the segment) of data, which means reading the data from the repository, decrypting and decompressing it. This is a cryptographic verification, which will detect (accidental) corruption. For encrypted repositories it is -tamper-resistant as well, unless the attacker has access to the keys. - -It is also very slow. \ No newline at end of file +tamper-resistant as well, unless the attacker has access to the keys. It is also very +slow. \ No newline at end of file diff --git a/docs/usage/common-options.rst.inc b/docs/usage/common-options.rst.inc index 07ce07a5b..41fededf1 100644 --- a/docs/usage/common-options.rst.inc +++ b/docs/usage/common-options.rst.inc @@ -8,6 +8,7 @@ -p, --progress show progress information --log-json Output one JSON object per log line instead of formatted text. --lock-wait SECONDS wait at most SECONDS for acquiring a repository/cache lock (default: 1). +--bypass-lock Bypass locking mechanism --show-version show/log the borg version --show-rc show/log the return code (rc) --umask M set umask to M (local and remote, default: 0077) diff --git a/docs/usage/compact.rst.inc b/docs/usage/compact.rst.inc index 90fb93529..01ae8e0fd 100644 --- a/docs/usage/compact.rst.inc +++ b/docs/usage/compact.rst.inc @@ -12,19 +12,21 @@ borg compact .. class:: borg-options-table - +-------------------------------------------------------+-----------------------+-------------------------------------------+ - | **positional arguments** | - +-------------------------------------------------------+-----------------------+-------------------------------------------+ - | | ``REPOSITORY`` | repository to compact | - +-------------------------------------------------------+-----------------------+-------------------------------------------+ - | **optional arguments** | - +-------------------------------------------------------+-----------------------+-------------------------------------------+ - | | ``--cleanup-commits`` | cleanup commit-only 17-byte segment files | - +-------------------------------------------------------+-----------------------+-------------------------------------------+ - | .. class:: borg-common-opt-ref | - | | - | :ref:`common_options` | - +-------------------------------------------------------+-----------------------+-------------------------------------------+ + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ + | **positional arguments** | + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ + | | ``REPOSITORY`` | repository to compact | + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ + | **optional arguments** | + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ + | | ``--cleanup-commits`` | cleanup commit-only 17-byte segment files | + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ + | | ``--threshold PERCENT`` | set minimum threshold for saved space in PERCENT (Default: 10) | + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ + | .. class:: borg-common-opt-ref | + | | + | :ref:`common_options` | + +-------------------------------------------------------+-------------------------+----------------------------------------------------------------+ .. raw:: html @@ -41,7 +43,8 @@ borg compact optional arguments - --cleanup-commits cleanup commit-only 17-byte segment files + --cleanup-commits cleanup commit-only 17-byte segment files + --threshold PERCENT set minimum threshold for saved space in PERCENT (Default: 10) :ref:`common_options` @@ -62,6 +65,8 @@ client or also from the server. Depending on the amount of segments that need compaction, it may take a while, so consider using the ``--progress`` option. +A segment is compacted if the amount of saved space is above the percentage value +given by the ``--threshold`` option. If ommitted, a threshold of 10% is used. When using ``--verbose``, borg will output an estimate of the freed space. After upgrading borg (server) to 1.2+, you can use ``borg compact --cleanup-commits`` diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc index e48e41148..effab7abe 100644 --- a/docs/usage/create.rst.inc +++ b/docs/usage/create.rst.inc @@ -71,7 +71,9 @@ borg create +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--nobirthtime`` | do not store birthtime (creation date) into archive | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--nobsdflags`` | do not read and store bsdflags (e.g. NODUMP, IMMUTABLE) into archive | + | | ``--nobsdflags`` | deprecated, use ``--noflags`` instead | + +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--noflags`` | do not read and store flags (e.g. NODUMP, IMMUTABLE) into archive | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--files-cache MODE`` | operate files cache in MODE. default: ctime,size,inode | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -137,7 +139,8 @@ borg create --atime do store atime into archive --noctime do not store ctime into archive --nobirthtime do not store birthtime (creation date) into archive - --nobsdflags do not read and store bsdflags (e.g. NODUMP, IMMUTABLE) into archive + --nobsdflags deprecated, use ``--noflags`` instead + --noflags do not read and store flags (e.g. NODUMP, IMMUTABLE) into archive --files-cache MODE operate files cache in MODE. default: ctime,size,inode --read-special open and read block and char device files as well as FIFOs as if they were regular files. Also follows symlinks pointing to these kinds of files. diff --git a/docs/usage/delete.rst.inc b/docs/usage/delete.rst.inc index a33a9ae17..060f5e76d 100644 --- a/docs/usage/delete.rst.inc +++ b/docs/usage/delete.rst.inc @@ -29,6 +29,8 @@ borg delete +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--force`` | force deletion of corrupted archives, use ``--force --force`` in case ``--force`` does not work. | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--keep-security-info`` | keep the local security info when deleting a repository | + +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--save-space`` | work slower, but using less space | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | @@ -69,6 +71,7 @@ borg delete -s, --stats print statistics for the deleted archive --cache-only delete only the local cache for the given repository --force force deletion of corrupted archives, use ``--force --force`` in case ``--force`` does not work. + --keep-security-info keep the local security info when deleting a repository --save-space work slower, but using less space @@ -91,9 +94,10 @@ This command deletes an archive from the repository or the complete repository. Important: When deleting archives, repository disk space is **not** freed until you run ``borg compact``. -If you delete the complete repository, the local cache for it (if any) is -also deleted. Alternatively, you can delete just the local cache with the -``--cache-only`` option. +When you delete a complete repository, the security info and local cache for it +(if any) are also deleted. Alternatively, you can delete just the local cache +with the ``--cache-only`` option, or keep the security info with the +``--keep-security-info`` option. When using ``--stats``, you will get some statistics about how much data was deleted - the "Deleted data" deduplicated size there is most interesting as diff --git a/docs/usage/extract.rst.inc b/docs/usage/extract.rst.inc index 495f20fca..c0e86b0cc 100644 --- a/docs/usage/extract.rst.inc +++ b/docs/usage/extract.rst.inc @@ -27,7 +27,9 @@ borg extract +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``--numeric-owner`` | only obey numeric user and group identifiers | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ - | | ``--nobsdflags`` | do not extract/set bsdflags (e.g. NODUMP, IMMUTABLE) | + | | ``--nobsdflags`` | deprecated, use ``--noflags`` instead | + +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ + | | ``--noflags`` | do not extract/set flags (e.g. NODUMP, IMMUTABLE) | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``--stdout`` | write all extracted data to stdout | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ @@ -70,7 +72,8 @@ borg extract --list output verbose list of items (files, dirs, ...) -n, --dry-run do not actually change any files --numeric-owner only obey numeric user and group identifiers - --nobsdflags do not extract/set bsdflags (e.g. NODUMP, IMMUTABLE) + --nobsdflags deprecated, use ``--noflags`` instead + --noflags do not extract/set flags (e.g. NODUMP, IMMUTABLE) --stdout write all extracted data to stdout --sparse create holes in output sparse file from all-zero chunks diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc index 88c9bf70c..ccc1d2dd0 100644 --- a/docs/usage/help.rst.inc +++ b/docs/usage/help.rst.inc @@ -96,6 +96,9 @@ selector prefix is also supported for patterns loaded from a file. Due to whitespace removal paths with whitespace at the beginning or end can only be excluded using regular expressions. +To test your exclusion patterns without performing an actual backup you can +run ``borg create --list --dry-run ...``. + Examples:: # Exclude '/home/user/file.o' but not '/home/user/file.odt': @@ -134,6 +137,13 @@ Examples:: A root path starts with the prefix `R`, followed by a path (a plain path, not a file pattern). An include rule starts with the prefix +, an exclude rule starts with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern. + + .. note:: + + Via ``--pattern`` or ``--patterns-from`` you can define BOTH inclusion and exclusion + of files using pattern prefixes ``+`` and ``-``. With ``--exclude`` and + ``--exlude-from`` ONLY excludes are defined. + Inclusion patterns are useful to include paths that are contained in an excluded path. The first matching pattern is used so if an include pattern matches before an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc index b4f6db468..709ca0bc7 100644 --- a/docs/usage/list.rst.inc +++ b/docs/usage/list.rst.inc @@ -21,6 +21,8 @@ borg list +-----------------------------------------------------------------------------+-----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | **optional arguments** | +-----------------------------------------------------------------------------+-----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--consider-checkpoints`` | Show checkpoint archives in the repository contents list (default: hidden). | + +-----------------------------------------------------------------------------+-----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--short`` | only print file/directory names, nothing else | +-----------------------------------------------------------------------------+-----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--format FORMAT``, ``--list-format FORMAT`` | specify format for file listing (default: "{mode} {user:6} {group:6} {size:8d} {mtime} {path}{extra}{NL}") | @@ -73,6 +75,7 @@ borg list optional arguments + --consider-checkpoints Show checkpoint archives in the repository contents list (default: hidden). --short only print file/directory names, nothing else --format FORMAT, --list-format FORMAT specify format for file listing (default: "{mode} {user:6} {group:6} {size:8d} {mtime} {path}{extra}{NL}") --json Only valid for listing repository contents. Format output as JSON. The form of ``--format`` is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "barchive" key is therefore not available. diff --git a/docs/usage/mount.rst.inc b/docs/usage/mount.rst.inc index f813f5583..1b297fd1c 100644 --- a/docs/usage/mount.rst.inc +++ b/docs/usage/mount.rst.inc @@ -23,6 +23,8 @@ borg mount +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | **optional arguments** | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--consider-checkpoints`` | Show checkpoint archives in the repository contents list (default: hidden). | + +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-f``, ``--foreground`` | stay in foreground, do not daemonize | +-----------------------------------------------------------------------------+---------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-o`` | Extra mount options | @@ -75,6 +77,7 @@ borg mount optional arguments + --consider-checkpoints Show checkpoint archives in the repository contents list (default: hidden). -f, --foreground stay in foreground, do not daemonize -o Extra mount options