From 61299d23db93cf73a1b67bd23d2828017f198fe9 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 18 May 2022 17:31:10 +0200 Subject: [PATCH 01/22] Revert "Remove scp syntax for locations (#6697)" This reverts commit 1b4b84dfd8f1ec2a3aa156fb9cc2714ef6263143. --- docs/deployment/central-backup-server.rst | 8 ++-- docs/quickstart.rst | 2 +- docs/usage/general/repository-urls.rst.inc | 46 ++++++++-------------- docs/usage/init.rst | 4 +- src/borg/helpers/parseformat.py | 17 +++++--- src/borg/testsuite/archiver.py | 2 +- src/borg/testsuite/helpers.py | 37 +++++++++++++++++ src/borg/testsuite/repository.py | 11 +++--- 8 files changed, 80 insertions(+), 47 deletions(-) diff --git a/docs/deployment/central-backup-server.rst b/docs/deployment/central-backup-server.rst index 740c6a6f..431bd531 100644 --- a/docs/deployment/central-backup-server.rst +++ b/docs/deployment/central-backup-server.rst @@ -81,7 +81,7 @@ The options which are added to the key will perform the following: Due to the ``cd`` command we use, the server automatically changes the current working directory. Then client doesn't need to have knowledge of the absolute or relative remote repository path and can directly access the repositories at -``ssh://@/./``. +``@:``. .. note:: The setup above ignores all client given commandline parameters which are normally appended to the `borg serve` command. @@ -93,21 +93,21 @@ The client needs to initialize the `pictures` repository like this: :: - borg init ssh://backup@backup01.srv.local/./pictures + borg init backup@backup01.srv.local:pictures Or with the full path (should actually never be used, as only for demonstrational purposes). The server should automatically change the current working directory to the `` folder. :: - borg init ssh://backup@backup01.srv.local/home/backup/repos/johndoe.clnt.local/pictures + borg init backup@backup01.srv.local:/home/backup/repos/johndoe.clnt.local/pictures When `johndoe.clnt.local` tries to access a not restricted path the following error is raised. John Doe tries to backup into the Web 01 path: :: - borg init ssh://backup@backup01.srv.local/home/backup/repos/web01.srv.local/pictures + borg init backup@backup01.srv.local:/home/backup/repos/web01.srv.local/pictures :: diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 33eeda41..dd9d45ea 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -405,7 +405,7 @@ Borg can initialize and access repositories on remote hosts if the host is accessible using SSH. This is fastest and easiest when Borg is installed on the remote host, in which case the following syntax is used:: - $ borg init ssh://user@hostname/path/to/repo + $ borg init user@hostname:/path/to/repo Note: please see the usage chapter for a full documentation of repo URLs. diff --git a/docs/usage/general/repository-urls.rst.inc b/docs/usage/general/repository-urls.rst.inc index 153a0397..2855c19b 100644 --- a/docs/usage/general/repository-urls.rst.inc +++ b/docs/usage/general/repository-urls.rst.inc @@ -14,16 +14,32 @@ Note: you may also prepend a ``file://`` to a filesystem path to get URL style. **Remote repositories** accessed via ssh user@host: -``ssh://user@host:port/path/to/repo`` - remote repo, absolute path +``user@host:/path/to/repo`` - remote repo, absolute path + +``ssh://user@host:port/path/to/repo`` - same, alternative syntax, port can be given + **Remote repositories with relative paths** can be given using this syntax: +``user@host:path/to/repo`` - path relative to current directory + +``user@host:~/path/to/repo`` - path relative to user's home directory + +``user@host:~other/path/to/repo`` - path relative to other's home directory + +Note: giving ``user@host:/./path/to/repo`` or ``user@host:/~/path/to/repo`` or +``user@host:/~other/path/to/repo`` is also supported, but not required here. + + +**Remote repositories with relative paths, alternative syntax with port**: + ``ssh://user@host:port/./path/to/repo`` - path relative to current directory ``ssh://user@host:port/~/path/to/repo`` - path relative to user's home directory ``ssh://user@host:port/~other/path/to/repo`` - path relative to other's home directory + If you frequently need the same repo URL, it is a good idea to set the ``BORG_REPO`` environment variable to set a default for the repo URL: @@ -36,31 +52,3 @@ to use the default - it will be read from BORG_REPO then. Use ``::`` syntax to give the repo URL when syntax requires giving a positional argument for the repo (e.g. ``borg mount :: /mnt``). - -Converting from scp-style repo URLs -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Borg does not support scp-style repo URLs any more. - -Here is how you can convert to URL style: - -:: - - user@host:path/to/repo # relative to cwd - --> - ssh://user@host:22/./path/to/repo # relative to cwd - or (usually the cwd is the home dir after ssh login) - ssh://user@host:22/~/path/to/repo # relative to home dir - - user@host:/path/to/repo # absolute repo path - --> - ssh://user@host:22/path/to/repo # absolute repo path - -Notes: - -Port 22 is the default, so you can omit the ``:22`` if you like. - -If you used some hack to use a non-standard port (which was not directly -supported by the scp-style repo specification), you can now do it that way -with the ``ssh:`` URL and remove the hack (usually in ssh configuration or -given via ``--rsh`` borg option). diff --git a/docs/usage/init.rst b/docs/usage/init.rst index 844bba4c..73e62bb7 100644 --- a/docs/usage/init.rst +++ b/docs/usage/init.rst @@ -15,8 +15,8 @@ Examples # Remote repository (accesses a remote borg via ssh) # repokey: stores the (encrypted) key into /config - $ borg init --encryption=repokey-aes-ocb ssh://user@hostname/./backup + $ borg init --encryption=repokey-aes-ocb user@hostname:backup # Remote repository (accesses a remote borg via ssh) # keyfile: stores the (encrypted) key into ~/.config/borg/keys/ - $ borg init --encryption=keyfile-aes-ocb ssh://user@hostname/./backup + $ borg init --encryption=keyfile-aes-ocb user@hostname:backup diff --git a/src/borg/helpers/parseformat.py b/src/borg/helpers/parseformat.py index 3e145ada..b6815712 100644 --- a/src/borg/helpers/parseformat.py +++ b/src/borg/helpers/parseformat.py @@ -315,7 +315,7 @@ class Location: # path must not contain :: (it ends at :: or string end), but may contain single colons. # to avoid ambiguities with other regexes, it must also not start with ":" nor with "//" nor with "ssh://". - local_path_re = r""" + scp_path_re = r""" (?!(:|//|ssh://)) # not starting with ":" or // or ssh:// (?P([^:]|(:(?!:)))+) # any chars, but no "::" """ @@ -361,8 +361,13 @@ class Location: (?Pfile):// # file:// """ + file_path_re + optional_archive_re, re.VERBOSE) # servername/path, path or path::archive - local_re = re.compile( - local_path_re + optional_archive_re, re.VERBOSE) # local path with optional archive + # note: scp_re is also used for local paths + scp_re = re.compile(r""" + ( + """ + optional_user_re + host_re + r""" # user@ (optional), host name or address + : # : (required!) + )? # user@host: part is optional + """ + scp_path_re + optional_archive_re, re.VERBOSE) # path with optional archive # get the repo from BORG_REPO env and the optional archive from param. # if the syntax requires giving REPOSITORY (see "borg mount"), @@ -438,11 +443,13 @@ class Location: self.path = normpath_special(m.group('path')) self.archive = m.group('archive') return True - m = self.local_re.match(text) + m = self.scp_re.match(text) if m: + self.user = m.group('user') + self._host = m.group('host') self.path = normpath_special(m.group('path')) self.archive = m.group('archive') - self.proto = 'file' + self.proto = self._host and 'ssh' or 'file' return True return False diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py index 450ba175..0983a271 100644 --- a/src/borg/testsuite/archiver.py +++ b/src/borg/testsuite/archiver.py @@ -4014,7 +4014,7 @@ class ManifestAuthenticationTest(ArchiverTestCaseBase): class RemoteArchiverTestCase(ArchiverTestCase): - prefix = 'ssh://__testsuite__' + prefix = '__testsuite__:' def open_repository(self): return RemoteRepository(Location(self.repository_location)) diff --git a/src/borg/testsuite/helpers.py b/src/borg/testsuite/helpers.py index b181da56..8014101a 100644 --- a/src/borg/testsuite/helpers.py +++ b/src/borg/testsuite/helpers.py @@ -113,6 +113,32 @@ class TestLocationWithoutEnv: "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)" assert Location('file:///some/path').to_key_filename() == keys_dir + 'some_path' + def test_scp(self, monkeypatch, keys_dir): + monkeypatch.delenv('BORG_REPO', raising=False) + assert repr(Location('user@host:/some/path::archive')) == \ + "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')" + assert repr(Location('user@host:/some/path')) == \ + "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)" + assert repr(Location('user@[::]:/some/path::archive')) == \ + "Location(proto='ssh', user='user', host='::', port=None, path='/some/path', archive='archive')" + assert repr(Location('user@[::]:/some/path')) == \ + "Location(proto='ssh', user='user', host='::', port=None, path='/some/path', archive=None)" + assert repr(Location('user@[2001:db8::]:/some/path::archive')) == \ + "Location(proto='ssh', user='user', host='2001:db8::', port=None, path='/some/path', archive='archive')" + assert repr(Location('user@[2001:db8::]:/some/path')) == \ + "Location(proto='ssh', user='user', host='2001:db8::', port=None, path='/some/path', archive=None)" + assert repr(Location('user@[2001:db8::c0:ffee]:/some/path::archive')) == \ + "Location(proto='ssh', user='user', host='2001:db8::c0:ffee', port=None, path='/some/path', archive='archive')" + assert repr(Location('user@[2001:db8::c0:ffee]:/some/path')) == \ + "Location(proto='ssh', user='user', host='2001:db8::c0:ffee', port=None, path='/some/path', archive=None)" + assert repr(Location('user@[2001:db8::192.0.2.1]:/some/path::archive')) == \ + "Location(proto='ssh', user='user', host='2001:db8::192.0.2.1', port=None, path='/some/path', archive='archive')" + assert repr(Location('user@[2001:db8::192.0.2.1]:/some/path')) == \ + "Location(proto='ssh', user='user', host='2001:db8::192.0.2.1', port=None, path='/some/path', archive=None)" + assert Location('user@[2001:db8::192.0.2.1]:/some/path').to_key_filename() == keys_dir + '2001_db8__192_0_2_1__some_path' + assert repr(Location('user@[2a02:0001:0002:0003:0004:0005:0006:0007]:/some/path')) == \ + "Location(proto='ssh', user='user', host='2a02:0001:0002:0003:0004:0005:0006:0007', port=None, path='/some/path', archive=None)" + def test_smb(self, monkeypatch, keys_dir): monkeypatch.delenv('BORG_REPO', raising=False) assert repr(Location('file:////server/share/path::archive')) == \ @@ -171,6 +197,8 @@ class TestLocationWithoutEnv: def test_user_parsing(self): # see issue #1930 + assert repr(Location('host:path::2016-12-31@23:59:59')) == \ + "Location(proto='ssh', user=None, host='host', port=None, path='path', archive='2016-12-31@23:59:59')" assert repr(Location('ssh://host/path::2016-12-31@23:59:59')) == \ "Location(proto='ssh', user=None, host='host', port=None, path='/path', archive='2016-12-31@23:59:59')" @@ -254,6 +282,15 @@ class TestLocationWithEnv: assert repr(Location()) == \ "Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)" + def test_scp(self, monkeypatch): + monkeypatch.setenv('BORG_REPO', 'user@host:/some/path') + assert repr(Location('::archive')) == \ + "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')" + assert repr(Location('::')) == \ + "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)" + assert repr(Location()) == \ + "Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)" + def test_folder(self, monkeypatch): monkeypatch.setenv('BORG_REPO', 'path') assert repr(Location('::archive')) == \ diff --git a/src/borg/testsuite/repository.py b/src/borg/testsuite/repository.py index b4944e58..83cf728e 100644 --- a/src/borg/testsuite/repository.py +++ b/src/borg/testsuite/repository.py @@ -861,7 +861,7 @@ class RemoteRepositoryTestCase(RepositoryTestCase): repository = None # type: RemoteRepository def open(self, create=False): - return RemoteRepository(Location('ssh://__testsuite__' + os.path.join(self.tmppath, 'repository')), + return RemoteRepository(Location('__testsuite__:' + os.path.join(self.tmppath, 'repository')), exclusive=True, create=create) def _get_mock_args(self): @@ -937,11 +937,12 @@ class RemoteRepositoryTestCase(RepositoryTestCase): def test_ssh_cmd(self): args = self._get_mock_args() self.repository._args = args + assert self.repository.ssh_cmd(Location('example.com:foo')) == ['ssh', 'example.com'] assert self.repository.ssh_cmd(Location('ssh://example.com/foo')) == ['ssh', 'example.com'] assert self.repository.ssh_cmd(Location('ssh://user@example.com/foo')) == ['ssh', 'user@example.com'] assert self.repository.ssh_cmd(Location('ssh://user@example.com:1234/foo')) == ['ssh', '-p', '1234', 'user@example.com'] os.environ['BORG_RSH'] = 'ssh --foo' - assert self.repository.ssh_cmd(Location('ssh://example.com/foo')) == ['ssh', '--foo', 'example.com'] + assert self.repository.ssh_cmd(Location('example.com:foo')) == ['ssh', '--foo', 'example.com'] def test_borg_cmd(self): assert self.repository.borg_cmd(None, testing=True) == [sys.executable, '-m', 'borg.archiver', 'serve'] @@ -963,7 +964,7 @@ class RemoteRepositoryTestCase(RepositoryTestCase): '--storage-quota=314159265'] args.rsh = 'ssh -i foo' self.repository._args = args - assert self.repository.ssh_cmd(Location('ssh://example.com/foo')) == ['ssh', '-i', 'foo', 'example.com'] + assert self.repository.ssh_cmd(Location('example.com:foo')) == ['ssh', '-i', 'foo', 'example.com'] class RemoteLegacyFree(RepositoryTestCaseBase): @@ -971,7 +972,7 @@ class RemoteLegacyFree(RepositoryTestCaseBase): def open(self, create=False): with patch.object(RemoteRepository, 'dictFormat', True): - return RemoteRepository(Location('ssh://__testsuite__' + os.path.join(self.tmppath, 'repository')), + return RemoteRepository(Location('__testsuite__:' + os.path.join(self.tmppath, 'repository')), exclusive=True, create=create) def test_legacy_free(self): @@ -994,7 +995,7 @@ class RemoteLegacyFree(RepositoryTestCaseBase): class RemoteRepositoryCheckTestCase(RepositoryCheckTestCase): def open(self, create=False): - return RemoteRepository(Location('ssh://__testsuite__' + os.path.join(self.tmppath, 'repository')), + return RemoteRepository(Location('__testsuite__:' + os.path.join(self.tmppath, 'repository')), exclusive=True, create=create) def test_crash_before_compact(self): From b4d3859b9ef7fd5cffa414933ede5312fe531b00 Mon Sep 17 00:00:00 2001 From: Thalian Date: Fri, 27 May 2022 08:24:27 +0200 Subject: [PATCH 02/22] #6407 - Document Borg 1.2 pattern behavior change Make clear that absolute paths always go into the matcher as if they are relative (without leading slash). Adapt all examples accordingly. fixes #6407 --- docs/changes.rst | 4 +++ docs/deployment/automated-local.rst | 6 ++-- docs/deployment/pull-backup.rst | 4 +-- docs/faq.rst | 2 +- docs/quickstart.rst | 4 +-- docs/usage/create.rst | 2 +- src/borg/archiver.py | 56 ++++++++++++++--------------- 7 files changed, 40 insertions(+), 38 deletions(-) diff --git a/docs/changes.rst b/docs/changes.rst index 3253f9d5..757d453f 100644 --- a/docs/changes.rst +++ b/docs/changes.rst @@ -305,6 +305,10 @@ and maybe just were not noticed. Compatibility notes: +- matching of path patterns has been aligned with borg storing relative paths. + Borg archives file paths without leading slashes. Previously, include/exclude + patterns could contain leading slashes. You should check your patterns and + remove leading slashes. - dropped support / testing for older Pythons, minimum requirement is 3.8. In case your OS does not provide Python >= 3.8, consider using our binary, which does not need an external Python interpreter. Or continue using diff --git a/docs/deployment/automated-local.rst b/docs/deployment/automated-local.rst index b22ce60a..3ed3cd46 100644 --- a/docs/deployment/automated-local.rst +++ b/docs/deployment/automated-local.rst @@ -136,8 +136,8 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.). # This is just an example, change it however you see fit borg create $BORG_OPTS \ - --exclude /root/.cache \ - --exclude /var/lib/docker/devicemapper \ + --exclude root/.cache \ + --exclude var/lib/docker/devicemapper \ $TARGET::$DATE-$$-system \ / /boot @@ -145,7 +145,7 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.). # Even if it isn't (add --exclude /home above), it probably makes sense # to have /home in a separate archive. borg create $BORG_OPTS \ - --exclude 'sh:/home/*/.cache' \ + --exclude 'sh:home/*/.cache' \ $TARGET::$DATE-$$-home \ /home/ diff --git a/docs/deployment/pull-backup.rst b/docs/deployment/pull-backup.rst index 97accf8d..258a7fb7 100644 --- a/docs/deployment/pull-backup.rst +++ b/docs/deployment/pull-backup.rst @@ -98,9 +98,9 @@ create the backup, retaining the original paths, excluding the repository: :: - borg create --exclude /borgrepo --files-cache ctime,size /borgrepo::archive / + borg create --exclude borgrepo --files-cache ctime,size /borgrepo::archive / -For the sake of simplicity only ``/borgrepo`` is excluded here. You may want to +For the sake of simplicity only ``borgrepo`` is excluded here. You may want to set up an exclude file with additional files and folders to be excluded. Also note that we have to modify Borg's file change detection behaviour – SSHFS cannot guarantee stable inode numbers, so we have to supply the diff --git a/docs/faq.rst b/docs/faq.rst index 05f5176b..f77aa495 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -433,7 +433,7 @@ Say you want to prune ``/var/log`` faster than the rest of archive *names* and then implement different prune policies for different prefixes. For example, you could have a script that does:: - borg create --exclude /var/log $REPOSITORY:main-$(date +%Y-%m-%d) / + borg create --exclude var/log $REPOSITORY:main-$(date +%Y-%m-%d) / borg create $REPOSITORY:logs-$(date +%Y-%m-%d) /var/log Then you would have two different prune calls with different policies:: diff --git a/docs/quickstart.rst b/docs/quickstart.rst index dd9d45ea..a306783a 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -182,8 +182,8 @@ backed up and that the ``prune`` command is keeping and deleting the correct bac --show-rc \ --compression lz4 \ --exclude-caches \ - --exclude '/home/*/.cache/*' \ - --exclude '/var/tmp/*' \ + --exclude 'home/*/.cache/*' \ + --exclude 'var/tmp/*' \ \ ::'{hostname}-{now}' \ /etc \ diff --git a/docs/usage/create.rst b/docs/usage/create.rst index 06d2748e..16906d7e 100644 --- a/docs/usage/create.rst +++ b/docs/usage/create.rst @@ -19,7 +19,7 @@ Examples # Backup home directories excluding image thumbnails (i.e. only # /home//.thumbnails is excluded, not /home/*/*/.thumbnails etc.) $ borg create /path/to/repo::my-files /home \ - --exclude 'sh:/home/*/.thumbnails' + --exclude 'sh:home/*/.thumbnails' # Backup the root filesystem into an archive named "root-YYYY-MM-DD" # use zlib compression (good, but slow) - default is lz4 (fast, low compression ratio) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index de9489ff..dced9fcc 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -2489,15 +2489,12 @@ class Archiver: currently active recursion root. You usually give the recursion root(s) when invoking borg and these can be either relative or absolute paths. - So, when you give `relative/` as root, the paths going into the matcher - will look like `relative/.../file.ext`. When you give `/absolute/` as - root, they will look like `/absolute/.../file.ext`. - - File paths in Borg archives are always stored normalized and relative. - This means that e.g. ``borg create /path/to/repo ../some/path`` will - store all files as `some/path/.../file.ext` and ``borg create - /path/to/repo /home/user`` will store all files as - `home/user/.../file.ext`. + If you give `/absolute/` as root, the paths going into the matcher will + look relative like `absolute/.../file.ext`, because file paths in Borg + archives are always stored normalized and relative. This means that e.g. + ``borg create /path/to/repo ../some/path`` will store all files as + `some/path/.../file.ext` and ``borg create /path/to/repo /home/user`` + will store all files as `home/user/.../file.ext`. A directory exclusion pattern can end either with or without a slash ('/'). If it ends with a slash, such as `some/path/`, the directory will be @@ -2510,10 +2507,11 @@ class Archiver: option. For commands that support patterns in their ``PATH`` argument like (``borg list``), the default pattern is path prefix. - Starting with Borg 1.2, for all but regular expression pattern matching - styles, all paths are treated as relative, meaning that a leading path - separator is removed after normalizing and before matching. This allows - you to use absolute or relative patterns arbitrarily. + Starting with Borg 1.2, discovered fs paths are normalised, have leading + slashes removed and then are matched against your patterns. + Note: You need to review your include / exclude patterns and make + sure they do not expect leading slashes. Borg can only deal with this + for some very simple patterns by removing leading slashes there also. If followed by a colon (':') the first two characters of a pattern are used as a style selector. Explicit style selection is necessary when a @@ -2603,26 +2601,26 @@ class Archiver: # Exclude '/home/user/junk' and '/home/user/subdir/junk' but # not '/home/user/importantjunk' or '/etc/junk': - $ borg create -e '/home/*/junk' backup / + $ borg create -e 'home/*/junk' backup / # Exclude the contents of '/home/user/cache' but not the directory itself: $ borg create -e home/user/cache/ backup / # The file '/home/user/cache/important' is *not* backed up: - $ borg create -e /home/user/cache/ backup / /home/user/cache/important + $ borg create -e home/user/cache/ backup / /home/user/cache/important # The contents of directories in '/home' are not backed up when their name # ends in '.tmp' - $ borg create --exclude 're:^/home/[^/]+\\.tmp/' backup / + $ borg create --exclude 're:^home/[^/]+\\.tmp/' backup / # Load exclusions from file $ cat >exclude.txt < Date: Fri, 27 May 2022 15:17:43 +0200 Subject: [PATCH 03/22] #6407 - Document Borg 1.2 pattern behavior change Fix wrong root path that was accidently changed in last commit. --- src/borg/archiver.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index dced9fcc..0dd91bce 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -2698,8 +2698,8 @@ class Archiver: # note that excludes use fm: by default and patternfiles use sh: by default. # therefore, we need to specify fm: to have the same exact behavior. P fm - R home/bobby - R home/susan + R /home/bobby + R /home/susan - home/bobby/junk From 7b082222560a11c965109db616dfd27da29d2c29 Mon Sep 17 00:00:00 2001 From: TW Date: Sat, 28 May 2022 00:33:49 +0200 Subject: [PATCH 04/22] Merge pull request #6722 from ThomasWaldmann/debug-get-chunk-1.2 borg debug dump-repo-objs --ghost: new --segment=S --offset=O options --- src/borg/archiver.py | 6 +++++- src/borg/repository.py | 18 +++++++++++++----- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index 0dd91bce..356ae242 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -2290,7 +2290,7 @@ class Archiver: key = key_factory(repository, cdata) break i = 0 - for id, cdata, tag, segment, offset in repository.scan_low_level(): + for id, cdata, tag, segment, offset in repository.scan_low_level(segment=args.segment, offset=args.offset): if tag == TAG_PUT: decrypt_dump(i, id, cdata, tag='put', segment=segment, offset=offset) elif tag == TAG_DELETE: @@ -3917,6 +3917,10 @@ class Archiver: help='repository to dump') subparser.add_argument('--ghost', dest='ghost', action='store_true', help='dump all segment file contents, including deleted/uncommitted objects and commits.') + subparser.add_argument('--segment', metavar='SEG', dest='segment', default=None, type=positive_int_validator, + help='used together with --ghost: limit processing to given segment.') + subparser.add_argument('--offset', metavar='OFFS', dest='offset', default=None, type=positive_int_validator, + help='used together with --ghost: limit processing to given offset.') debug_search_repo_objs_epilog = process_epilog(""" This command searches raw (but decrypted and decompressed) repo objects for a specific bytes sequence. diff --git a/src/borg/repository.py b/src/borg/repository.py index 9267fe0e..c6565cda 100644 --- a/src/borg/repository.py +++ b/src/borg/repository.py @@ -1102,7 +1102,7 @@ class Repository: logger.info('Finished %s repository check, no problems found.', mode) return not error_found or repair - def scan_low_level(self): + def scan_low_level(self, segment=None, offset=None): """Very low level scan over all segment file entries. It does NOT care about what's committed and what not. @@ -1111,13 +1111,21 @@ class Repository: This is intended as a last-resort way to get access to all repo contents of damaged repos, when there is uncommitted, but valuable data in there... + + When segment or segment+offset is given, limit processing to this location only. """ - for segment, filename in self.io.segment_iterator(): + for current_segment, filename in self.io.segment_iterator(segment=segment): + if segment is not None and current_segment > segment: + break try: - for tag, key, offset, data in self.io.iter_objects(segment, include_data=True): - yield key, data, tag, segment, offset + for tag, key, current_offset, data in self.io.iter_objects(segment=current_segment, + offset=offset or 0, include_data=True): + if offset is not None and current_offset > offset: + break + yield key, data, tag, current_segment, current_offset except IntegrityError as err: - logger.error('Segment %d (%s) has IntegrityError(s) [%s] - skipping.' % (segment, filename, str(err))) + logger.error('Segment %d (%s) has IntegrityError(s) [%s] - skipping.' % ( + current_segment, filename, str(err))) def _rollback(self, *, cleanup): """ From c34df51e3e163fa4e809dfd10901cdf10babc708 Mon Sep 17 00:00:00 2001 From: Elmar Hoffmann Date: Sat, 28 May 2022 16:07:20 +0200 Subject: [PATCH 05/22] import IntegrityError used as base class with according name This not only brings code style in line with the other helpers that do the same thing this way, but also does away with an unnecessary absolute import using the borg module name explicitly. --- src/borg/helpers/errors.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/borg/helpers/errors.py b/src/borg/helpers/errors.py index 699f6f0a..f07f7bb9 100644 --- a/src/borg/helpers/errors.py +++ b/src/borg/helpers/errors.py @@ -1,6 +1,6 @@ from ..constants import * # NOQA -import borg.crypto.low_level +from ..crypto.low_level import IntegrityError as IntegrityErrorBase class Error(Exception): @@ -30,7 +30,7 @@ class ErrorWithTraceback(Error): traceback = True -class IntegrityError(ErrorWithTraceback, borg.crypto.low_level.IntegrityError): +class IntegrityError(ErrorWithTraceback, IntegrityErrorBase): """Data integrity error: {}""" From fd34fa2d029aec978ae433862d035c9f3af699c4 Mon Sep 17 00:00:00 2001 From: Elmar Hoffmann Date: Sat, 28 May 2022 16:22:58 +0200 Subject: [PATCH 06/22] use relative imports Use relative imports where trivially possible for more consistency and to avoid using the borg module name explicitly. --- src/borg/archiver.py | 12 ++++++------ src/borg/helpers/checks.py | 5 ++--- src/borg/platform/base.py | 6 +++--- src/borg/testsuite/helpers.py | 4 ++-- 4 files changed, 13 insertions(+), 14 deletions(-) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index 0dd91bce..995a87bf 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -622,7 +622,7 @@ class Archiver: key_96 = os.urandom(12) import io - from borg.chunker import get_chunker + from .chunker import get_chunker print("Chunkers =======================================================") size = "1GB" @@ -639,7 +639,7 @@ class Archiver: print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s") import zlib - from borg.checksums import crc32, deflate_crc32, xxh64 + from .checksums import crc32, deflate_crc32, xxh64 print("Non-cryptographic checksums / hashes ===========================") size = "1GB" tests = [ @@ -656,7 +656,7 @@ class Archiver: for spec, func in tests: print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s") - from borg.crypto.low_level import hmac_sha256, blake2b_256 + from .crypto.low_level import hmac_sha256, blake2b_256 print("Cryptographic hashes / MACs ====================================") size = "1GB" for spec, func in [ @@ -665,8 +665,8 @@ class Archiver: ]: print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s") - from borg.crypto.low_level import AES256_CTR_BLAKE2b, AES256_CTR_HMAC_SHA256 - from borg.crypto.low_level import AES256_OCB, CHACHA20_POLY1305 + from .crypto.low_level import AES256_CTR_BLAKE2b, AES256_CTR_HMAC_SHA256 + from .crypto.low_level import AES256_OCB, CHACHA20_POLY1305 print("Encryption =====================================================") size = "1GB" @@ -691,7 +691,7 @@ class Archiver: ]: print(f"{spec:<24} {count:<10} {timeit(func, number=count):.3f}s") - from borg.compress import CompressionSpec + from .compress import CompressionSpec print("Compression ====================================================") for spec in [ 'lz4', diff --git a/src/borg/helpers/checks.py b/src/borg/helpers/checks.py index 1d14788b..316368b0 100644 --- a/src/borg/helpers/checks.py +++ b/src/borg/helpers/checks.py @@ -23,15 +23,14 @@ class ExtensionModuleError(Error): def check_extension_modules(): - import borg.crypto.low_level - from .. import platform, compress, item, chunker, hashindex + from .. import platform, compress, crypto, item, chunker, hashindex if hashindex.API_VERSION != '1.2_01': raise ExtensionModuleError if chunker.API_VERSION != '1.2_01': raise ExtensionModuleError if compress.API_VERSION != '1.2_02': raise ExtensionModuleError - if borg.crypto.low_level.API_VERSION != '1.3_01': + if crypto.low_level.API_VERSION != '1.3_01': raise ExtensionModuleError if item.API_VERSION != '1.2_01': raise ExtensionModuleError diff --git a/src/borg/platform/base.py b/src/borg/platform/base.py index 2bd712cf..3cfd8297 100644 --- a/src/borg/platform/base.py +++ b/src/borg/platform/base.py @@ -4,9 +4,9 @@ import socket import tempfile import uuid -from borg.constants import UMASK_DEFAULT -from borg.helpers import safe_unlink -from borg.platformflags import is_win32 +from ..constants import UMASK_DEFAULT +from ..helpers import safe_unlink +from ..platformflags import is_win32 """ platform base module diff --git a/src/borg/testsuite/helpers.py b/src/borg/testsuite/helpers.py index 8014101a..54343eda 100644 --- a/src/borg/testsuite/helpers.py +++ b/src/borg/testsuite/helpers.py @@ -245,7 +245,7 @@ class TestLocationWithoutEnv: Location('ssh://user@host:/path') def test_omit_archive(self): - from borg.platform import hostname + from ..platform import hostname loc = Location('ssh://user@host:1234/repos/{hostname}::archive') loc_without_archive = loc.omit_archive() assert loc_without_archive.archive is None @@ -264,7 +264,7 @@ class TestLocationWithEnv: "Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)" def test_ssh_placeholder(self, monkeypatch): - from borg.platform import hostname + from ..platform import hostname monkeypatch.setenv('BORG_REPO', 'ssh://user@host:1234/{hostname}') assert repr(Location('::archive')) == \ f"Location(proto='ssh', user='user', host='host', port=1234, path='/{hostname}', archive='archive')" From c2317c4cce6c3a724c86caf3f1ebbbc0e288f6b4 Mon Sep 17 00:00:00 2001 From: Elmar Hoffmann Date: Mon, 30 May 2022 14:01:19 +0200 Subject: [PATCH 07/22] make constants for files cache mode more clear (#6724) * make constants for files cache mode more clear Traditionally, DEFAULT_FILES_CACHE_MODE_UI and DEFAULT_FILES_CACHE_MODE were - as the naming scheme implies - the same setting, one being the UI representation as given to the --files-cache command line option and the other being the same default value in the internal representation. It happended that the actual value used in borg create always comes from DEFAULT_FILES_CACHE_MODE_UI (because that does have the --files-cache option) whereas for all other commands (that do not use the files cache) it comes from DEFAULT_FILES_CACHE_MODE. PR #5777 then abused this fact to implement the optimisation to skip loading of the files cache in those other commands by changing the value of DEFAULT_FILES_CACHE_MODE to disabled. This however also changes the meaning of that variable and thus redesignates it to something not matching the original naming anymore. Anyone not aware of this change and the intention behind it looking at the code would have a hard time figuring this out and be easily mislead. This does away with the confusion making the code more maintainable by renaming DEFAULT_FILES_CACHE_MODE to FILES_CACHE_MODE_DISABLED, making the new intention of that internal default clear. * make constant for files cache mode UI default match naming scheme --- src/borg/archiver.py | 8 ++++---- src/borg/cache.py | 6 +++--- src/borg/constants.py | 4 ++-- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index 4817c01a..7e3f1090 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -196,7 +196,7 @@ def with_repository(fake=False, invert_fake=False, create=False, lock=True, if cache: with Cache(repository, kwargs['key'], kwargs['manifest'], progress=getattr(args, 'progress', False), lock_wait=self.lock_wait, - cache_mode=getattr(args, 'files_cache_mode', DEFAULT_FILES_CACHE_MODE), + cache_mode=getattr(args, 'files_cache_mode', FILES_CACHE_MODE_DISABLED), consider_part_files=getattr(args, 'consider_part_files', False), iec=getattr(args, 'iec', False)) as cache_: return method(self, args, repository=repository, cache=cache_, **kwargs) @@ -240,7 +240,7 @@ def with_other_repository(manifest=False, key=False, cache=False, compatibility= if cache: with Cache(repository, key_, manifest_, progress=False, lock_wait=self.lock_wait, - cache_mode=getattr(args, 'files_cache_mode', DEFAULT_FILES_CACHE_MODE), + cache_mode=getattr(args, 'files_cache_mode', FILES_CACHE_MODE_DISABLED), consider_part_files=getattr(args, 'consider_part_files', False), iec=getattr(args, 'iec', False)) as cache_: kwargs['other_cache'] = cache_ @@ -3799,8 +3799,8 @@ class Archiver: fs_group.add_argument('--sparse', dest='sparse', action='store_true', help='detect sparse holes in input (supported only by fixed chunker)') fs_group.add_argument('--files-cache', metavar='MODE', dest='files_cache_mode', action=Highlander, - type=FilesCacheMode, default=DEFAULT_FILES_CACHE_MODE_UI, - help='operate files cache in MODE. default: %s' % DEFAULT_FILES_CACHE_MODE_UI) + type=FilesCacheMode, default=FILES_CACHE_MODE_UI_DEFAULT, + help='operate files cache in MODE. default: %s' % FILES_CACHE_MODE_UI_DEFAULT) fs_group.add_argument('--read-special', dest='read_special', action='store_true', help='open and read block and char device files as well as FIFOs as if they were ' 'regular files. Also follows symlinks pointing to these kinds of files.') diff --git a/src/borg/cache.py b/src/borg/cache.py index 6fa74e69..9a835459 100644 --- a/src/borg/cache.py +++ b/src/borg/cache.py @@ -13,7 +13,7 @@ logger = create_logger() files_cache_logger = create_logger('borg.debug.files_cache') -from .constants import CACHE_README, DEFAULT_FILES_CACHE_MODE +from .constants import CACHE_README, FILES_CACHE_MODE_DISABLED from .hashindex import ChunkIndex, ChunkIndexEntry, CacheSynchronizer from .helpers import Location from .helpers import Error @@ -371,7 +371,7 @@ class Cache: shutil.rmtree(path) def __new__(cls, repository, key, manifest, path=None, sync=True, warn_if_unencrypted=True, - progress=False, lock_wait=None, permit_adhoc_cache=False, cache_mode=DEFAULT_FILES_CACHE_MODE, + progress=False, lock_wait=None, permit_adhoc_cache=False, cache_mode=FILES_CACHE_MODE_DISABLED, consider_part_files=False, iec=False): def local(): @@ -457,7 +457,7 @@ class LocalCache(CacheStatsMixin): """ def __init__(self, repository, key, manifest, path=None, sync=True, warn_if_unencrypted=True, - progress=False, lock_wait=None, cache_mode=DEFAULT_FILES_CACHE_MODE, consider_part_files=False, + progress=False, lock_wait=None, cache_mode=FILES_CACHE_MODE_DISABLED, consider_part_files=False, iec=False): """ :param warn_if_unencrypted: print warning if accessing unknown unencrypted repository diff --git a/src/borg/constants.py b/src/borg/constants.py index 0b2ef16a..f0a03a35 100644 --- a/src/borg/constants.py +++ b/src/borg/constants.py @@ -84,8 +84,8 @@ ITEMS_CHUNKER_PARAMS = (CH_BUZHASH, 15, 19, 17, HASH_WINDOW_SIZE) CH_DATA, CH_ALLOC, CH_HOLE = 0, 1, 2 # operating mode of the files cache (for fast skipping of unchanged files) -DEFAULT_FILES_CACHE_MODE_UI = 'ctime,size,inode' # default for "borg create" command (CLI UI) -DEFAULT_FILES_CACHE_MODE = 'd' # most borg commands do not use the files cache at all (disable) +FILES_CACHE_MODE_UI_DEFAULT = 'ctime,size,inode' # default for "borg create" command (CLI UI) +FILES_CACHE_MODE_DISABLED = 'd' # most borg commands do not use the files cache at all (disable) # return codes returned by borg command # when borg is killed by signal N, rc = 128 + N From bd005c11a226f7e3b36c9f046cc572ed751bd267 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 1 Jun 2022 19:51:55 +0200 Subject: [PATCH 08/22] vagrant/testing: upgrade development.lock.txt esp. the Cython version upgrade to 0.29.30 is important for python 3.11. --- requirements.d/development.lock.txt | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/requirements.d/development.lock.txt b/requirements.d/development.lock.txt index ef1c219a..131097e7 100644 --- a/requirements.d/development.lock.txt +++ b/requirements.d/development.lock.txt @@ -1,13 +1,13 @@ -setuptools==60.7.1 +setuptools==62.1.0 setuptools-scm==6.4.2 -pip==22.0.3 -virtualenv==20.13.0 +pip==22.1.2 +virtualenv==20.14.1 pkgconfig==1.5.5 -tox==3.24.5 -pytest==7.0.0 +tox==3.25.0 +pytest==7.0.1 pytest-xdist==2.5.0 pytest-cov==3.0.0 pytest-benchmark==3.4.1 -Cython==0.29.27 +Cython==0.29.30 twine==3.8.0 python-dateutil==2.8.2 From c00c48a27b69f82293382431a4de42727534a610 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 1 Jun 2022 19:49:37 +0200 Subject: [PATCH 09/22] vagrant: use pyinstaller 4.10 when installed via pip, this automatically build the bootloader now. --- Vagrantfile | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 359ae6d6..4b49f502 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -206,10 +206,7 @@ def install_pyinstaller() . ~/.bash_profile cd /vagrant/borg . borg-env/bin/activate - git clone https://github.com/thomaswaldmann/pyinstaller.git - cd pyinstaller - git checkout v4.7-maint - python setup.py install + pip install 'pyinstaller==4.10' EOF end From de4b9198c3d242107f456b77d6e176038d7fa3ff Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 1 Jun 2022 19:48:10 +0200 Subject: [PATCH 10/22] vagrant: use python 3.9.13 for binary build --- Vagrantfile | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 4b49f502..10d3a4cd 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -163,7 +163,7 @@ def install_pythons(boxname) return <<-EOF . ~/.bash_profile pyenv install 3.10.0 # tests, version supporting openssl 1.1 - pyenv install 3.9.12 # tests, version supporting openssl 1.1, binary build + pyenv install 3.9.13 # tests, version supporting openssl 1.1, binary build pyenv rehash EOF end @@ -181,8 +181,8 @@ def build_pyenv_venv(boxname) . ~/.bash_profile cd /vagrant/borg # use the latest 3.9 release - pyenv global 3.9.12 - pyenv virtualenv 3.9.12 borg-env + pyenv global 3.9.13 + pyenv virtualenv 3.9.13 borg-env ln -s ~/.pyenv/versions/borg-env . EOF end @@ -229,8 +229,8 @@ def run_tests(boxname, skip_env) . ../borg-env/bin/activate if which pyenv 2> /dev/null; then # for testing, use the earliest point releases of the supported python versions: - pyenv global 3.9.12 3.10.0 - pyenv local 3.9.12 3.10.0 + pyenv global 3.9.13 3.10.0 + pyenv local 3.9.13 3.10.0 fi # otherwise: just use the system python # some OSes can only run specific test envs, e.g. because they miss FUSE support: From a970f000b04f808c712eee5e53a32dec66efe803 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Wed, 25 May 2022 12:49:09 +0200 Subject: [PATCH 11/22] allow msgpack 1.0.4, fixes #6716 --- setup.cfg | 2 +- src/borg/helpers/msgpack.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.cfg b/setup.cfg index 9b3d7b19..c3738628 100644 --- a/setup.cfg +++ b/setup.cfg @@ -38,7 +38,7 @@ python_requires = >=3.9 setup_requires = setuptools_scm[toml] >= 6.2 install_requires = - msgpack >=1.0.3, <=1.0.3 + msgpack >=1.0.3, <=1.0.4 packaging argon2-cffi tests_require = diff --git a/src/borg/helpers/msgpack.py b/src/borg/helpers/msgpack.py index 2ace88fe..5815dc94 100644 --- a/src/borg/helpers/msgpack.py +++ b/src/borg/helpers/msgpack.py @@ -142,7 +142,7 @@ def is_slow_msgpack(): def is_supported_msgpack(): # DO NOT CHANGE OR REMOVE! See also requirements and comments in setup.py. import msgpack - return (1, 0, 3) <= msgpack.version <= (1, 0, 3) and \ + return (1, 0, 3) <= msgpack.version <= (1, 0, 4) and \ msgpack.version not in [] # < add bad releases here to deny list From 18a7debf75192de87a27f1635bcb6a6e07314f24 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sun, 10 Apr 2022 01:18:13 +0200 Subject: [PATCH 12/22] CI: also test on python 3.11-dev --- .github/workflows/ci.yml | 3 +++ tox.ini | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cf14f239..9a331e78 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -57,6 +57,9 @@ jobs: - os: ubuntu-20.04 python-version: '3.10' toxenv: py310-fuse3 + - os: ubuntu-20.04 + python-version: '3.11-dev' + toxenv: py311-fuse2 - os: macos-10.15 # macos-latest is macos 11.6.2 and hanging at test_fuse, #6099 python-version: '3.9' toxenv: py39-fuse2 diff --git a/tox.ini b/tox.ini index 4e2ae623..37ed3b14 100644 --- a/tox.ini +++ b/tox.ini @@ -2,7 +2,7 @@ # fakeroot -u tox --recreate [tox] -envlist = py{39,310}-{none,fuse2,fuse3} +envlist = py{39,310,311}-{none,fuse2,fuse3} minversion = 3.2 requires = pkgconfig From c07afb26e4f5cb1fb68d5e831730a22000eddce1 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Fri, 3 Jun 2022 09:59:40 +0200 Subject: [PATCH 13/22] add python 3.11 to pypi metadata --- setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.cfg b/setup.cfg index c3738628..074cf961 100644 --- a/setup.cfg +++ b/setup.cfg @@ -20,6 +20,7 @@ classifiers = Programming Language :: Python :: 3 Programming Language :: Python :: 3.9 Programming Language :: Python :: 3.10 + Programming Language :: Python :: 3.11 Topic :: Security :: Cryptography Topic :: System :: Archiving :: Backup platforms = Linux, MacOS X, FreeBSD, OpenBSD, NetBSD From 421a7ef52fa52253daf630e90c45d4e978a42390 Mon Sep 17 00:00:00 2001 From: Thalian Date: Wed, 1 Jun 2022 23:18:56 +0200 Subject: [PATCH 14/22] [DOCS] #5310 - Overhaul borg help patterns fixes #5310 --- src/borg/archiver.py | 213 +++++++++++++++++++++++-------------------- 1 file changed, 113 insertions(+), 100 deletions(-) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index 7e3f1090..4209b43e 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -2485,40 +2485,35 @@ class Archiver: helptext = collections.OrderedDict() helptext['patterns'] = textwrap.dedent(''' - The path/filenames used as input for the pattern matching start from the - currently active recursion root. You usually give the recursion root(s) - when invoking borg and these can be either relative or absolute paths. + When specifying one or more file paths in a Borg command that supports + patterns for the respective option or argument, you can apply the + patterns described here to include only desired files and/or exclude + unwanted ones. Patterns can be used - If you give `/absolute/` as root, the paths going into the matcher will - look relative like `absolute/.../file.ext`, because file paths in Borg - archives are always stored normalized and relative. This means that e.g. - ``borg create /path/to/repo ../some/path`` will store all files as - `some/path/.../file.ext` and ``borg create /path/to/repo /home/user`` - will store all files as `home/user/.../file.ext`. + - for ``--exclude`` option, + - in the file given with ``--exclude-from`` option, + - for ``--pattern`` option, + - in the file given with ``--patterns-from`` option and + - for ``PATH`` arguments that explicitly support them. - A directory exclusion pattern can end either with or without a slash ('/'). - If it ends with a slash, such as `some/path/`, the directory will be - included but not its content. If it does not end with a slash, such as - `some/path`, both the directory and content will be excluded. + Borg always stores all file paths normalized and relative to the + current recursion root. The recursion root is also named ``PATH`` in + Borg commands like `borg create` that do a file discovery, so do not + confuse the root with the ``PATH`` argument of e.g. `borg extract`. - File patterns support these styles: fnmatch, shell, regular expressions, - path prefixes and path full-matches. By default, fnmatch is used for - ``--exclude`` patterns and shell-style is used for the ``--pattern`` - option. For commands that support patterns in their ``PATH`` argument - like (``borg list``), the default pattern is path prefix. + Starting with Borg 1.2, paths that are matched against patterns always + appear relative. If you give ``/absolute/`` as root, the paths going + into the matcher will look relative like ``absolute/.../file.ext``. + If you give ``../some/path`` as root, the paths will look like + ``some/path/.../file.ext``. - Starting with Borg 1.2, discovered fs paths are normalised, have leading - slashes removed and then are matched against your patterns. - Note: You need to review your include / exclude patterns and make - sure they do not expect leading slashes. Borg can only deal with this - for some very simple patterns by removing leading slashes there also. + File patterns support five different styles. If followed by a colon ':', + the first two characters of a pattern are used as a style selector. + Explicit style selection is necessary if a non-default style is desired + or when the desired pattern starts with two alphanumeric characters + followed by a colon (i.e. ``aa:something/*``). - If followed by a colon (':') the first two characters of a pattern are - used as a style selector. Explicit style selection is necessary when a - non-default style is desired or when the desired pattern starts with - two alphanumeric characters followed by a colon (i.e. `aa:something/*`). - - `Fnmatch `_, selector `fm:` + `Fnmatch `_, selector ``fm:`` This is the default style for ``--exclude`` and ``--exclude-from``. These patterns use a variant of shell pattern syntax, with '\\*' matching any number of characters, '?' matching any single character, '[...]' @@ -2526,7 +2521,7 @@ class Archiver: matching any character not specified. For the purpose of these patterns, the path separator (backslash for Windows and '/' on other systems) is not treated specially. Wrap meta-characters in brackets for a literal - match (i.e. `[?]` to match the literal character `?`). For a path + match (i.e. ``[?]`` to match the literal character '?'). For a path to match a pattern, the full path must match, or it must match from the start of the full path to just before a path separator. Except for the root path, paths will never end in the path separator when @@ -2534,33 +2529,31 @@ class Archiver: separator, a '\\*' is appended before matching is attempted. A leading path separator is always removed. - Shell-style patterns, selector `sh:` + Shell-style patterns, selector ``sh:`` This is the default style for ``--pattern`` and ``--patterns-from``. Like fnmatch patterns these are similar to shell patterns. The difference - is that the pattern may include `**/` for matching zero or more directory - levels, `*` for matching zero or more arbitrary characters with the + is that the pattern may include ``**/`` for matching zero or more directory + levels, ``*`` for matching zero or more arbitrary characters with the exception of any path separator. A leading path separator is always removed. - Regular expressions, selector `re:` - Regular expressions similar to those found in Perl are supported. Unlike - shell patterns regular expressions are not required to match the full + `Regular expressions `_, selector ``re:`` + Unlike shell patterns, regular expressions are not required to match the full path and any substring match is sufficient. It is strongly recommended to anchor patterns to the start ('^'), to the end ('$') or both. Path separators (backslash for Windows and '/' on other systems) in paths are - always normalized to a forward slash ('/') before applying a pattern. The - regular expression syntax is described in the `Python documentation for - the re module `_. + always normalized to a forward slash '/' before applying a pattern. - Path prefix, selector `pp:` + Path prefix, selector ``pp:`` This pattern style is useful to match whole sub-directories. The pattern - `pp:root/somedir` matches `root/somedir` and everything therein. A leading - path separator is always removed. + ``pp:root/somedir`` matches ``root/somedir`` and everything therein. + A leading path separator is always removed. - Path full-match, selector `pf:` + Path full-match, selector ``pf:`` This pattern style is (only) useful to match full paths. This is kind of a pseudo pattern as it can not have any variable or - unspecified parts - the full path must be given. `pf:root/file.ext` matches - `root/file.ext` only. A leading path separator is always removed. + unspecified parts - the full path must be given. ``pf:root/file.ext`` + matches ``root/file.ext`` only. A leading path separator is always + removed. Implementation note: this is implemented via very time-efficient O(1) hashtable lookups (this means you can have huge amounts of such patterns @@ -2573,20 +2566,20 @@ class Archiver: .. note:: - `re:`, `sh:` and `fm:` patterns are all implemented on top of the Python SRE - engine. It is very easy to formulate patterns for each of these types which - requires an inordinate amount of time to match paths. If untrusted users - are able to supply patterns, ensure they cannot supply `re:` patterns. - Further, ensure that `sh:` and `fm:` patterns only contain a handful of - wildcards at most. + ``re:``, ``sh:`` and ``fm:`` patterns are all implemented on top of + the Python SRE engine. It is very easy to formulate patterns for each + of these types which requires an inordinate amount of time to match + paths. If untrusted users are able to supply patterns, ensure they + cannot supply ``re:`` patterns. Further, ensure that ``sh:`` and + ``fm:`` patterns only contain a handful of wildcards at most. Exclusions can be passed via the command line option ``--exclude``. When used from within a shell, the patterns should be quoted to protect them from expansion. The ``--exclude-from`` option permits loading exclusion patterns from a text - file with one pattern per line. Lines empty or starting with the number sign - ('#') after removing whitespace on both ends are ignored. The optional style + file with one pattern per line. Lines empty or starting with the hash sign + '#' after removing whitespace on both ends are ignored. The optional style selector prefix is also supported for patterns loaded from a file. Due to whitespace removal, paths with whitespace at the beginning or end can only be excluded using regular expressions. @@ -2597,21 +2590,21 @@ class Archiver: Examples:: # Exclude '/home/user/file.o' but not '/home/user/file.odt': - $ borg create -e '*.o' backup / + $ borg create -e '*.o' /path/to/repo::archive / # Exclude '/home/user/junk' and '/home/user/subdir/junk' but # not '/home/user/importantjunk' or '/etc/junk': - $ borg create -e 'home/*/junk' backup / + $ borg create -e 'home/*/junk' /path/to/repo::archive / # Exclude the contents of '/home/user/cache' but not the directory itself: - $ borg create -e home/user/cache/ backup / + $ borg create -e home/user/cache/ /path/to/repo::archive / # The file '/home/user/cache/important' is *not* backed up: - $ borg create -e home/user/cache/ backup / /home/user/cache/important + $ borg create -e home/user/cache/ /path/to/repo::archive / /home/user/cache/important # The contents of directories in '/home' are not backed up when their name # ends in '.tmp' - $ borg create --exclude 're:^home/[^/]+\\.tmp/' backup / + $ borg create --exclude 're:^home/[^/]+\\.tmp/' /path/to/repo::archive / # Load exclusions from file $ cat >exclude.txt <