1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2025-02-19 04:41:50 +00:00

Merge pull request #2643 from enkore/f/experimental-patterns

mark --pattern, --patterns-from as experimental
This commit is contained in:
enkore 2017-06-10 17:15:22 +02:00 committed by GitHub
commit 33a7331bda
12 changed files with 171 additions and 121 deletions

View file

@ -24,7 +24,7 @@ This command benchmarks borg CRUD (create, read, update, delete) operations.
It creates input data below the given PATH and backups this data into the given REPO.
The REPO must already exist (it could be a fresh empty repo or an existing repo, the
command will create / read / update / delete some archives named borg-test-data* there.
command will create / read / update / delete some archives named borg-test-data\* there.
Make sure you have free space there, you'll need about 1GB each (+ overhead).

View file

@ -41,9 +41,9 @@ Exclusion options
``--keep-exclude-tags``, ``--keep-tag-files``
| if tag objects are specified with --exclude-if-present, don't omit the tag objects themselves from the backup archive
``--pattern PATTERN``
| include/exclude paths matching PATTERN
| experimental: include/exclude paths matching PATTERN
``--patterns-from PATTERNFILE``
| read include/exclude patterns from PATTERNFILE, one per line
| experimental: read include/exclude patterns from PATTERNFILE, one per line
Filesystem options
``-x``, ``--one-file-system``

View file

@ -39,9 +39,9 @@ Exclusion options
``--keep-exclude-tags``, ``--keep-tag-files``
| if tag objects are specified with --exclude-if-present, don't omit the tag objects themselves from the backup archive
``--pattern PATTERN``
| include/exclude paths matching PATTERN
| experimental: include/exclude paths matching PATTERN
``--patterns-from PATTERNFILE``
| read include/exclude patterns from PATTERNFILE, one per line
| experimental: read include/exclude patterns from PATTERNFILE, one per line
Description
~~~~~~~~~~~

View file

@ -26,9 +26,9 @@ optional arguments
``--exclude-from EXCLUDEFILE``
| read exclude patterns from EXCLUDEFILE, one per line
``--pattern PATTERN``
| include/exclude paths matching PATTERN
| experimental: include/exclude paths matching PATTERN
``--patterns-from PATTERNFILE``
| read include/exclude patterns from PATTERNFILE, one per line
| experimental: read include/exclude patterns from PATTERNFILE, one per line
``--strip-components NUMBER``
| Remove the specified number of leading path elements. Pathnames with fewer elements will be silently skipped.

View file

@ -24,9 +24,9 @@ optional arguments
``--exclude-from EXCLUDEFILE``
| read exclude patterns from EXCLUDEFILE, one per line
``--pattern PATTERN``
| include/exclude paths matching PATTERN
| experimental: include/exclude paths matching PATTERN
``--patterns-from PATTERNFILE``
| read include/exclude patterns from PATTERNFILE, one per line
| experimental: read include/exclude patterns from PATTERNFILE, one per line
``--numeric-owner``
| only obey numeric user and group identifiers
``--strip-components NUMBER``

View file

@ -8,8 +8,10 @@ borg help patterns
File patterns support these styles: fnmatch, shell, regular expressions,
path prefixes and path full-matches. By default, fnmatch is used for
`--exclude` patterns and shell-style is used for `--pattern`. If followed
by a colon (':') the first two characters of a pattern are used as a
`--exclude` patterns and shell-style is used for the experimental `--pattern`
option.
If followed by a colon (':') the first two characters of a pattern are used as a
style selector. Explicit style selection is necessary when a
non-default style is desired or when the desired pattern starts with
two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
@ -17,7 +19,7 @@ two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
`Fnmatch <https://docs.python.org/3/library/fnmatch.html>`_, selector `fm:`
This is the default style for --exclude and --exclude-from.
These patterns use a variant of shell pattern syntax, with '*' matching
These patterns use a variant of shell pattern syntax, with '\*' matching
any number of characters, '?' matching any single character, '[...]'
matching any single character specified, including ranges, and '[!...]'
matching any character not specified. For the purpose of these patterns,
@ -28,7 +30,7 @@ two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
must match from the start to just before a path separator. Except
for the root path, paths will never end in the path separator when
matching is attempted. Thus, if a given pattern ends in a path
separator, a '*' is appended before matching is attempted.
separator, a '\*' is appended before matching is attempted.
Shell-style patterns, selector `sh:`
@ -111,39 +113,40 @@ Examples::
EOF
$ borg create --exclude-from exclude.txt backup /
.. container:: experimental
A more general and easier to use way to define filename matching patterns exists
with the `--pattern` and `--patterns-from` options. Using these, you may specify
the backup roots (starting points) and patterns for inclusion/exclusion. A
root path starts with the prefix `R`, followed by a path (a plain path, not a
file pattern). An include rule starts with the prefix +, an exclude rule starts
with the prefix -, both followed by a pattern.
Inclusion patterns are useful to include paths that are contained in an excluded
path. The first matching pattern is used so if an include pattern matches before
an exclude pattern, the file is backed up.
A more general and easier to use way to define filename matching patterns exists
with the experimental `--pattern` and `--patterns-from` options. Using these, you
may specify the backup roots (starting points) and patterns for inclusion/exclusion.
A root path starts with the prefix `R`, followed by a path (a plain path, not a
file pattern). An include rule starts with the prefix +, an exclude rule starts
with the prefix -, both followed by a pattern.
Inclusion patterns are useful to include paths that are contained in an excluded
path. The first matching pattern is used so if an include pattern matches before
an exclude pattern, the file is backed up.
Note that the default pattern style for `--pattern` and `--patterns-from` is
shell style (`sh:`), so those patterns behave similar to rsync include/exclude
patterns. The pattern style can be set via the `P` prefix.
Note that the default pattern style for `--pattern` and `--patterns-from` is
shell style (`sh:`), so those patterns behave similar to rsync include/exclude
patterns. The pattern style can be set via the `P` prefix.
Patterns (`--pattern`) and excludes (`--exclude`) from the command line are
considered first (in the order of appearance). Then patterns from `--patterns-from`
are added. Exclusion patterns from `--exclude-from` files are appended last.
Patterns (`--pattern`) and excludes (`--exclude`) from the command line are
considered first (in the order of appearance). Then patterns from `--patterns-from`
are added. Exclusion patterns from `--exclude-from` files are appended last.
An example `--patterns-from` file could look like that::
An example `--patterns-from` file could look like that::
# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache
# they're downloads for a reason
- /home/*/Downloads
# susan is a nice person
# include susans home
+ /home/susan
# don't backup the other home directories
- /home/*
# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache
# they're downloads for a reason
- /home/*/Downloads
# susan is a nice person
# include susans home
+ /home/susan
# don't backup the other home directories
- /home/*
.. _borg_placeholders:

View file

@ -50,9 +50,9 @@ Exclusion options
``--keep-exclude-tags``, ``--keep-tag-files``
| if tag objects are specified with --exclude-if-present, don't omit the tag objects themselves from the backup archive
``--pattern PATTERN``
| include/exclude paths matching PATTERN
| experimental: include/exclude paths matching PATTERN
``--patterns-from PATTERNFILE``
| read include/exclude patterns from PATTERNFILE, one per line
| experimental: read include/exclude patterns from PATTERNFILE, one per line
Description
~~~~~~~~~~~

View file

@ -39,9 +39,9 @@ Exclusion options
``--keep-exclude-tags``, ``--keep-tag-files``
| if tag objects are specified with --exclude-if-present, don't omit the tag objects themselves from the backup archive
``--pattern PATTERN``
| include/exclude paths matching PATTERN
| experimental: include/exclude paths matching PATTERN
``--patterns-from PATTERNFILE``
| read include/exclude patterns from PATTERNFILE, one per line
| experimental: read include/exclude patterns from PATTERNFILE, one per line
Archive options
``--target TARGET``
@ -82,7 +82,7 @@ There is no risk of data loss by this.
used to have upgraded Borg 0.xx or Attic archives deduplicate with
Borg 1.x archives.
USE WITH CAUTION.
**USE WITH CAUTION.**
Depending on the PATHs and patterns given, recreate can be used to permanently
delete files from archives.
When in doubt, use "--dry-run --verbose --list" to see how patterns/PATHS are

View file

@ -64,7 +64,7 @@
from .helpers import replace_placeholders
from .helpers import ChunkIteratorFileWrapper
from .helpers import popen_with_error_handling
from .nanorst import RstToTextLazy, ansi_escapes
from .nanorst import rst_to_terminal
from .patterns import ArgparsePatternAction, ArgparseExcludeFileAction, ArgparsePatternFileAction, parse_exclude_pattern
from .patterns import PatternMatcher
from .item import Item
@ -1837,8 +1837,10 @@ def do_break_lock(self, args, repository):
helptext['patterns'] = textwrap.dedent('''
File patterns support these styles: fnmatch, shell, regular expressions,
path prefixes and path full-matches. By default, fnmatch is used for
`--exclude` patterns and shell-style is used for `--pattern`. If followed
by a colon (':') the first two characters of a pattern are used as a
`--exclude` patterns and shell-style is used for the experimental `--pattern`
option.
If followed by a colon (':') the first two characters of a pattern are used as a
style selector. Explicit style selection is necessary when a
non-default style is desired or when the desired pattern starts with
two alphanumeric characters followed by a colon (i.e. `aa:something/*`).
@ -1846,7 +1848,7 @@ def do_break_lock(self, args, repository):
`Fnmatch <https://docs.python.org/3/library/fnmatch.html>`_, selector `fm:`
This is the default style for --exclude and --exclude-from.
These patterns use a variant of shell pattern syntax, with '*' matching
These patterns use a variant of shell pattern syntax, with '\*' matching
any number of characters, '?' matching any single character, '[...]'
matching any single character specified, including ranges, and '[!...]'
matching any character not specified. For the purpose of these patterns,
@ -1857,7 +1859,7 @@ def do_break_lock(self, args, repository):
must match from the start to just before a path separator. Except
for the root path, paths will never end in the path separator when
matching is attempted. Thus, if a given pattern ends in a path
separator, a '*' is appended before matching is attempted.
separator, a '\*' is appended before matching is attempted.
Shell-style patterns, selector `sh:`
@ -1940,39 +1942,40 @@ def do_break_lock(self, args, repository):
EOF
$ borg create --exclude-from exclude.txt backup /
.. container:: experimental
A more general and easier to use way to define filename matching patterns exists
with the `--pattern` and `--patterns-from` options. Using these, you may specify
the backup roots (starting points) and patterns for inclusion/exclusion. A
root path starts with the prefix `R`, followed by a path (a plain path, not a
file pattern). An include rule starts with the prefix +, an exclude rule starts
with the prefix -, both followed by a pattern.
Inclusion patterns are useful to include paths that are contained in an excluded
path. The first matching pattern is used so if an include pattern matches before
an exclude pattern, the file is backed up.
A more general and easier to use way to define filename matching patterns exists
with the experimental `--pattern` and `--patterns-from` options. Using these, you
may specify the backup roots (starting points) and patterns for inclusion/exclusion.
A root path starts with the prefix `R`, followed by a path (a plain path, not a
file pattern). An include rule starts with the prefix +, an exclude rule starts
with the prefix -, both followed by a pattern.
Inclusion patterns are useful to include paths that are contained in an excluded
path. The first matching pattern is used so if an include pattern matches before
an exclude pattern, the file is backed up.
Note that the default pattern style for `--pattern` and `--patterns-from` is
shell style (`sh:`), so those patterns behave similar to rsync include/exclude
patterns. The pattern style can be set via the `P` prefix.
Note that the default pattern style for `--pattern` and `--patterns-from` is
shell style (`sh:`), so those patterns behave similar to rsync include/exclude
patterns. The pattern style can be set via the `P` prefix.
Patterns (`--pattern`) and excludes (`--exclude`) from the command line are
considered first (in the order of appearance). Then patterns from `--patterns-from`
are added. Exclusion patterns from `--exclude-from` files are appended last.
Patterns (`--pattern`) and excludes (`--exclude`) from the command line are
considered first (in the order of appearance). Then patterns from `--patterns-from`
are added. Exclusion patterns from `--exclude-from` files are appended last.
An example `--patterns-from` file could look like that::
An example `--patterns-from` file could look like that::
# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache
# they're downloads for a reason
- /home/*/Downloads
# susan is a nice person
# include susans home
+ /home/susan
# don't backup the other home directories
- /home/*\n\n''')
# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache
# they're downloads for a reason
- /home/*/Downloads
# susan is a nice person
# include susans home
+ /home/susan
# don't backup the other home directories
- /home/*\n\n''')
helptext['placeholders'] = textwrap.dedent('''
Repository (or Archive) URLs, --prefix and --remote-path values support these
placeholders:
@ -2099,7 +2102,7 @@ def do_help(self, parser, commands, args):
if not args.topic:
parser.print_help()
elif args.topic in self.helptext:
print(self.helptext[args.topic])
print(rst_to_terminal(self.helptext[args.topic]))
elif args.topic in commands:
if args.epilog_only:
print(commands[args.topic].epilog)
@ -2257,11 +2260,6 @@ def resolve(self, args: argparse.Namespace): # Namespace has "in" but otherwise
setattr(args, dest, option_value)
def build_parser(self):
if hasattr(sys.stdout, 'isatty') and sys.stdout.isatty() and (sys.platform != 'win32' or 'ANSICON' in os.environ):
rst_state_hook = ansi_escapes
else:
rst_state_hook = None
# You can use :ref:`xyz` in the following usage pages. However, for plain-text view,
# e.g. through "borg ... --help", define a substitution for the reference here.
# It will replace the entire :ref:`foo` verbatim.
@ -2279,7 +2277,7 @@ def process_epilog(epilog):
epilog = [line for line in epilog if not line.startswith('.. man')]
epilog = '\n'.join(epilog)
if mode == 'command-line':
epilog = RstToTextLazy(epilog, rst_state_hook, rst_plain_text_references)
epilog = rst_to_terminal(epilog, rst_plain_text_references)
return epilog
def define_common_options(add_common_option):
@ -2793,9 +2791,9 @@ def define_common_options(add_common_option):
'objects themselves from the backup archive')
exclude_group.add_argument('--pattern',
action=ArgparsePatternAction,
metavar="PATTERN", help='include/exclude paths matching PATTERN')
metavar="PATTERN", help='experimental: include/exclude paths matching PATTERN')
exclude_group.add_argument('--patterns-from', action=ArgparsePatternFileAction,
metavar='PATTERNFILE', help='read include/exclude patterns from PATTERNFILE, one per line')
metavar='PATTERNFILE', help='experimental: read include/exclude patterns from PATTERNFILE, one per line')
fs_group = subparser.add_argument_group('Filesystem options')
fs_group.add_argument('-x', '--one-file-system', dest='one_file_system',
@ -2878,9 +2876,9 @@ def define_common_options(add_common_option):
subparser.add_argument('--exclude-from', action=ArgparseExcludeFileAction,
metavar='EXCLUDEFILE', help='read exclude patterns from EXCLUDEFILE, one per line')
subparser.add_argument('--pattern', action=ArgparsePatternAction,
metavar="PATTERN", help='include/exclude paths matching PATTERN')
metavar="PATTERN", help='experimental: include/exclude paths matching PATTERN')
subparser.add_argument('--patterns-from', action=ArgparsePatternFileAction,
metavar='PATTERNFILE', help='read include/exclude patterns from PATTERNFILE, one per line')
metavar='PATTERNFILE', help='experimental: read include/exclude patterns from PATTERNFILE, one per line')
subparser.add_argument('--numeric-owner', dest='numeric_owner',
action='store_true', default=False,
help='only obey numeric user and group identifiers')
@ -2951,9 +2949,9 @@ def define_common_options(add_common_option):
subparser.add_argument('--exclude-from', action=ArgparseExcludeFileAction,
metavar='EXCLUDEFILE', help='read exclude patterns from EXCLUDEFILE, one per line')
subparser.add_argument('--pattern', action=ArgparsePatternAction,
metavar="PATTERN", help='include/exclude paths matching PATTERN')
metavar="PATTERN", help='experimental: include/exclude paths matching PATTERN')
subparser.add_argument('--patterns-from', action=ArgparsePatternFileAction,
metavar='PATTERNFILE', help='read include/exclude patterns from PATTERNFILE, one per line')
metavar='PATTERNFILE', help='experimental: read include/exclude patterns from PATTERNFILE, one per line')
subparser.add_argument('--strip-components', dest='strip_components',
type=int, default=0, metavar='NUMBER',
help='Remove the specified number of leading path elements. Pathnames with fewer elements will be silently skipped.')
@ -3027,9 +3025,9 @@ def define_common_options(add_common_option):
'objects themselves from the backup archive')
exclude_group.add_argument('--pattern',
action=ArgparsePatternAction,
metavar="PATTERN", help='include/exclude paths matching PATTERN')
metavar="PATTERN", help='experimental: include/exclude paths matching PATTERN')
exclude_group.add_argument('--patterns-from', action=ArgparsePatternFileAction,
metavar='PATTERNFILE', help='read include/exclude patterns from PATTERNFILE, one per line')
metavar='PATTERNFILE', help='experimental: read include/exclude patterns from PATTERNFILE, one per line')
rename_epilog = process_epilog("""
This command renames an archive in the repository.
@ -3148,9 +3146,9 @@ def define_common_options(add_common_option):
'objects themselves from the backup archive')
exclude_group.add_argument('--pattern',
action=ArgparsePatternAction,
metavar="PATTERN", help='include/exclude paths matching PATTERN')
metavar="PATTERN", help='experimental: include/exclude paths matching PATTERN')
exclude_group.add_argument('--patterns-from', action=ArgparsePatternFileAction,
metavar='PATTERNFILE', help='read include/exclude patterns from PATTERNFILE, one per line')
metavar='PATTERNFILE', help='experimental: read include/exclude patterns from PATTERNFILE, one per line')
mount_epilog = process_epilog("""
This command mounts an archive as a FUSE filesystem. This can be useful for
@ -3522,9 +3520,9 @@ def define_common_options(add_common_option):
'objects themselves from the backup archive')
exclude_group.add_argument('--pattern',
action=ArgparsePatternAction,
metavar="PATTERN", help='include/exclude paths matching PATTERN')
metavar="PATTERN", help='experimental: include/exclude paths matching PATTERN')
exclude_group.add_argument('--patterns-from', action=ArgparsePatternFileAction,
metavar='PATTERNFILE', help='read include/exclude patterns from PATTERNFILE, one per line')
metavar='PATTERNFILE', help='experimental: read include/exclude patterns from PATTERNFILE, one per line')
archive_group = subparser.add_argument_group('Archive options')
archive_group.add_argument('--target', dest='target', metavar='TARGET', default=None,

View file

@ -1,5 +1,6 @@
import io
import os
import sys
class TextPecker:
@ -31,6 +32,21 @@ def readline(self):
return out
def process_directive(directive, arguments, out, state_hook):
if directive == 'container' and arguments == 'experimental':
state_hook('text', '**', out)
out.write('++ Experimental ++')
state_hook('**', 'text', out)
else:
state_hook('text', '**', out)
out.write(directive.title())
out.write(':\n')
state_hook('**', 'text', out)
if arguments:
out.write(arguments)
out.write('\n')
def rst_to_text(text, state_hook=None, references=None):
"""
Convert rST to a more human text form.
@ -54,8 +70,10 @@ def rst_to_text(text, state_hook=None, references=None):
next = text.peek(1) # type: str
if state == 'text':
if char == '\\' and text.peek(1) in inline_single:
continue
if text.peek(-1) != '\\':
if char in inline_single and next not in inline_single:
if char in inline_single and next != char:
state_hook(state, char, out)
state = char
continue
@ -88,21 +106,19 @@ def rst_to_text(text, state_hook=None, references=None):
raise ValueError("Undefined reference in Archiver help: %r — please add reference substitution"
"to 'rst_plain_text_references'" % ref)
continue
if char == ':' and text.peek(2) == ':\n': # End of line code block
text.read(2)
state_hook(state, 'code-block', out)
state = 'code-block'
out.write(':\n')
continue
if text.peek(-2) in ('\n\n', '') and char == next == '.':
text.read(2)
try:
directive, arguments = text.peekline().split('::', maxsplit=1)
except ValueError:
directive = None
text.readline()
directive, is_directive, arguments = text.readline().partition('::')
text.read(1)
if not directive:
if not is_directive:
continue
out.write(directive.title())
out.write(':\n')
if arguments:
out.write(arguments)
out.write('\n')
process_directive(directive, arguments.strip(), out, state_hook)
continue
if state in inline_single and char == state:
state_hook(state, 'text', out)
@ -118,21 +134,22 @@ def rst_to_text(text, state_hook=None, references=None):
state = 'text'
text.read(1)
continue
if state == 'code-block' and char == next == '\n' and text.peek(5)[1:] != ' ':
# Foo::
#
# *stuff* *code* *ignore .. all markup*
#
# More arcane stuff
#
# Regular text...
state_hook(state, 'text', out)
state = 'text'
out.write(char)
assert state == 'text', 'Invalid final state %r (This usually indicates unmatched */**)' % state
return out.getvalue()
def ansi_escapes(old_state, new_state, out):
if old_state == 'text' and new_state in ('*', '`', '``'):
out.write('\033[4m')
if old_state == 'text' and new_state == '**':
out.write('\033[1m')
if old_state in ('*', '`', '``', '**') and new_state == 'text':
out.write('\033[0m')
class RstToTextLazy:
def __init__(self, str, state_hook=None, references=None):
self.str = str
@ -160,3 +177,26 @@ def __iter__(self):
def __contains__(self, item):
return item in self.rst
def ansi_escapes(old_state, new_state, out):
if old_state == 'text' and new_state in ('*', '`', '``'):
out.write('\033[4m')
if old_state == 'text' and new_state == '**':
out.write('\033[1m')
if old_state in ('*', '`', '``', '**') and new_state == 'text':
out.write('\033[0m')
def rst_to_terminal(rst, references=None, destination=sys.stdout):
"""
Convert *rst* to a lazy string.
If *destination* is a file-like object connected to a terminal,
enrich text with suitable ANSI escapes. Otherwise return plain text.
"""
if hasattr(destination, 'isatty') and destination.isatty() and (sys.platform != 'win32' or 'ANSICON' in os.environ):
rst_state_hook = ansi_escapes
else:
rst_state_hook = None
return RstToTextLazy(rst, rst_state_hook, references)

View file

@ -45,7 +45,7 @@
from ..helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
from ..helpers import bin_to_hex
from ..helpers import MAX_S
from ..nanorst import RstToTextLazy
from ..nanorst import RstToTextLazy, rst_to_terminal
from ..patterns import IECommand, PatternMatcher, parse_pattern
from ..item import Item
from ..logger import setup_logging
@ -3366,3 +3366,8 @@ def discover_level(prefix, parser, Archiver):
def test_help_formatting(command, parser):
if isinstance(parser.epilog, RstToTextLazy):
assert parser.epilog.rst
@pytest.mark.parametrize('topic, helptext', list(Archiver.helptext.items()))
def test_help_formatting_helptexts(topic, helptext):
assert str(rst_to_terminal(helptext))

View file

@ -16,6 +16,10 @@ def test_comment_inline():
assert rst_to_text('Foo and Bar\n.. foo\nbar') == 'Foo and Bar\n.. foo\nbar'
def test_inline_escape():
assert rst_to_text('Such as "\\*" characters.') == 'Such as "*" characters.'
def test_comment():
assert rst_to_text('Foo and Bar\n\n.. foo\nbar') == 'Foo and Bar\n\nbar'