Commit Graph

5980 Commits

Author SHA1 Message Date
Michael Eischer 53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Michael Eischer cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat 2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer 12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer 5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat 13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Alexander Neumann 13aae82635
Merge pull request #3673 from restic/update-go
Update go
2022-03-20 12:02:32 +01:00
Alexander Neumann b85d035956 Fix calens install step 2022-03-20 11:36:45 +01:00
Alexander Neumann 47aa4613f7 Force Go to use Module Mode 2022-03-20 11:30:01 +01:00
Alexander Neumann a9a5acb8ce Update golangci-lint 2022-03-20 11:26:56 +01:00
Alexander Neumann 6dee59b789 Install gox before checking out code
Otherwise newer Go versions complain that the hash for the installed
version of gox is not in the go.mod, which we don't want anyways because
the tests should use the latest version of gox.
2022-03-20 11:26:56 +01:00
Alexander Neumann 2e19d19216 Use latest Go version for cross-compile and lint 2022-03-20 11:26:56 +01:00
Alexander Neumann 18a1de0de1 Use "go get" or "go install" selectively
Go 1.18 dropped support for installing binaries via "go get", Go <= 1.16
does not support it. So we need to use the right verb depending on the
Go version.
2022-03-20 11:26:56 +01:00
Alexander Neumann 9b57fcc6b0 Fix build.go, minimum Go version is 1.14 2022-03-20 10:54:33 +01:00
Alexander Neumann 17878036d8 Update tests to Go 1.18 2022-03-20 10:54:24 +01:00
Jason Lenz 2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
MichaelEischer daea461f15
Merge pull request #3663 from jimt/find-msgs
Remove period from find messages
2022-03-07 22:23:49 +01:00
Jim Tittsler a3d99217a4 Remove period from find messages
Simplifies cut-and-paste of IDs (and makes stylistically
consistent with other messages) #3659
2022-03-07 11:16:04 +09:00
MichaelEischer e0ab689ccd
Merge pull request #3664 from DanielG/fix-doc-warning
doc: Fix block quote warning
2022-03-06 21:56:26 +01:00
Michael Eischer 7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
Daniel Gröber 49b67c8aaa doc: Fix block quote warning 2022-03-06 18:15:55 +01:00
rawtaz 44d543ede3
Merge pull request #3653 from MichaelEischer/fix-ls-option-help
ls: Fix description for --host, --tag and --path options
2022-02-19 23:06:24 +01:00
Michael Eischer 5ef4ee7760 ls: Fix description for --host, --tag and --path options 2022-02-19 22:36:02 +01:00
Michael Eischer 254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
MichaelEischer ad4f4dbc7a
Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar 63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
MichaelEischer 4a2d5a146d
Merge pull request #3507 from ahmgithubahm/document-AWS_PROFILE-support
Document AWS_PROFILE support
2022-02-18 23:40:25 +01:00
Michael Eischer 1efc26899d Update docs for AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE 2022-02-18 23:31:10 +01:00
MichaelEischer 8df246d0f3
Merge pull request #3628 from gum3ng/issue_3127
[#issue 3127] Add xattr support for Solaris
2022-02-17 22:07:39 +01:00
Michael Eischer 27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
gum3ng dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer fb4c5af5c4
Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer 18ec49ddfa
Merge pull request #3644 from duritong/centos-epel-repo
add a note about installation via epel for RHEL / CentOS Stream 8 & 9
2022-02-12 21:41:46 +01:00
Michael Eischer 5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
Michael Eischer d72181c8c1 Ensure that the lock cleanup handler is run after the global one
cleanup handlers run in the order in which they are added. As Go calls
init() functions in lexical order, the cleanup handler from global.go
was registered before that from lock.go, which is the correct order.

Make this order explicit to ensure that this won't break accidentally.
2022-02-12 20:21:58 +01:00
Michael Eischer c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
Michael Eischer bba8ba7a5b repository: cancel streampack context after error 2022-02-12 20:18:25 +01:00
Michael Eischer 47554a3428 repository: Fix error handling in repack
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer 4b3dc415ef checker: cleanup header extraction 2022-02-12 20:18:25 +01:00
Michael Eischer 930a00ad54 checker: reuse bufio reader 2022-02-12 20:18:25 +01:00
Michael Eischer 34ebafb8b6 repository: don't crash if blob size is too short 2022-02-12 20:18:25 +01:00
Michael Eischer becebf5d88 repository: remove unused DownloadAndHash 2022-02-12 20:18:25 +01:00
Michael Eischer f1e58e7c7f checker: rewrite ReadData to stream packs 2022-02-12 20:18:25 +01:00
Michael Eischer f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer f00f690658 repository: stream packs during repacking 2022-02-12 20:18:25 +01:00
Michael Eischer c4a2bfcb39 repository: Add StreamPacks function
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer 153e2ba859 repository: Implement lisiting blobs per pack file 2022-02-12 20:18:24 +01:00
MichaelEischer cc90f2ba6b
Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00