Commit Graph

135 Commits

Author SHA1 Message Date
greatroar 0db1d11b2e archiver: Remove cleanup goroutine from BufferPool
This isn't doing anything. Channels should get cleaned up by the GC when
the last reference to them disappears, just like all other data
structures. Also inlined BufferPool.Put in Buffer.Release, its only
caller.
2022-05-29 17:09:16 +02:00
Michael Eischer 9ffb8920f1 repository: run blackbox tests using old and new repo version 2022-04-30 11:34:10 +02:00
Alexander Neumann db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Michael Eischer c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer d6db5a1fc2 archiver: Fix test
The test relied on an undeocumented sideeffect of the LoadBlob implementation
2022-03-28 22:09:49 +02:00
Alexander Neumann fb5d9345a7
Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
greatroar c892c0bab9 internal/restic: Don't allocate in Tree.Insert
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
Michael Eischer e0d615c264 archiver: Fix TestArchiverAbortEarlyOnError test
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
Alexander Neumann 0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
Alexander Neumann 200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann 5c617859ab backup/scanner: Fix total size for overlapping targets
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann 81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
greatroar 6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer debc4a3a99 archiver: fix race condition during worker startup
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
Michael Eischer 08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer fc60b560ba archiver: Let saveTree report a canceled context as an error
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer 736e964317 archiver: Don't loose error if background context is canceled
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
Alexander Weiss aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Quentin Lemaire ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Weiss 826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Neumann 445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann 56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
Michael Eischer 50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
MichaelEischer 862ee4b2c9
Merge pull request #2970 from labkode/fopenskip
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador 958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
Michael Eischer 1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer efbb850d92 Remove a few redundant type specifiers
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
greatroar c4e2203e45 Check error in archiver before calling Select
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar 9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Michael Eischer dc31529fc3 Unindent else block after if block ending with a return statement 2020-09-05 10:07:16 +02:00
Michael Eischer 84ea2389ae archiver_test: Only skip symlinks on windows 2020-09-05 10:07:16 +02:00
Michael Eischer b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Michael Eischer ee05501ce7 archiver: Remove unused chmod function 2020-09-05 10:07:16 +02:00
Michael Eischer 014600bee6 archiver: Remove unused variable from test 2020-09-05 10:07:16 +02:00
Michael Eischer 460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer b25978a53c backup: Fix reporting of directory count in summary
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
Michael Eischer c847aace35 Rename Index interface to MasterIndex
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
Alexander Weiss 9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
Alexander Weiss 9175795fdb Check contents in archiver
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
MichaelEischer 6896c6449b
Merge pull request #2779 from greatroar/archiver-comment
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
greatroar 0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss 91906911b0 Fix non-intuitive repository behavior
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
Martin Michlmayr 5cc1760fdf
Fix typos 2020-05-16 14:05:26 +08:00
greatroar be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
Michael Eischer bdf7ba20cb archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
The Save methods of the BlobSaver, FileSaver and TreeSaver return early
on when the archiver is stopped due to an error. For that they select on
both the tomb.Dying() and context.Done() channels, which can lead to a
race condition when the tomb is killed due to an error: The tomb first
closes its Dying channel before canceling all child contexts.
Archiver.SaveDir only aborts its execution once the context was
canceled. When the tomb killing is paused between closing its Dying
channel and canceling the child contexts, this lets the
FileSaver/TreeSaver.Save methods return immediately, however, ScanDir
still reads further files causing the test case to fail.

As a killed tomb always cancels all child contexts and as the Savers
always use a context bound to the tomb, it is sufficient to just use
context.Done() as escape hatch in the Save functions. This fixes the
mismatch between SaveDir and Save.

Adjust the tests to use contexts bound to the tomb for all interactions
with the Savers.
2020-04-13 18:23:17 +02:00
Alexander Neumann 52abec967f
Merge pull request #2605 from middelink/fix-2604
Fix running tests on a SELinux enabled system
2020-03-01 20:25:28 +01:00
Pauline Middelink 2828a9c2b0 Fix running tests on a SELinux enabled system
Archivers TestMetadataChanged incorrectly clears the Extended Attributes
from the expected metadata of the temporary file. This is incorrect as on
SELinux enabled filesystem, as the kernel will automaticly add a SElinux
label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we
still need to clear them if there are no attributes found.
2020-03-01 20:23:22 +01:00
Alexander Neumann 95da6c1c1d
Merge pull request #2589 from greatroar/no-stable-sort
Replace sort.Stable by sort.Strings
2020-03-01 19:40:28 +01:00
Alexander Neumann b67b7ebfe6
Merge pull request #2583 from greatroar/unused
Remove some unused or duplicated code
2020-03-01 10:46:17 +01:00
greatroar 4f6fd9fb98 Remove remnant of Go 1.9 compatibility code from tests 2020-02-26 22:23:38 +01:00