43732bb885
Add CreateIndexFromPacks()
2020-11-15 07:04:51 +01:00
c45f8ee075
Replace restic.statT interface by concrete types
...
name old time/op new time/op delta
NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20)
NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
...
Fix #2688 : Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
...
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list
2020-11-14 15:48:56 +00:00
17bb77b1f9
check: Also check blob length and offset
2020-11-14 00:42:49 +01:00
80dcfca191
check: Check sizes computed from index and pack header
2020-11-14 00:42:49 +01:00
94a154c7ca
Remove `--drive-use-trash=false` from rclone param
...
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
04d1983800
Merge pull request #3090 from fgma/vss-fix-386
...
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
8ad9f88993
helpers: Improve error message
2020-11-12 20:38:31 +01:00
f9223cd827
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-12 19:31:00 +01:00
dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
...
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
4a0b7328ec
s3: Remove dots for config description
2020-11-11 20:20:35 +01:00
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
829959390a
Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1
2020-11-11 11:54:38 +01:00
ccd55d529d
Add s3.list-objects-v1 extended option and default to false
2020-11-11 11:54:36 +01:00
4ddcc17135
Add support for boolean extended options
2020-11-11 11:54:27 +01:00
407843c5f9
Merge pull request #3034 from hoyho/gh_master
...
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
46d31ab86d
Merge pull request #3058 from greatroar/counter
...
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
c986823d3f
Merge pull request #3048 from aawsome/check-pack-index
...
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
239931578c
check: check index for packs that are read
2020-11-09 17:28:14 +01:00
9df52327cc
bugfix: omit ENOTDATA for extended attributes
...
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
3bf43d7951
Merge pull request #2475 from restic/use-github-actions
...
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
5cf42884c8
Run tests on GitHub Actions
2020-11-08 16:04:57 +01:00
826cfa0533
fix context in archiver tests
2020-11-08 08:24:24 +01:00
fef408a8bd
Return context error in mem backend
2020-11-08 00:05:53 +01:00
a2d4209322
Don't recurse in local backend's List if not required
...
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.
Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9 ).
2020-11-07 08:54:13 +01:00
fd33030556
Use in-memory index to rebuild index in prune
2020-11-06 20:23:30 +01:00
38cc4393f6
Add Masterindex.Save(); Add Index.Packs()
2020-11-06 20:23:30 +01:00
aaf1c44362
Fix #3062
2020-11-05 17:05:42 +01:00
ab2790d9de
Fix http2 stream reset between restic and rest backends #3014
2020-11-05 15:57:40 +03:00
86b5d8ffaa
s3: add bucket-lookup parameter to select path or dns style bucket lookup
...
This is to enable restic working with Alibaba cloud
Fixes #2528
2020-11-05 12:20:10 +01:00
ae5302c7a8
Add comment that keepBlobs is modified
2020-11-05 10:33:38 +01:00
866a52ad4e
Remove unneeded seek
...
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
a4507610a0
Fix typo
2020-11-05 10:31:49 +01:00
7def2d8ea7
Use a non-constant seed
2020-11-05 10:31:49 +01:00
ee0112ab3b
Clarify message about expected error
2020-11-05 10:31:49 +01:00
b373f164fe
prune: Parallelize repack command
2020-11-05 10:31:49 +01:00
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
916b2d303b
vss: fix potential crash (not reachable in restic) ( #3045 )
...
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
a06f5c28c0
Merge pull request #3055 from greatroar/cleanup
...
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
c2f3eee5af
Don't separately allocate sync.{Mutex,Once} if not necessary
...
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00
ce7d613749
Add Blob.Handle()
2020-11-03 16:42:21 +01:00
581d90cf91
Make some pack parameters public
2020-11-03 16:42:21 +01:00
9a88fb253b
Merge pull request #3051 from greatroar/sanitize-env
...
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
f14436953a
Merge pull request #3050 from greatroar/widechars
...
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
11fbaaae9a
Sanitize environment before starting backend processes (rclone, ssh)
...
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.
Fixes #2192 .
2020-11-02 16:41:23 +01:00
3ed84ff0c6
Fix string truncation in ui/termstatus
...
Fixes #3046 .
2020-11-02 12:50:49 +01:00
8e965ed4eb
Merge pull request #3039 from MichaelEischer/tar-proper-dirs
...
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
5f0fa2129e
Improve readability
...
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes
...
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
...
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio
...
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
e708628cfd
Remove unused function
...
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
bb4b3481a6
Merge pull request #3031 from greatroar/debug-no-seek
...
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
ad3a52e6f0
Merge pull request #3026 from greatroar/refactor-ui
...
internal/ui refactoring
2020-10-25 17:51:39 +01:00
8d7d6ad2d5
dump: include username in tar
2020-10-24 22:42:01 +02:00
fe09e6f865
dump: test proper permissions and directory name
2020-10-24 22:42:01 +02:00
1e3c9a2c11
dump: Fix file permission to tar mapping
...
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
e21dcb0eea
dump: Additional ACL tests
2020-10-24 22:41:01 +02:00
31b8d7a639
Merge pull request #2274 from fgma/master
...
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
5695f9ebd2
vss: Implement VSS support for Windows
...
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.
Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
63be3704d9
Allow debug logging to pipes and terminals
2020-10-19 22:30:30 +02:00
35419de232
Simplify ui.StdioWrapper.Write
...
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
863a590a81
Refactor termstatus.Term.{Print,Error} methods
2020-10-17 20:30:46 +02:00
7c0b6a82db
Remove unused public method ui.linesWriter.Flush
2020-10-17 20:30:24 +02:00
96a912b65a
Merge pull request #3025 from aawsome/fix-index-ids
...
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
b44ecde8b0
Fix setting of ID in DecodeIndex
2020-10-17 09:12:58 +02:00
39fe1e96fe
Merge pull request #3018 from greatroar/background-check
...
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
4ba237bb93
Merge pull request #3019 from greatroar/refactor-decodeindex
...
Refactor index decoding
2020-10-15 23:22:33 +02:00
b27375f5ce
defer close(ch) outside repository.RunWorkers
2020-10-14 15:50:16 +02:00
720e0ee0c7
if cond { return true }; return false => return cond
2020-10-13 20:56:43 +02:00
27db3ec262
Refactor index decoding
...
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
f80b07b2c8
Fix IsProcessBackground on Linux with stdin redirection
...
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup
...
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
...
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
f0cd16e5ea
Cache uint32-typed ids in lookup{Username,Group}
...
NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10)
NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10)
name old alloc/op new alloc/op delta
NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
a03fe4562e
Remove unused error return in lookup{Username,Group}
2020-10-12 14:48:23 +02:00
c193cea119
Pass Context to NewRestorer()
2020-10-10 15:24:26 +02:00
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo
...
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
6099f81692
Fix #1212 restore code produces inconsistent timestamps/permissions.
...
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
9d7f616190
Improve restorer debug log information
2020-10-10 13:46:43 +02:00
295ddb9e57
Add test case for inconsistent timestamps and permissions restoration
...
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
e638b46a13
Embed context into ReaderAt
...
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
d6cfe857b7
pass proper context into MasterIndex.RebuildIndex
2020-10-09 22:39:07 +02:00
37a5e2d681
rest: use global context on repository creation
2020-10-09 22:39:06 +02:00
4b0fcaed45
unlock: use proper context for locks cleanup
...
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
645a6efaf2
restorer: remove redundant type specification
2020-10-09 22:37:56 +02:00
a449450021
init: pass proper context to master key generation
...
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
2020-10-09 22:37:56 +02:00
603bb0e309
restore: Use proper context while loading snapshot
2020-10-09 22:37:56 +02:00
c458e114d4
pass context to Find / FindSnapshot
...
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
45e9a55c62
Wire context into backend layout detection
2020-10-09 22:37:24 +02:00
307a6ba3a3
Upgrade minio sdk to v7
...
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
50da20d93d
Warn if backup failed to read tree blob
2020-10-09 22:36:27 +02:00
5fd3dbccb7
Merge pull request #2963 from MichaelEischer/fix-status-deadlock
...
backup: Fix possible deadlock of scanner goroutine
2020-10-09 21:35:50 +02:00
35655a481b
Merge pull request #2931 from MichaelEischer/reduce-debug-overhead
...
Reduce debug log overhead
2020-10-09 21:35:11 +02:00
30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create
...
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
356ac404cd
Merge pull request #2996 from greatroar/setconsolecursorposition
...
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-09 18:19:02 +02:00
88c8e903d2
filter: Fix glob matching on absolute path marker on windows
...
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
2020-10-09 16:11:05 +02:00
740758a5fa
Optimize filter pattern matching
...
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.
name old time/op new time/op delta
FilterLines-8 44.7ms ± 5% 44.9ms ± 5% ~ (p=0.631 n=10+10)
FilterPatterns/Relative-8 13.6ms ± 4% 13.4ms ± 5% ~ (p=0.165 n=10+10)
FilterPatterns/Absolute-8 10.9ms ± 5% 10.7ms ± 4% ~ (p=0.052 n=10+10)
FilterPatterns/Wildcard-8 53.7ms ± 5% 50.4ms ± 5% -6.00% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8 128ms ± 2% 95ms ± 1% -25.54% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=1.000 n=9+8)
FilterPatterns/Absolute-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.903 n=9+8)
FilterPatterns/Wildcard-8 19.7MB ± 0% 19.7MB ± 0% -0.00% (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-8 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-09 15:51:14 +02:00
862ee4b2c9
Merge pull request #2970 from labkode/fopenskip
...
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
958dc6aafc
Skip fopen for file change check to avoid network penalty
2020-10-09 10:53:28 +02:00
88cc444779
Merge pull request #2934 from MichaelEischer/upgrade-backoff
...
Upgrade github.com/cenkalti/backoff module
2020-10-08 20:33:30 +02:00
b79f18209f
Upgrade github.com/cenkalti/backoff module
...
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
2020-10-07 22:04:59 +02:00
8388e67c4b
filter: cleanup path separator conversion
2020-10-07 21:14:07 +02:00
0acc3c5923
filter: special case patterns without globbing characters
...
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.
name old time/op new time/op delta
FilterLines-4 34.8ms ±17% 41.2ms ±23% +18.36% (p=0.000 n=10+10)
FilterPatterns/Relative-4 21.7ms ± 6% 12.1ms ±23% -44.46% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 10.0ms ± 5% 9.1ms ±11% -9.80% (p=0.006 n=10+9)
FilterPatterns/Wildcard-4 47.0ms ± 7% 42.2ms ± 5% -10.19% (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4 190ms ± 1% 131ms ±20% -31.47% (p=0.000 n=8+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.870 n=9+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.145 n=10+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 19.7MB ± 0% +37.91% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.421 n=10+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:55:43 +02:00
54a124de3b
filter: explicitly test separate ListWithChild function
2020-10-07 20:47:52 +02:00
bcc3bddcf4
filter: only check whether a child path could match when necessary
...
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.
name old time/op new time/op delta
FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10)
FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:52 +02:00
17c53efb0d
filter: Optimize double wildcard expansion
...
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.
name old time/op new time/op delta
FilterPatterns/Relative-4 22.7ms ± 5% 23.3ms ± 9% ~ (p=0.353 n=10+10)
FilterPatterns/Absolute-4 14.2ms ±13% 13.9ms ± 7% ~ (p=0.853 n=10+10)
FilterPatterns/Wildcard-4 266ms ±16% 51ms ± 7% -80.67% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 554ms ± 6% 551ms ± 9% ~ (p=0.436 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.349 n=10+10)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.073 n=10+9)
FilterPatterns/Wildcard-4 141MB ± 0% 14MB ± 0% -89.89% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 1.63M ± 0% 0.09M ± 0% -94.56% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:48 +02:00
7959796269
filter: Special case for absolute paths
...
name old time/op new time/op delta
FilterPatterns/Relative-4 23.6ms ±20% 22.7ms ± 5% ~ (p=0.684 n=10+10)
FilterPatterns/Absolute-4 32.3ms ± 8% 14.2ms ±13% -56.01% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 334ms ±17% 266ms ±16% -20.56% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 709ms ± 7% 554ms ± 6% -21.89% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% +0.00% (p=0.046 n=9+10)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.464 n=10+10)
FilterPatterns/Wildcard-4 141MB ± 0% 141MB ± 0% ~ (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 1.63M ± 0% 1.63M ± 0% ~ (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:29 +02:00
375c2a56de
filter: Parse filter patterns only once
...
name old time/op new time/op delta
FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10)
FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9)
FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
b8eacd1364
filter: Reduce redundant path and pattern splitting
...
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.
name old time/op new time/op delta
FilterPatterns/Relative-4 62.1ms ±15% 30.3ms ±10% -51.22% (p=0.000 n=9+10)
FilterPatterns/Absolute-4 111ms ±10% 49ms ± 3% -56.08% (p=0.000 n=10+8)
FilterPatterns/Wildcard-4 393ms ±15% 345ms ± 9% -12.30% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 10.0s ± 3% 3.9s ± 2% -60.53% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 16.4MB ± 0% 4.6MB ± 0% -71.76% (p=0.000 n=10+9)
FilterPatterns/Absolute-4 31.4MB ± 0% 8.5MB ± 0% -72.77% (p=0.000 n=9+10)
FilterPatterns/Wildcard-4 168MB ± 0% 146MB ± 0% -13.19% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 3.23GB ± 0% 0.91GB ± 0% -71.96% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 178k ± 0% 67k ± 0% -62.50% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 266k ± 0% 89k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.87M ± 0% 1.70M ± 0% -9.47% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 17.7M ± 0% 4.5M ± 0% -74.87% (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
e73c281142
filter: Benchmark absolute paths, wildcards and long filter lists
2020-10-07 17:54:36 +02:00
fdd3b14db3
filter: test some corner cases
2020-10-07 17:09:44 +02:00
f4282aa6fd
local: mark repository files as read-only
...
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258 .
2020-10-07 12:29:37 +02:00
40ee17167e
local: Ignore permission errors on chmod call in Save/Remove operation
...
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
2020-10-07 12:29:37 +02:00
f3e933f0c1
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-07 11:18:54 +02:00
09d39e260d
Merge pull request #2980 from greatroar/error-wrapping
...
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
1579d2a8ec
Remove some unused assignments
2020-10-06 14:55:13 +02:00
eba5dd831f
Fix typos reported by misspell
2020-10-06 14:55:13 +02:00
9ffb698c8d
Modernize context import in internal/fuse
...
Migrated using `go fix`.
2020-10-06 14:55:13 +02:00
efbb850d92
Remove a few redundant type specifiers
...
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
673dda77c0
Less repetitive error messages in internal/cache
...
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling
...
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
c84643c6a9
Merge pull request #2981 from greatroar/mkdircachedir
...
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
1e1a1f3078
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:25:39 +02:00
c4e2203e45
Check error in archiver before calling Select
...
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
7d0fa1a686
Simplify cache directory creation
2020-10-05 10:46:45 +02:00
8b8e230771
Swap deprecated GCS lib with replacement
2020-10-03 18:55:56 +02:00
00cedd22aa
Replace deprecated method in gs backend
2020-10-01 10:02:42 +02:00
d44df9d00d
backup: Always remove the status lines once a backup ends
2020-09-30 23:13:10 +02:00
8d0ba55ecd
backup: Fix possible deadlock of scanner goroutine
...
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.
This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.
This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
2020-09-30 23:13:10 +02:00
48d0ab5276
Merge pull request #2959 from restic/keep-text
...
backup: Correct keep policy text
2020-09-30 17:57:00 +02:00
fd02407863
Merge pull request #2849 from classmarkets/gcs-access-token
...
gs: support authentication with access token
2020-09-30 17:42:56 +02:00
028c9a5343
backup: Correct keep policy text
...
Makes the following corrections to the "Applying Policy:" output:
- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241
...
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
206cadfab4
Hide password from repository URLs
2020-09-22 22:00:51 +02:00
4875f7b659
Merge pull request #2614 from greatroar/simplify-fs
...
Simplify internal/fs
2020-09-21 22:05:10 +02:00
0d65b78168
Simplify os.ModeType|os.ModeCharDevice => os.ModeType
...
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
2020-09-21 14:21:32 +02:00
95ebba85ff
Remove stray Printf from internal/fs
2020-09-21 14:21:32 +02:00
59b343a9bf
Remove OS-specific versions of fs.MkdirAll
...
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
2020-09-21 14:21:32 +02:00
1557c58eef
Fix and simplify fs.Reader
...
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
2020-09-21 14:21:32 +02:00
c504aa505c
Remove unused fs.Rename
2020-09-21 14:21:32 +02:00
1b20f6beec
Remove io.Writer from fs.File
...
It was only used in a single test, which now uses plain *os.File instead.
2020-09-21 14:21:32 +02:00
8c36317b71
rclone: use configured number of connections during create
2020-09-19 19:11:43 +02:00
60a5c35de9
rclone: close connection to rclone if open fails
2020-09-19 19:11:43 +02:00
07f4e7d10b
Only log HTTP requests when a debug log is configured
...
The logged HTTP requests are only visible when a debug log is
configured.
2020-09-19 17:07:53 +02:00
f003410402
init: Add `--copy-chunker-params` option
...
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
9ad8250a78
Reduce overhead of debug calls if no log is enabled
...
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
2020-09-19 16:45:01 +02:00
cb3f531050
Make Cache.Path private
...
It's only used inside the package.
2020-09-18 11:17:29 +02:00
23fcbb275a
Remove restic.Cache interface
...
It was used in one code path, which then asserted its concrete type as
*cache.Cache.
Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
6e3215a80d
Remove unused error type from internal/cache
2020-09-18 10:38:14 +02:00
9abef3bf1a
Move internal/fs.TestChdir to internal/test.Chdir
2020-09-17 10:43:33 +02:00
0c48e515f0
Merge pull request #2630 from MichaelEischer/fix-staticcheck
...
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
97950ab81a
options: Fix test for Go >= 1.15
2020-09-12 17:36:44 +02:00
88664ba222
Merge pull request #2893 from MichaelEischer/restore-preallocate
...
restorer: Preallocate files
2020-09-08 22:43:05 +02:00
121233e1b3
Add preallocate tests
2020-09-07 21:41:47 +02:00
8cc9514879
restorer: pre-allocate files before loading chunks
2020-09-07 21:41:47 +02:00
2e7d475029
Process packs in order of first appearance
2020-09-06 21:25:35 +02:00
c46edcd9d6
error strings should not end with punctuation
2020-09-05 10:07:17 +02:00
1ede018ea6
error variable names should start with 'Err'
2020-09-05 10:07:17 +02:00
b77e933d80
Convert underscore variable names to camelCase
2020-09-05 10:07:16 +02:00
d0329cf3eb
Adjust comments to match name of exported methods
2020-09-05 10:07:16 +02:00
dc31529fc3
Unindent else block after if block ending with a return statement
2020-09-05 10:07:16 +02:00
4784540f04
repository: Simplify worker group code
2020-09-05 10:07:16 +02:00
84ea2389ae
archiver_test: Only skip symlinks on windows
2020-09-05 10:07:16 +02:00
b4a7ce86cf
uint cannot be less than zero
2020-09-05 10:07:16 +02:00
7ee0964880
crypto: Remove unused error
2020-09-05 10:07:16 +02:00
f6f11400c2
restic: Remove unused variable value
2020-09-05 10:07:16 +02:00
e2dc5034d3
restic/file: Use FileType for all File constants
...
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
9a1b3cb5d9
restorer: Remove unused treeID struct member from test
2020-09-05 10:07:16 +02:00
ee05501ce7
archiver: Remove unused chmod function
2020-09-05 10:07:16 +02:00
014600bee6
archiver: Remove unused variable from test
2020-09-05 10:07:16 +02:00
d9a80e07b9
repository: Simplify index age calculation
2020-09-05 10:07:16 +02:00
460e2ffbf6
Collapse a few boolean operations
2020-09-05 10:07:14 +02:00
2f8335554c
Remove a few unused variables
2020-09-05 10:06:23 +02:00
37113282ca
Merge a few variable declaration and initializations
2020-09-05 10:05:34 +02:00
337725c354
Use sort.Strings
2020-09-05 10:05:34 +02:00
dcf9ded977
dump: Use slashes as path separators in tar on windows
...
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
2020-08-30 18:26:50 +02:00
fe6445e0f4
dump: Added unit tests for tar output
2020-08-30 16:00:48 +02:00
80a11960dd
dump: Always dump relative paths into tarballs
...
Tarballs should only contain relative paths.
2020-08-29 21:14:34 +02:00
d6f739ec22
dump: Extract tar code to internal/dump
2020-08-29 21:14:34 +02:00
b25978a53c
backup: Fix reporting of directory count in summary
...
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks
...
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
51c22f4223
local backend: ignore not supported error on sync()
...
Closes #2395
2020-08-22 01:27:07 +02:00
f4b9544ab2
prune: Add test that repack aborts on wrong blob
2020-08-16 11:34:01 +02:00
367449dede
prune: Reduce memory allocations while repacking
...
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
2020-08-16 11:34:01 +02:00
7042bafea5
prune: Abort repacking when a pack contains a wrong blob
...
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
2020-08-16 11:34:01 +02:00
0fed6a8dfc
Use "pack file" instead of "data file" ( #2885 )
...
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
eca0f0ad24
Merge pull request #2863 from aawsome/index-no-duplicates
...
Don't save exact duplicates in merged index
2020-08-08 18:24:14 +02:00
b112533812
Don't save exact duplicates when merging indexes
2020-08-05 06:32:02 +02:00
5e63294355
Add benchmark MasterIndexAlloc
2020-08-05 06:32:02 +02:00
05116e4787
prune: Cleanup progress bar handling while repacking
2020-08-03 19:32:46 +02:00
b7b479b668
Merge pull request #2599 from MichaelEischer/tweak-mem-usage
...
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
4cf9656f12
Merge pull request #2861 from MichaelEischer/fix-rclone-crash
...
rclone: Don't panic after unexpected subprocess exit
2020-08-01 13:08:50 +02:00
2580eef2aa
Merge pull request #2318 from classmarkets/2175-named-keys
...
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
a178e5628e
Remove duplicate TreeLoader interface
2020-08-01 12:29:16 +02:00
af66a62c04
FindUsedBlobs: Test that seen blobs are skipped
...
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
9ea1a78bd4
FindUsedBlobs: Check for seen blobs before loading trees
...
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
184103647a
FindUsedBlobs: merge seen into blobs BlobSet
...
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.
Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
c81b122374
rclone: Don't panic after unexpected subprocess exit
...
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
2020-08-01 12:17:40 +02:00
f8316948d1
Optimize FUSE - make command `restic mount` faster and consume less memory
...
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse
=> Thanks at Michael Eischer for suggesting the last two improvements
2020-07-28 23:01:18 +02:00
ea97ff1ba4
rclone: Skip crash test when rclone is not found
2020-07-26 12:06:18 +02:00
01b9581453
rclone: Better field names for stdio conn
2020-07-26 00:29:25 +02:00
3cd927d180
rclone: Give rclone time to finish before closing stdin pipe
...
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.
Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
2020-07-26 00:29:25 +02:00
bf7b1f12ea
rclone: Add test for pipe handling when rclone exits
2020-07-26 00:29:25 +02:00
8554332894
rclone: Close rclone side of stdio_conn pipes
...
restic did not notice when the rclone subprocess exited unexpectedly.
restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.
However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
2020-07-26 00:29:25 +02:00
3e93b36ca4
Make rclone.New private
2020-07-26 00:28:45 +02:00
c847aace35
Rename Index interface to MasterIndex
...
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
9d1fb94c6c
make Lookup() return all blobs
...
+ simplify syntax
2020-07-25 21:18:34 +02:00
020cab8e08
Merge pull request #2787 from greatroar/no-blobsize-cache
...
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
07da61baee
Remove blob size cache from restic mount
2020-07-25 19:39:14 +02:00
3d530dfc91
Merge pull request #2827 from aawsome/archiver-test-contents
...
Make self-healing work when backing up with parent snapshot
2020-07-25 13:13:18 +02:00
bbc960f957
Merge pull request #2635 from greatroar/optimize-sortbycached
...
Optimize sorting blobs by cache status
2020-07-25 12:35:42 +02:00
309598c237
Simplify sortCachedPacksFirst test in internal/repository
...
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
2020-07-25 12:12:59 +02:00
03d23e6faa
Speed up blob sorting in internal/repository
...
name old time/op new time/op delta
SortCachedPacksFirst-8 208µs ± 3% 186µs ± 3% -10.74% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
SortCachedPacksFirst-8 213kB ± 0% 139kB ± 0% -34.62% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
SortCachedPacksFirst-8 1.03k ± 0% 1.03k ± 0% -0.19% (p=0.000 n=10+10)
2020-07-25 12:12:59 +02:00
b10acd2af7
Test and benchmark blob sorting in internal/repository
2020-07-25 12:12:58 +02:00
9175795fdb
Check contents in archiver
...
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11
...
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
7c23381a2b
Drop support for Go version 1.11
2020-07-24 18:52:39 +02:00
34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks
...
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
a666a6d576
Add tests and merge indexes in index benchmarks
2020-07-22 21:54:02 +02:00
e388d962a5
Merge final indexes together for faster index access
2020-07-22 21:54:02 +02:00
758b44b9c0
gs: support authentication with access token
...
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.
The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
2020-07-22 16:23:03 +02:00
3b7a3711e6
Add more realistic index benchmarks
...
+ reduce test size of BenchmarkMasterIndexLookupParallel
2020-07-21 07:18:20 +02:00
9b0e718852
checker: Test that blob types are not confused
2020-07-20 23:43:47 +02:00
82c908871d
Merge pull request #2812 from greatroar/chaining
...
Chaining hash table for repository.Index
2020-07-20 23:29:36 +02:00
ddf0b8cd0b
checker: Properly distinguish between data and tree blobs
...
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
2d0c138c9b
checker: Test that check only decodes trees once
...
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
2020-07-20 22:51:53 +02:00
ef325ffc02
checker: Cleanup error handling code
...
This change only moves code around but does not result in any change in
behavior.
2020-07-20 22:51:53 +02:00
7a165f32a9
checker: Traverse trees in depth-first order
...
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation. In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.
Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
2020-07-20 22:51:53 +02:00
36c69e3ca7
checker: Unify blobs, processed trees and referenced blobs map
...
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
2020-07-20 22:51:47 +02:00
35d8413639
checker: Remove dead index map
2020-07-20 22:37:31 +02:00
c66a0e408c
checker: Reduce cost of debug log
...
Avoid duplicate allocation of the Subtree list.
2020-07-20 22:37:31 +02:00
70f4c014ef
checker: Decode identical tree nodes only once
...
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
2020-07-20 22:37:31 +02:00
f0d8710611
Add benchmark for checker scaling with snapshot count
2020-07-20 22:37:31 +02:00
8074879c5f
Remove 'go generate'
2020-07-19 17:28:42 +02:00
7bda28f31f
Chaining hash table for repository.Index
...
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.
Benchmark results on linux/amd64, -benchtime=3s -count=20:
name old time/op new time/op delta
PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.231 n=20+20)
DecodeIndex-8 4.54s ± 0% 4.30s ± 0% -5.20% (p=0.000 n=18+17)
DecodeIndexParallel-8 4.54s ± 0% 4.30s ± 0% -5.22% (p=0.000 n=19+18)
IndexHasUnknown-8 44.4ns ± 5% 50.5ns ±11% +13.82% (p=0.000 n=19+17)
IndexHasKnown-8 48.3ns ± 0% 51.5ns ±12% +6.68% (p=0.001 n=16+20)
IndexAlloc-8 758ms ± 1% 616ms ± 1% -18.69% (p=0.000 n=19+19)
IndexAllocParallel-8 234ms ± 3% 204ms ± 2% -12.60% (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8 122ns ± 0% 145ns ± 9% +18.44% (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8 369ns ± 2% 429ns ± 8% +16.27% (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8 68.4ns ± 5% 74.9ns ±13% +9.47% (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8 315ns ± 3% 369ns ±11% +17.14% (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8 743ns ± 1% 816ns ± 2% +9.87% (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8 238ns ± 1% 260ns ± 2% +9.14% (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8 1.01µs ± 3% 1.11µs ± 2% +9.79% (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8 222ns ± 0% 269ns ± 2% +20.83% (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8 1.06µs ± 2% 1.19µs ± 2% +12.95% (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8 413ns ± 1% 530ns ± 1% +28.19% (p=0.000 n=18+20)
SaveAndEncrypt-8 30.2ms ± 1% 30.4ms ± 0% +0.71% (p=0.000 n=19+19)
LoadTree-8 540µs ± 1% 576µs ± 1% +6.73% (p=0.000 n=20+20)
LoadBlob-8 5.64ms ± 0% 5.64ms ± 0% ~ (p=0.883 n=18+17)
LoadAndDecrypt-8 5.93ms ± 0% 5.95ms ± 1% ~ (p=0.247 n=20+19)
LoadIndex-8 25.1ms ± 0% 24.5ms ± 1% -2.54% (p=0.000 n=18+17)
name old speed new speed delta
PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.229 n=20+20)
SaveAndEncrypt-8 139MB/s ± 1% 138MB/s ± 0% -0.71% (p=0.000 n=19+19)
LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.890 n=18+17)
LoadAndDecrypt-8 169MB/s ± 0% 168MB/s ± 1% ~ (p=0.227 n=20+19)
name old alloc/op new alloc/op delta
PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.772 n=12+19)
IndexAlloc-8 786MB ± 0% 400MB ± 0% -49.04% (p=0.000 n=20+18)
IndexAllocParallel-8 786MB ± 0% 401MB ± 0% -49.04% (p=0.000 n=19+15)
SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% +0.00% (p=0.000 n=19+19)
name old allocs/op new allocs/op delta
PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal)
IndexAlloc-8 977k ± 0% 907k ± 0% -7.18% (p=0.000 n=20+20)
IndexAllocParallel-8 977k ± 0% 907k ± 0% -7.17% (p=0.000 n=19+15)
SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)
2020-07-19 13:58:22 +02:00
255ba83c4b
Parallel index benchmarks + benchmark optimizations
...
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.
BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)
Results on linux/amd64, -benchtime=3s -count=20:
name old time/op new time/op delta
PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.165 n=20+20)
DecodeIndex-8 13.6µs ± 2% 4539886.8µs ± 0% +33293901.38% (p=0.000 n=20+18)
IndexHasUnknown-8 44.4ns ± 7% 44.4ns ± 5% ~ (p=0.873 n=20+19)
IndexHasKnown-8 49.2ns ± 3% 48.3ns ± 0% -1.86% (p=0.000 n=20+16)
IndexAlloc-8 802ms ± 1% 758ms ± 1% -5.51% (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8 124ns ± 1% 122ns ± 0% -1.41% (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8 373ns ± 2% 369ns ± 2% -1.13% (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8 67.8ns ± 3% 68.4ns ± 5% ~ (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8 316ns ± 3% 315ns ± 3% ~ (p=0.846 n=20+20)
SaveAndEncrypt-8 30.5ms ± 1% 30.2ms ± 1% -1.09% (p=0.000 n=19+19)
LoadTree-8 527µs ± 1% 540µs ± 1% +2.37% (p=0.000 n=19+20)
LoadBlob-8 5.65ms ± 0% 5.64ms ± 0% -0.21% (p=0.000 n=19+18)
LoadAndDecrypt-8 7.07ms ± 2% 5.93ms ± 0% -16.15% (p=0.000 n=19+20)
LoadIndex-8 32.1ms ± 2% 25.1ms ± 0% -21.64% (p=0.000 n=20+18)
name old speed new speed delta
PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.159 n=20+20)
SaveAndEncrypt-8 138MB/s ± 1% 139MB/s ± 1% +1.10% (p=0.000 n=19+19)
LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% +0.21% (p=0.000 n=19+18)
LoadAndDecrypt-8 141MB/s ± 2% 169MB/s ± 0% +19.24% (p=0.000 n=19+20)
name old alloc/op new alloc/op delta
PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.826 n=19+12)
IndexAlloc-8 786MB ± 0% 786MB ± 0% +0.01% (p=0.000 n=20+20)
SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% -0.00% (p=0.012 n=20+19)
name old allocs/op new allocs/op delta
PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal)
IndexAlloc-8 977k ± 0% 977k ± 0% +0.01% (p=0.022 n=20+20)
SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)
2020-07-19 13:58:05 +02:00
9ac90cf5cd
internal/fuse: fix dropped test error
2020-07-12 21:42:31 -07:00
58719e1f47
Replace mount's per-file cache by a global LRU cache
2020-07-12 18:27:16 +02:00
d42c169458
Fix quadratic file reading in restic mount
2020-07-12 18:27:16 +02:00
02bec13ef2
Fix repository_test.BenchmarkSaveAndEncrypt
...
The benchmark was actually testing the speed of index lookups.
name old time/op new time/op delta
SaveAndEncrypt-8 101ns ± 2% 31505824ns ± 1% +31311591.31% (p=0.000 n=10+10)
name old speed new speed delta
SaveAndEncrypt-8 41.7TB/s ± 2% 0.0TB/s ± 1% -100.00% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
SaveAndEncrypt-8 1.00B ± 0% 20989508.40B ± 0% +2098950740.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
SaveAndEncrypt-8 0.00 123.00 ± 0% +Inf% (p=0.000 n=10+9)
(The actual speed is ca. 131MiB/s.)
2020-07-05 17:41:42 +02:00
212607dc8a
Merge pull request #2760 from greatroar/backend-benchmark
...
Fix backend benchmarks + a micro-optimization
2020-06-17 23:17:05 +02:00
190d8e2f51
Flatten backend.LimitedReadCloser structure
...
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:
name old time/op new time/op delta
Backend/BenchmarkLoadPartialFile-8 412µs ± 4% 413µs ± 4% ~ (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8 455µs ±13% 441µs ±10% ~ (p=0.072 n=20+18)
name old speed new speed delta
Backend/BenchmarkLoadPartialFile-8 10.2GB/s ± 3% 10.2GB/s ± 4% ~ (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8 9.25GB/s ±12% 9.54GB/s ± 9% ~ (p=0.072 n=20+18)
name old alloc/op new alloc/op delta
Backend/BenchmarkLoadPartialFile-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15)
name old allocs/op new allocs/op delta
Backend/BenchmarkLoadPartialFile-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)
2020-06-17 13:11:45 +02:00
f4cd2a7120
Make backend benchmarks fairer by removing checks
...
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:
name old time/op new time/op delta
Backend/BenchmarkLoadFile-8 4.89ms ± 0% 2.72ms ± 1% -44.26% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8 936µs ± 6% 439µs ±15% -53.07% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8 940µs ± 1% 456µs ±10% -51.50% (p=0.008 n=5+5)
Backend/BenchmarkSave-8 23.9ms ±14% 24.8ms ±41% ~ (p=0.690 n=5+5)
name old speed new speed delta
Backend/BenchmarkLoadFile-8 3.43GB/s ± 0% 6.16GB/s ± 1% +79.40% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8 4.48GB/s ± 6% 9.63GB/s ±14% +114.78% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8 4.46GB/s ± 1% 9.22GB/s ±10% +106.74% (p=0.008 n=5+5)
Backend/BenchmarkSave-8 706MB/s ±13% 698MB/s ±31% ~ (p=0.690 n=5+5)
2020-06-17 13:11:45 +02:00
1361341c58
don't save duplicate packIDs when using internal/repository/Index.Store
2020-06-14 07:56:24 +02:00
Alexander Weiss