greatroar
740758a5fa
Optimize filter pattern matching
...
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.
name old time/op new time/op delta
FilterLines-8 44.7ms ± 5% 44.9ms ± 5% ~ (p=0.631 n=10+10)
FilterPatterns/Relative-8 13.6ms ± 4% 13.4ms ± 5% ~ (p=0.165 n=10+10)
FilterPatterns/Absolute-8 10.9ms ± 5% 10.7ms ± 4% ~ (p=0.052 n=10+10)
FilterPatterns/Wildcard-8 53.7ms ± 5% 50.4ms ± 5% -6.00% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8 128ms ± 2% 95ms ± 1% -25.54% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=1.000 n=9+8)
FilterPatterns/Absolute-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.903 n=9+8)
FilterPatterns/Wildcard-8 19.7MB ± 0% 19.7MB ± 0% -0.00% (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-8 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-09 15:51:14 +02:00
Michael Eischer
8388e67c4b
filter: cleanup path separator conversion
2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923
filter: special case patterns without globbing characters
...
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.
name old time/op new time/op delta
FilterLines-4 34.8ms ±17% 41.2ms ±23% +18.36% (p=0.000 n=10+10)
FilterPatterns/Relative-4 21.7ms ± 6% 12.1ms ±23% -44.46% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 10.0ms ± 5% 9.1ms ±11% -9.80% (p=0.006 n=10+9)
FilterPatterns/Wildcard-4 47.0ms ± 7% 42.2ms ± 5% -10.19% (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4 190ms ± 1% 131ms ±20% -31.47% (p=0.000 n=8+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.870 n=9+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.145 n=10+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 19.7MB ± 0% +37.91% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.421 n=10+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b
filter: explicitly test separate ListWithChild function
2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4
filter: only check whether a child path could match when necessary
...
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.
name old time/op new time/op delta
FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10)
FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d
filter: Optimize double wildcard expansion
...
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.
name old time/op new time/op delta
FilterPatterns/Relative-4 22.7ms ± 5% 23.3ms ± 9% ~ (p=0.353 n=10+10)
FilterPatterns/Absolute-4 14.2ms ±13% 13.9ms ± 7% ~ (p=0.853 n=10+10)
FilterPatterns/Wildcard-4 266ms ±16% 51ms ± 7% -80.67% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 554ms ± 6% 551ms ± 9% ~ (p=0.436 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.349 n=10+10)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.073 n=10+9)
FilterPatterns/Wildcard-4 141MB ± 0% 14MB ± 0% -89.89% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 1.63M ± 0% 0.09M ± 0% -94.56% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269
filter: Special case for absolute paths
...
name old time/op new time/op delta
FilterPatterns/Relative-4 23.6ms ±20% 22.7ms ± 5% ~ (p=0.684 n=10+10)
FilterPatterns/Absolute-4 32.3ms ± 8% 14.2ms ±13% -56.01% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 334ms ±17% 266ms ±16% -20.56% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 709ms ± 7% 554ms ± 6% -21.89% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% +0.00% (p=0.046 n=9+10)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.464 n=10+10)
FilterPatterns/Wildcard-4 141MB ± 0% 141MB ± 0% ~ (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 1.63M ± 0% 1.63M ± 0% ~ (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de
filter: Parse filter patterns only once
...
name old time/op new time/op delta
FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10)
FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9)
FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364
filter: Reduce redundant path and pattern splitting
...
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.
name old time/op new time/op delta
FilterPatterns/Relative-4 62.1ms ±15% 30.3ms ±10% -51.22% (p=0.000 n=9+10)
FilterPatterns/Absolute-4 111ms ±10% 49ms ± 3% -56.08% (p=0.000 n=10+8)
FilterPatterns/Wildcard-4 393ms ±15% 345ms ± 9% -12.30% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 10.0s ± 3% 3.9s ± 2% -60.53% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 16.4MB ± 0% 4.6MB ± 0% -71.76% (p=0.000 n=10+9)
FilterPatterns/Absolute-4 31.4MB ± 0% 8.5MB ± 0% -72.77% (p=0.000 n=9+10)
FilterPatterns/Wildcard-4 168MB ± 0% 146MB ± 0% -13.19% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 3.23GB ± 0% 0.91GB ± 0% -71.96% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 178k ± 0% 67k ± 0% -62.50% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 266k ± 0% 89k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.87M ± 0% 1.70M ± 0% -9.47% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 17.7M ± 0% 4.5M ± 0% -74.87% (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142
filter: Benchmark absolute paths, wildcards and long filter lists
2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3
filter: test some corner cases
2020-10-07 17:09:44 +02:00
MichaelEischer
0ae02f3030
Merge pull request #2844 from aawsome/prune-integration-tests
...
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
rawtaz
e401859afb
Merge pull request #2992 from lbausch/repo_wording
...
Replace repo/repos with repository/repositories in `--no-lock` help text
2020-10-06 23:03:55 +02:00
Lorenz Bausch
af4100e07d
Replace repo/repos with repository/repositories in --no-lock help text
2020-10-06 22:49:48 +02:00
Alexander Neumann
ea5bbe0857
Remove dysfunctional link to saythanks
2020-10-06 20:43:44 +02:00
Alexander Weiss
6822a58413
Add prune integration tests for many edge cases
2020-10-06 20:20:05 +02:00
MichaelEischer
09d39e260d
Merge pull request #2980 from greatroar/error-wrapping
...
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
MichaelEischer
cfd57c480a
Merge pull request #2991 from restic/nonsense
...
cache/check: Clarify error message when given arguments
2020-10-06 00:40:12 +02:00
Leo R. Lundgren
e105a3f391
cache/check: Clarify error message when given arguments
2020-10-06 00:08:59 +02:00
rawtaz
5d8cfff3f2
Merge pull request #2988 from greatroar/snapshots-json
...
Regression test for #2979
2020-10-05 21:02:52 +02:00
greatroar
673dda77c0
Less repetitive error messages in internal/cache
...
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
MichaelEischer
1ab4c710e1
Merge pull request #2987 from greatroar/no-password-for-help
...
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
greatroar
feedf0ebce
Don't read password for generate, help or self-update
...
Fixes #2951 .
2020-10-05 19:54:07 +02:00
MichaelEischer
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling
...
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
MichaelEischer
c84643c6a9
Merge pull request #2981 from greatroar/mkdircachedir
...
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
MichaelEischer
8390a8aaf3
Merge pull request #2986 from greatroar/upgrade-xxhash
...
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:49:08 +02:00
greatroar
1e1a1f3078
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:25:39 +02:00
MichaelEischer
abe9fa261f
Merge pull request #2983 from greatroar/upgrade-siphash
...
Upgrade siphash to 1.2.2
2020-10-05 17:18:07 +02:00
MichaelEischer
6001b45bf7
Merge pull request #2984 from restic/snapshots-json
...
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 17:14:23 +02:00
greatroar
6bee0aafc2
Regression test for #2979
2020-10-05 15:08:09 +02:00
greatroar
8252ea8e3d
Upgrade siphash to 1.2.2
...
New version is potentially faster on 32-bit ARM.
2020-10-05 14:56:13 +02:00
Leo R. Lundgren
137d20a06a
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 13:14:10 +02:00
greatroar
c4e2203e45
Check error in archiver before calling Select
...
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686
Simplify cache directory creation
2020-10-05 10:46:45 +02:00
rawtaz
a23d90d270
Merge pull request #2975 from restic/rawtaz-doc-resume
...
Add FAQ on whether restic can resume backups
2020-10-03 23:12:34 +02:00
MichaelEischer
2630411530
Merge pull request #2974 from gottwald/swap-gcs-lib
...
Swap deprecated GCS lib with replacement
2020-10-03 21:58:11 +02:00
rawtaz
b9b82d878d
Add FAQ on whether restic can resume backups
...
This new FAQ entry explains that restic can resume interrupted backups.
2020-10-03 21:39:45 +02:00
Ingo Gottwald
8b8e230771
Swap deprecated GCS lib with replacement
2020-10-03 18:55:56 +02:00
rawtaz
7c3c6fa431
Merge pull request #2977 from MichaelEischer/downgrade-cobra
...
Downgrade cobra to v0.0.5
2020-10-03 18:23:13 +02:00
Michael Eischer
29908906b7
Downgrade cobra to v0.0.5
2020-10-03 18:13:07 +02:00
rawtaz
bbeb439f41
Merge pull request #2937 from andreaso/self-update-output-path-fix
...
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
Andreas Olsson
ce14df303b
Don't require self-update --output
placeholder file
...
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.
We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.
While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.
Resolves #2491
2020-10-03 07:02:23 +02:00
MichaelEischer
3c6671b18b
Merge pull request #2973 from restic/rawtaz-repo2
...
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
MichaelEischer
86ff1f2bf6
Merge pull request #2972 from restic/rawtaz-verbose
...
Clarify max verbose level in option description
2020-10-02 22:21:45 +02:00
rawtaz
0cce6dc31c
Correct data type for --repo2 option
2020-10-02 21:10:52 +02:00
rawtaz
6253ff0187
Clarify max verbose level in option description
...
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
rawtaz
e9943e864f
Merge pull request #2955 from jtagcat/clarify--compact
...
help: --compact: clarify description
2020-10-02 15:59:08 +02:00
jtagcat
a687261804
--help: --compact: clarify description
2020-10-02 16:55:56 +03:00
MichaelEischer
4df8861e09
Merge pull request #2967 from gottwald/fix-deprctd-gs-constructor
...
Replace deprecated method in gs backend
2020-10-01 11:01:49 +02:00
Ingo Gottwald
00cedd22aa
Replace deprecated method in gs backend
2020-10-01 10:02:42 +02:00