15d7313387
local: Fix fd leak when encountering files directly inside data/
2021-11-06 19:44:57 +01:00
78c7dd53ef
Merge pull request #3526 from greatroar/dump-refactor
...
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
c71729dfc4
Refactor internal/dump + concurrent load/write
...
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
fb5d9345a7
Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
...
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
f2a3b3b4a1
Merge pull request #3537 from greatroar/dump-lru
...
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
708d7a2574
s3: Add warning if key ID or secret is empty
...
Also add debug message if no credential types are available.
Closes #2388
2021-10-06 23:13:40 +01:00
634a9c162d
Check cap instead of len in bloblru
...
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
c892c0bab9
internal/restic: Don't allocate in Tree.Insert
...
name old time/op new time/op delta
BuildTree-8 34.6µs ± 4% 7.0µs ± 3% -79.68% (p=0.000 n=18+19)
name old alloc/op new alloc/op delta
BuildTree-8 34.0kB ± 0% 0.9kB ± 0% -97.37% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
BuildTree-8 108 ± 0% 1 ± 0% -99.07% (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
78dac2fd48
Merge pull request #3523 from greatroar/swift-v2
...
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
5ea8bba1a1
swift: restore context err check for list()
2021-09-24 21:19:46 +02:00
a5e103a212
Merge pull request #3522 from greatroar/dump-lru
...
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
e7ec0453b1
Upgrade ncw/swift to v2
2021-09-24 19:08:37 +02:00
fe04d024c7
Use LRU cache in restic dump
2021-09-24 15:45:08 +02:00
718966a81a
Move Blobcache into dedicated internal package
2021-09-24 15:45:00 +02:00
4f33eca634
Remove unused Writer arg to internal/dump.writeDump
2021-09-24 15:40:42 +02:00
897d8e662c
Add --insecure-tls flag to disable SSL cert verification
...
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
1827b16ade
Merge pull request #3519 from greatroar/maphash
...
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
8d2996eaaa
Replace siphash by hash/maphash
...
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.
Benchmark results on linux/amd64, -benchtime=3s:
name old time/op new time/op delta
IndexHasUnknown-8 50.6ns ±10% 41.0ns ±19% -18.92% (p=0.000 n=9+10)
IndexHasKnown-8 52.6ns ±12% 41.5ns ±12% -21.13% (p=0.000 n=9+10)
IndexMapHash-8 3.64µs ± 1% 2.00µs ± 0% -45.09% (p=0.000 n=10+9)
IndexAlloc-8 700ms ± 1% 601ms ± 6% -14.18% (p=0.000 n=8+10)
IndexAllocParallel-8 205ms ± 5% 192ms ± 8% -6.18% (p=0.043 n=10+10)
MasterIndexAlloc-8 319ms ± 1% 279ms ± 5% -12.58% (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8 156ns ± 8% 147ns ± 6% -5.46% (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8 150ns ± 7% 142ns ± 8% -5.69% (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8 74.4ns ± 6% 72.0ns ± 9% ~ (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8 67.4ns ± 9% 65.5ns ± 7% ~ (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8 461ns ± 2% 445ns ± 2% -3.49% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8 408ns ±11% 378ns ± 5% -7.22% (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8 479ns ± 1% 437ns ± 4% -8.82% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8 406ns ± 8% 343ns ±15% -15.44% (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8 480ns ± 1% 455ns ± 5% -5.15% (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8 391ns ±18% 382ns ± 8% ~ (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8 71.0ns ± 8% 57.2ns ±11% -19.36% (p=0.000 n=9+10)
PackerManager-8 254ms ± 1% 254ms ± 1% ~ (p=0.285 n=15+15)
name old speed new speed delta
IndexMapHash-8 1.12GB/s ± 1% 2.05GB/s ± 0% +82.13% (p=0.000 n=10+9)
PackerManager-8 208MB/s ± 1% 207MB/s ± 1% ~ (p=0.281 n=15+15)
name old alloc/op new alloc/op delta
IndexMapHash-8 0.00B 0.00B ~ (all equal)
IndexAlloc-8 400MB ± 0% 400MB ± 0% ~ (p=1.000 n=9+10)
IndexAllocParallel-8 401MB ± 0% 401MB ± 0% +0.00% (p=0.000 n=10+10)
MasterIndexAlloc-8 258MB ± 0% 262MB ± 0% +1.42% (p=0.000 n=9+10)
PackerManager-8 73.1kB ± 0% 73.1kB ± 0% ~ (p=0.382 n=13+13)
name old allocs/op new allocs/op delta
IndexMapHash-8 0.00 0.00 ~ (all equal)
IndexAlloc-8 907k ± 0% 907k ± 0% -0.00% (p=0.000 n=10+10)
IndexAllocParallel-8 907k ± 0% 907k ± 0% +0.00% (p=0.009 n=10+10)
MasterIndexAlloc-8 327k ± 0% 317k ± 0% -3.06% (p=0.000 n=10+10)
PackerManager-8 744 ± 0% 744 ± 0% ~ (all equal)
2021-09-19 16:05:18 +02:00
58efe21eca
Merge pull request #3264 from amozoss/upstream-master
...
Refactor backup progress
2021-09-19 14:54:42 +02:00
921e328b56
restore: Fix linting error
2021-09-19 14:41:07 +02:00
2cdc0719af
restorer: Sanitize verify errors
2021-09-19 14:01:26 +02:00
bdcdfaf6b4
restore --verify: buffer reuse consistency and comment
2021-09-19 13:11:27 +02:00
2b94742ca5
Replace no-op closures in restorer by nil check
2021-09-19 13:11:16 +02:00
d357744104
Handle canceled context in restore --verify properly
2021-09-19 13:11:05 +02:00
d4225ec803
Simplify buffer growing in Restorer.verifyFile
...
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
de8521ae56
Report number of successes from Restorer.VerifyFiles
...
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
bb066cf7d3
Concurrent Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
556424d61b
Reuse buffer in Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
973fa921cb
Test and document Restorer.VerifyFiles
2021-09-19 12:52:11 +02:00
e0d615c264
archiver: Fix TestArchiverAbortEarlyOnError test
...
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
fbb0e6499a
ui: consolidate backup ui in ui/backup package
2021-09-12 16:20:15 +02:00
cccb0d4064
ui: assert that ProgressPrinter interface is implemented
2021-09-12 15:26:40 +02:00
d62bfed65d
ui: move SetDryRun to ProgressReporter
2021-09-12 15:25:58 +02:00
77b129ec74
ui: remove unused Summary field
2021-09-12 15:25:58 +02:00
3024239e40
ui/json: restore summary output
2021-09-12 15:25:58 +02:00
5ccf583b8a
ui: restore a few comments
2021-09-12 15:25:58 +02:00
80cbaf6d38
ui: Simplify ReportTotal methods
2021-09-12 15:25:58 +02:00
448419990c
Refactor backup progress
...
Move the shared logic into the progress
Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
fa5ca8af81
Merge pull request #2856 from aawsome/remove-readahead
...
Simplify cache logic
2021-09-04 20:25:49 +02:00
b45d88e124
Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
...
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
a29777f467
Merge pull request #3501 from greatroar/printprogress
...
Streamline progress printing in cmd/restic
2021-09-03 23:34:36 +02:00
81876d5c1b
Simplify cache logic
2021-09-03 21:01:00 +02:00
5aaa3e93c1
internal/ui/termstatus: Optimize and publish Truncate
...
name old time/op new time/op delta
TruncateASCII-8 347ns ± 1% 69ns ± 1% -80.02% (p=0.000 n=9+10)
TruncateUnicode-8 447ns ± 3% 348ns ± 1% -22.04% (p=0.000 n=10+10)
2021-09-03 11:48:22 +02:00
ec2e3b260e
Merge pull request #3499 from greatroar/wrappedconn-pointer
...
rclone: Return one fewer value from run
2021-08-31 21:48:36 +02:00
950b818274
rclone: Return one fewer value from run
2021-08-26 18:12:08 +02:00
defe19fdf6
Quote snapshot prefix in error string
2021-08-25 16:11:28 +02:00
68370feeee
backends: Remove TestSaveFilenames test
...
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
2021-08-15 18:24:16 +02:00
574c83e47f
rest: Fix test to use paths which are the sha256 sum of the data
2021-08-15 18:19:43 +02:00
e6a5801155
rest: Fix test backend url
...
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
2021-08-15 18:16:17 +02:00
f4c5dec05d
backend: test that a wrong hash fails an upload
2021-08-04 22:17:46 +02:00
Michael Eischer