mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
6,255 Commits 7 Branches 49 Tags 92 MiB
Commit Graph

92 Commits

Author SHA1 Message Date
Michael Eischer a5ebd5de4b restorer: Fix race condition in partialFile.WriteAt 
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
 2022-09-24 21:39:39 +02:00
Michael Eischer 5b6a77058a Enable sparseness only conditionally 
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
 2022-09-24 21:20:00 +02:00
greatroar 5d4568d393 Write sparse files in restorer 
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.

Compared to the naive loop,

	for _, b := range p {
		if b != 0 {
			return false
		}
	}

the optimized allZero is about 10× faster:

name       old time/op    new time/op     delta
AllZero-8    1.09ms ± 1%     0.09ms ± 1%    -92.10%  (p=0.000 n=10+10)

name       old speed      new speed       delta
AllZero-8  3.84GB/s ± 1%  48.59GB/s ± 1%  +1166.51%  (p=0.000 n=10+10)
 2022-09-24 21:18:48 +02:00
Michael Eischer f414db987d gofmt all files 
Apparently the rules for comment formatting have changed with go 1.19.
 2022-08-19 19:12:26 +02:00
MichaelEischer 443cc49afd
Merge pull request #3830 from MichaelEischer/cleanup-repo 
Extract Load/SaveTree/JSONUnpacked from repository
 2022-07-23 10:46:13 +02:00
Michael Eischer c44b21d366 restorer: extract hardlinks index from restic package 2022-07-17 13:45:42 +02:00
Michael Eischer 8c11fc3ec9 crypto: move crypto buffer helpers 2022-07-17 13:42:23 +02:00
Michael Eischer 89d3ce852b repository: extract Load/StoreJSONUnpacked 
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
 2022-07-17 13:22:00 +02:00
Michael Eischer fbcbd5318c repository: extract LoadTree/SaveTree 
The repository has no real idea what a Tree is. So these methods never
belonged there.
 2022-07-17 13:11:28 +02:00
Michael Eischer 6f53ecc1ae adapt workers based on whether an operation is CPU or IO-bound 
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.

Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
 2022-07-03 12:19:26 +02:00
Michael Eischer 120ccc8754 repository: Rework blob saving to use an async pack uploader 
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.

Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.

To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
 2022-07-02 22:42:34 +02:00
Alexander Neumann 99634c0936 Return real size from SaveBlob 2022-07-02 18:55:12 +02:00
Michael Eischer 6fb408d90e repository: implement pack compression 2022-04-30 11:34:10 +02:00
Alexander Neumann db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings 
Fix golangci lint warnings
 2022-03-29 11:45:10 +02:00
Michael Eischer c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer 243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer 27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
Michael Eischer f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer 921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer 2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar 2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar d4225ec803 Simplify buffer growing in Restorer.verifyFile 
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
 2021-09-19 13:10:55 +02:00
greatroar de8521ae56 Report number of successes from Restorer.VerifyFiles 
Previously, the number of attempts was reported.
 2021-09-19 13:10:44 +02:00
greatroar bb066cf7d3 Concurrent Restorer.VerifyFiles 
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
 2021-09-19 13:10:41 +02:00
greatroar 556424d61b Reuse buffer in Restorer.VerifyFiles 
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
 2021-09-19 12:55:31 +02:00
greatroar 973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Alexander Weiss 81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
Michael Eischer bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer e8bbb05328 restore: Correctly handle partial pack download errors 
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
 2021-06-29 20:54:16 +02:00
MichaelEischer cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate 
Use FcntlFstore to preallocate on Mac
 2021-04-10 22:47:56 +02:00
greatroar 23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
Michael Eischer f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
Michael Eischer 47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
Alexander Weiss 5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss 573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
Alexander Neumann 04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann 0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Michael Eischer 69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Weiss 34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss 7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss 07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss 3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Alexander Neumann 36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Weiss aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss 92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann 56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings 
Fix some goreport warnings
 2020-10-12 20:44:56 +02:00
Alexander Neumann c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo 
Replace most usages of context.TODO()
 2020-10-10 15:03:44 +02:00