mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
5,594 Commits 7 Branches 48 Tags 92 MiB
Commit Graph

1079 Commits

Author SHA1 Message Date
Alexander Neumann 4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output 
Fix windows terminal output for mintty
 2021-05-18 09:29:24 +02:00
Michael Eischer cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer 75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer 55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
MichaelEischer 6d8ceefd67
Merge pull request #3373 from greatroar/simplify-limiter 
Simplify internal/limiter
 2021-05-14 20:27:31 +02:00
MichaelEischer 64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc 
Check for ENOSPC and remove broken files in SFTP
 2021-05-13 20:09:38 +02:00
greatroar ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
Michael Eischer 7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer 80564a9bc9 Properly detect mintty output redirection 
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
 2021-04-11 20:02:09 +02:00
Michael Eischer 5e6af77b7a Unify interactive terminal detection code 
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
 2021-04-11 20:02:09 +02:00
MichaelEischer cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate 
Use FcntlFstore to preallocate on Mac
 2021-04-10 22:47:56 +02:00
greatroar 23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
rawtaz 74c0607c92
Merge pull request #3319 from MichaelEischer/skip-prealloc-test 
restorer: Skip preallocate test if not supported by the filesystem
 2021-04-07 18:59:06 +02:00
greatroar dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
Alexander Neumann 88a23521dd
Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check 
s3: Fix sanity check
 2021-03-11 13:13:46 +01:00
greatroar 187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer 2a9f0f19b6 s3: Fix sanity check 
The sanity check shouldn't replace the error message if there is already
one.
 2021-03-08 20:23:57 +01:00
Michael Eischer f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
Michael Eischer a293fd9aef local: Ignore files in intermediate folders 
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
 2021-02-27 13:47:55 +01:00
Alexander Neumann 9985368d46
Merge pull request #3255 from MichaelEischer/restorer-check-error 
restorer: Check dropped error
 2021-02-03 16:10:46 +01:00
Alexander Neumann 2dd592a06c
Merge pull request #3254 from lorenz/allow-http2 
Allow HTTP/2
 2021-02-02 21:08:20 +01:00
Michael Eischer 47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
Alexander Weiss 5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss 573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
Lorenz Brun 5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Michael Eischer e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann 0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap 
backup: Fix total size for overlapping targets
 2021-01-30 21:17:21 +01:00
Alexander Neumann 04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann 0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann 200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann 3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann 16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann 75f53955ee errcheck: Add error checks 
Most added checks are straight forward.
 2021-01-30 20:02:37 +01:00
Michael Eischer a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Michael Eischer 8a486eafed gs: Don't drop error when finishing upload 
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
 2021-01-30 13:31:32 +01:00
Alexander Neumann cdd704920d azure: Pass data length to Azure libray 
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
 2021-01-29 21:08:41 +01:00
Alexander Neumann bbdf18c4a2
Merge pull request #3176 from MichaelEischer/backend-content-length 
Pass upload size to backends and sanity check it
 2021-01-29 20:33:44 +01:00
Michael Eischer 1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer c73316a111 backends: add sanity check for the uploaded file size 
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
 2021-01-29 13:51:51 +01:00
Michael Eischer 4526d5d197 swift: explicitly pass upload size to library 
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
 2021-01-29 13:50:46 +01:00
Michael Eischer dca9b6f5db azure: explicitly pass upload size 
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
 2021-01-29 13:50:46 +01:00
Alexander Neumann 5c617859ab backup/scanner: Fix total size for overlapping targets 
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
 2021-01-29 11:31:36 +01:00
Alexander Neumann 81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
greatroar 6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
MichaelEischer 43cb26010a
Merge pull request #3242 from greatroar/fprintln 
internal/ui/termstatus: Use Fprintln to get a newline
 2021-01-28 20:34:06 +01:00
Alexander Neumann 4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup 
Speed-up caching/pack download via SFTP
 2021-01-28 14:16:55 +01:00
Alexander Neumann 6aa7e9f9c6
Merge pull request #3174 from MichaelEischer/parallelize-lock-loading 
Parallelize lock file loading
 2021-01-28 13:52:12 +01:00
Alexander Neumann bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading 
Unify index loading
 2021-01-28 13:50:42 +01:00
greatroar b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Michael Eischer 68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer 313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer 505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer 258ce0c1e5 parallel: report progress for StreamTrees 
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
 2021-01-28 11:08:43 +01:00
Michael Eischer f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer 6e03f80ca2 check: Split the parallelized tree loader into a reusable component 
The actual code change is minimal
 2021-01-28 11:08:43 +01:00
Michael Eischer 1d7bb01a6b check: Cleanup tree loading and switch to use errgroup 
The helper methods are now wired up in the Structure method.
 2021-01-28 11:08:43 +01:00
Alexander Neumann a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter 
Don't print progress on non-interactive terminals
 2021-01-28 10:53:38 +01:00
Alexander Neumann c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Michael Eischer 678e75e1c2 sftp: enforce use of optimized upload method 
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
 2021-01-03 22:23:53 +01:00
Michael Eischer 6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
Michael Eischer 69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Weiss 34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss 7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss 07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss 3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Alexander Neumann 8b84c96d9d
Merge pull request #3204 from MichaelEischer/archiver-tomb-race 
archiver: fix race condition during worker startup
 2020-12-30 20:04:20 +01:00
Michael Eischer debc4a3a99 archiver: fix race condition during worker startup 
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
 2020-12-30 17:31:22 +01:00
MichaelEischer 31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183 
Fix tag handling for multiple tag lists
 2020-12-29 18:44:38 +01:00
Alexander Neumann b2efa0af39
Merge pull request #3164 from MichaelEischer/improve-context-cancel 
Improve context cancel handling in archiver and backends
 2020-12-29 17:03:42 +01:00
Michael Eischer cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer 92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer 34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer 023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer 684600cf42 ui: update status for the backup command on non-interactive terminals 
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
 2020-12-29 16:03:43 +01:00
Michael Eischer 13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer c2ef049f1b ui/progress: don't print progress on non-interactive terminals 
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
 2020-12-29 16:03:43 +01:00
Quentin Lemaire 334d8ce724
feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
greatroar 3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar 18531e3d6f Portability fixes to internal/restic 
syscall.Mknod is not available on AIX.
 2020-12-29 01:35:01 +01:00
Michael Eischer d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer 08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer e483b63c40 retrybackend: Fail operations when context is already canceled 
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
 2020-12-28 21:06:47 +01:00
Michael Eischer fc60b560ba archiver: Let saveTree report a canceled context as an error 
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
 2020-12-28 21:06:47 +01:00
Michael Eischer 736e964317 archiver: Don't loose error if background context is canceled 
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
 2020-12-28 21:06:47 +01:00
DRON-666 332b1896d1 Some `options` fixes 
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
 2020-12-23 23:26:04 +03:00
MichaelEischer cb6b0f6255
Merge pull request #3181 from tWido/feature-no-retry-permission 
Don't retry when "Permission denied" occurs in local backend
 2020-12-23 20:24:29 +01:00
MichaelEischer 1e73aac610
Merge pull request #3179 from aawsome/check-filesize 
check: Remove filesize counter
 2020-12-23 20:22:02 +01:00
Alexander Weiss 2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido 7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
Michael Eischer de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss 68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00