mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
4,511 Commits 7 Branches 49 Tags 92 MiB
Commit Graph

607 Commits

Author SHA1 Message Date
Alexander Neumann 754482fe6c checker: Disable size check for now 2018-07-15 21:52:38 +02:00
Alexander Neumann 92421ec47f restore: Make sure target directory exists 2018-07-15 16:02:04 +02:00
Alexander Neumann 9acc9243ba Add test for not-existing top-level dir and top-level file 2018-07-15 16:00:26 +02:00
Alexander Neumann df64998649 Merge pull request #1882 from duzvik/aws-credentials-priority 
Change AWS credentials priority, to accept AWS_SESSION_TOKEN
 2018-07-14 20:48:42 +02:00
Alexander Neumann abb18a830c Fix test 2018-07-14 11:51:34 +02:00
denis.uzvik 1e42f4f300 S3 backend: accept AWS_SESSION_TOKEN 2018-07-12 16:18:19 +03:00
Alexander Neumann bd742ddb69 cache: Don't recreate CACHEDIR.TAG 2018-07-08 12:05:12 +02:00
Alexander Neumann 7ddf91b65c Merge pull request #1772 from ifedorenko/restore-verify 
Restore verify
 2018-06-23 14:50:29 +02:00
Alexander Neumann 9ffc26883a archiver: Unroll tree 2018-06-20 22:56:41 +02:00
Alexander Neumann 83c51db903 fs: Add helper functions ReadDir/ReadDirNames 2018-06-20 22:53:53 +02:00
Alexander Neumann c2c06ae2c9 walker: Don't ignore empty trees by default 
Closes #1849
 2018-06-17 09:49:03 +02:00
Iku Iwasa f080142137 Support NetBSD without fuse 2018-06-16 15:55:04 +09:00
Igor Fedorenko e206680947 restore: New --verify flag to verify restored files content 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Igor Fedorenko 5fa6dc53cb Refactor: introduced restorer tree visitor 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Igor Fedorenko 26be094f28 Refactor: moved restorer to separate package 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Alexander Neumann 8f26fe271c ls: Use walker for ls 2018-06-09 23:35:20 +02:00
Alexander Neumann 3a86f4852b Add walker for trees in the repo 2018-06-09 23:35:20 +02:00
Alexander Neumann 14aead94b3 filter: Allow double wildcard in ChildMatch 2018-06-09 23:18:13 +02:00
Alexander Neumann fc5439a37a cache: Ensure failed downloads are retried 
This fixes #1833, which consists of two different bugs:

 * The `defer` in `cacheFile()` may remove a channel from the
   `inProgress` map although it is not responsible for downloading the
   file

 * If the download fails, goroutines waiting for the file to be cached
   assumed that the file was there, there was no way to signal the
   error.
 2018-06-09 17:50:56 +02:00
Alexander Neumann c44e808aa5 backup: Fix deadlock 
When the archiver is faster than the scanner, restic deadlocks. This
commit adds a `finished` channel to the struct in `ui/backup.go` so that
scanner results are ignored when the archiver is already finished.

Closes #1834
 2018-06-09 12:15:19 +02:00
Alexander Neumann ab37c6095a Merge pull request #1821 from michaelkoetter/fix-1795 
#1795 use unix.IoctlGetWinsize to get terminal size
 2018-06-07 20:20:06 +02:00
Alexander Neumann 141fabdd09 s3: Pass list errors up to the caller 2018-06-01 22:15:23 +02:00
Alexander Neumann 465700595c azure: Support uploading large files 
Closes #1822
 2018-06-01 14:52:16 +02:00
Jakob Unterwurzacher dd3b9910ee sftp: persist "ssh command exited" error 
If our ssh process has died, not only the next, but all subsequent
calls to clientError() should indicate the error.

restic output when the ssh process is killed with "kill -9":

  Save(<data/afb68adbf9>) returned error, retrying after 253.661803ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/afb68adbf9>) returned error, retrying after 580.752212ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 790.150468ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 1.769595051s: ssh command exited: signal: killed
  [...]
  error in cleanup handler: ssh command exited: signal: killed

Before this patch:

  Save(<data/de698d934f>) returned error, retrying after 252.84163ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 660.236963ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 568.049909ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 2.428813824s: OpenFile: failed to send packet header: write |1: file already closed
  [...]
  error in cleanup handler: failed to send packet header: write |1: file already closed
 2018-05-30 19:28:14 +02:00
Michael Kötter 589c23dc23 #1795 use unix.IoctlGetWinsize to get terminal size 2018-05-27 23:44:48 +02:00
Alexander Neumann 7d9642523b termstatus: Fix panic for non-terminal runs 
Closes #1803
 2018-05-27 12:52:01 +02:00
Alexander Neumann 2a976d795f b2: Remove extra error check 2018-05-26 10:12:30 +02:00
Alexander Neumann bfd923e81e rclone: Respect bandwith limits 2018-05-22 20:48:17 +02:00
Alexander Neumann adb682bc43 archiver: Don't open files with O_NONBLOCK 
This is not necessary any more, we're doing an lstat() before opening
an item, so we already known it's a file and not a pipe.
 2018-05-20 16:11:51 +02:00
Alexander Neumann 1e9744c9a4 archiver: Refuse to save an empty snapshot 2018-05-20 16:11:51 +02:00
Alexander Neumann 9a02f17cc2 archiver: Add tests for Save() for fs.Reader 2018-05-20 16:11:36 +02:00
Steve Kriss b358dd369b S3: rearrange credentials chain to be standard 
Signed-off-by: Steve Kriss <steve@heptio.com>
 2018-05-16 16:49:33 -07:00
Steve Kriss d67b9a32c6 S3: add file credentials to chain 
Signed-off-by: Steve Kriss <steve@heptio.com>
 2018-05-16 16:35:14 -07:00
Alexander Neumann 347a645450 Fix double error message 2018-05-15 11:03:33 +02:00
Alexander Neumann 9f5565b0fc Merge pull request #1735 from mholt/forget-max-age 
forget: Add --max-age policy to set hard cutoff for removing snapshots
 2018-05-14 21:18:06 +02:00
Alexander Neumann 060d8b57e0 Restructure TestApplyPolicy 2018-05-13 12:48:42 +02:00
Alexander Neumann cc627e832b Add custom Duration type 2018-05-13 12:48:42 +02:00
Alexander Neumann 5a0f0e3faa Add support for keeping a range of snapshots 2018-05-13 12:48:42 +02:00
Matthew Holt b52f2aa9a4 forget: Add policy to keep snapshots before a date 2018-05-13 12:48:10 +02:00
Alexander Neumann 60ea2435be Improve error message for readdir/readdirnames 
As mentioned in the forum[1], restic does not include the dir name when
readdir/readdirnames fails.

[1] https://forum.restic.net/t/readdirnames-readdirent-no-such-file-or-directory/653
 2018-05-13 10:34:50 +02:00
Alexander Neumann 159badf5ba Merge pull request #1778 from restic/fix-1771 
archiver: Improve error handling
 2018-05-13 00:13:54 +02:00
Alexander Neumann e43c9202a6 archiver: Make sure backend error is passed up 2018-05-12 23:55:59 +02:00
Alexander Neumann c5e75d1c98 archiver: Add test for early abort on unhandled error 2018-05-12 23:55:59 +02:00
Alexander Neumann 526956af35 archiver: Read files/dirs in order 2018-05-12 23:55:54 +02:00
Alexander Neumann 256104111d archiver: Clarify names 2018-05-12 23:55:54 +02:00
Alexander Neumann 21c83b1725 archiver: Add high-level documentation 2018-05-12 23:55:54 +02:00
Alexander Neumann 581c62ee72 archiver: Improve error handling 
This commit changes how the worker goroutines for saving e.g. blobs
interact. Before, it was possible to get stuck sending an instruction to
archive a file or dir when no worker goroutines were available any more.
This commit introduces a `done` channel for each of the worker pools,
which is set to the channel returned by `tomb.Dying()`, so it is closed
when the first worker returned an error.
 2018-05-12 23:55:54 +02:00
Gaige Lama a63989afcd
Ran gofmt on backend.go 
I ran gofmt on backend.go, this appears to valid edit.
 2018-05-11 11:07:16 -04:00
Alexander Neumann ca4af43c03 archiver: Return low-level errors 
This commit changes the archiver so that low-level errors saving data to
the repo are returned to the caller (instead of being handled by the
error callback function). This correctly bubbles up errors like a full
temp file system and makes restic abort early and makes all other worker
goroutines exit.
 2018-05-10 21:30:09 +02:00
Alexander Neumann 1f2463f42e archiver: Return correct error 2018-05-10 20:48:00 +02:00
Alexander Neumann 157c854d04 backup: Remove errored files from status display 2018-05-10 20:48:00 +02:00
Alexander Neumann b2208bb9c2 Rework termstatus 
This now keeps the cursor at the first column of the first status line
so that messages printed to stdout or stderr by some other part of the
progarm will still be visible. The message will overwrite the status
lines, but those are easily reprinted on the next status update.
 2018-05-07 21:40:07 +02:00
Alexander Neumann 4c25495d68 backup: Hide percent until total size is known 2018-05-06 20:20:25 +02:00
Alexander Neumann fd77646f8b Fix panic for nil-pointer dereference 
Closes #1755
 2018-05-03 20:49:30 +02:00
Alexander Neumann fca4fe4459 backup: Disable status output for --quiet 2018-05-02 21:24:18 +02:00
Alexander Neumann 26757ae2e5 Merge pull request #1750 from restic/archiver-fix-small-things 
backup: Tune user interface a bit
 2018-05-01 23:15:54 +02:00
Alexander Neumann 2218ecd049 archiver: Use lstat before open/fstat 
The previous code tried to be as efficient as possible and only do a
single open() on an item to save, and then fstat() on the fd to find out
what the item is (file, dir, other). For normal files, it would then
start reading the data without opening the file again, so it could not
be exchanged for e.g. a symlink.

This behavior starts the watchdog on my machine when /dev is saved
with restic, and after a few seconds, the machine reboots.

This commit reverts the behavior to the strategy the old archiver code
used: run lstat(), then decide what to do. For normal files, open the
file and then run fstat() on the fd to verify it's still a normal file,
then start reading the data.

The downside is that for normal files we now do two stat() calls
(lstat+fstat) instead of only one. On the upside, this does not start
the watchdog. :)
 2018-05-01 23:05:50 +02:00
Alexander Neumann d0974c155d backup: Tune user interface a bit 
Make non-verbose mode a bit more verbose.
 2018-05-01 22:18:32 +02:00
Alexander Neumann 8026e6fdfb Merge pull request #1749 from restic/add-cache-cmd 
Add 'cache' command to list and cleanup cache dirs
 2018-05-01 21:25:52 +02:00
Alexander Neumann f928aeec34 Add 'cache' command 2018-05-01 16:27:44 +02:00
Alexander Neumann eb6650b201 Add textfile 2018-05-01 14:38:59 +02:00
Alexander Neumann bc68d55e94 fs: Add TestTempFile 2018-05-01 14:38:41 +02:00
Alexander Neumann ecbbd851a1 Merge pull request #1744 from restic/fix-parent-detect 
backup: Clean target paths before finding parent
 2018-05-01 11:13:19 +02:00
Bryce Chidester e9f1721678 http backend: Parse the correct argument when loading --tls-client-cert 
Previously, the function read from ARGV[1] (hardcoded) rather than the
value passed to it, the command-line argument as it exists in globalOptions.

Resolves #1745
 2018-04-30 15:21:09 -07:00
Alexander Neumann 64d98945a6 backup: Clean target paths before finding parent 
This resolves an issue described in the forum where restic could not
find a parent snapshot if the target path ends with a slash:
https://forum.restic.net/t/new-archiver-code-please-test/623/23
 2018-04-30 22:03:11 +02:00
Alexander Neumann c83c03ed63 archiver: Fix blocking on pipes 2018-04-30 15:34:58 +02:00
Alexander Neumann 19b9c881ca fs: Add O_NONBLOCK 2018-04-30 15:34:58 +02:00
Alexander Neumann 4e34325035 archiver: Process dirs concurrently 2018-04-30 15:13:28 +02:00
Alexander Neumann 78bd591c7c archiver: Improve buffer pool 2018-04-30 15:13:28 +02:00
Alexander Neumann 39ac12f6ea archiver: Correct comment 2018-04-30 14:19:07 +02:00
Alexander Neumann 400730afca archiver: Improve memory usage, tune buffer pool 2018-04-30 14:19:07 +02:00
Alexander Neumann d80e108b03 backup: Clear status lines on finish 2018-04-30 14:19:07 +02:00
Alexander Neumann 846c2b6869 backup: Fix ETA calculation for >= 100% 2018-04-30 14:19:07 +02:00
Alexander Neumann 16c314ab7f termstatus: Don't print status if in background 2018-04-28 22:08:11 +02:00
Alexander Neumann 0e78ac92d8 Use new archiver code for backup 2018-04-28 22:08:11 +02:00
Alexander Neumann 1af96fc6dd Add termstatus 2018-04-27 21:42:15 +02:00
Alexander Neumann 38926d8576 Use new archiver code in tests 2018-04-25 14:42:45 +02:00
Alexander Neumann f279731168 Add new archiver code 2018-04-25 14:42:45 +02:00
Alexander Neumann 76b616451f Remove unneeded code 2018-04-23 21:40:33 +02:00
Alexander Neumann fd12a3af20 Remove old archiver code 2018-04-23 21:40:33 +02:00
Alexander Neumann a56b8fad87 repository: Improve buffer pooling 2018-04-22 11:37:05 +02:00
Alexander Neumann b6f98bdb02 node: Fill minimal info 2018-04-22 11:37:05 +02:00
Alexander Neumann c4b2486b7c fs: Add interface and FS implementations 
This adds two implementations of the new `FS` interface: One for the local
file system (`Local`) and one for a single file read from an
`io.Reader` (`Reader`).
 2018-04-22 11:37:05 +02:00
Alexander Neumann 83ca08245b checker: Check metadata size and blob sizes 2018-04-22 11:37:05 +02:00
Alexander Neumann 6a7c23d2ae tree: Add convenience functions 2018-04-22 11:37:05 +02:00
Alexander Neumann cc847a3d6d tree: Improve error for pre-existing node 2018-04-22 11:37:05 +02:00
Alexander Neumann baebf45e2e FindLatestSnapshot: Use absolute paths 2018-04-22 11:37:05 +02:00
Alexander Neumann fa4f438bc1 snapshot: Do not modify slice of paths 2018-04-22 11:37:05 +02:00
Alexander Neumann 4e0b2a8e3a snapshot: correct error handling for filepath.Abs 2018-04-22 11:37:05 +02:00
Alexander Neumann 0532f08048 Add test.Helper, also works with Go 1.8 2018-04-22 11:37:05 +02:00
Alexander Neumann a472868e06 fs: Add TestChdir() 2018-04-22 11:37:05 +02:00
Alexander Neumann e4fdc5eb76 fs: Add IsRegularFile() 2018-04-22 11:37:05 +02:00
Alexander Neumann 09bd924710 Do not restore sockets, correct error handling 
Closes #1730
 2018-04-20 13:53:11 +02:00
Alexander Neumann 577faa7570 local/sftp: Handling non-existing dirs in List() 2018-04-10 21:35:30 +02:00
Alexander Neumann 1c1fede399 Improve error message for orphaned pack files 2018-04-07 10:07:54 +02:00
Eri Bastos 901e1b129c Fixed issue #1608 - Use --time argument properly 
Backups via stdin will now handle --time argument and pass it down as
expected
 2018-04-03 14:40:42 -03:00
Alexander Neumann e68a7fea8a check: Allow filling the cache during check 
Closes #1665
 2018-04-01 13:59:27 +02:00
Alexander Neumann 2e7ec717c1 repository: Move cache preparation into function 2018-04-01 13:59:27 +02:00
Alexander Neumann 3f48e0e0f4 Add extra options to rclone 
For details see https://github.com/restic/restic/pull/1657#issuecomment-377707486
 2018-04-01 10:34:30 +02:00
Alexander Neumann 86f4b03730 Remove unneeded byte counters 2018-04-01 10:18:38 +02:00
Alexander Neumann c43c94776b rclone: Make concurrent connections configurable 2018-04-01 10:18:38 +02:00
Alexander Neumann 0b776e63e7 backend/rclone: Request random file name 
When `/` is requested, rclone returns the list of all files in the
remote, which is not what we want (and it can take quite some time).
 2018-04-01 10:18:38 +02:00
Alexander Neumann 737d93860a Extend first timeout to 60 seconds. 2018-04-01 10:18:38 +02:00
Alexander Neumann 17312d3a98 backend/rest: Ensure base URL ends with slash 
This makes it easier for rclone.
 2018-04-01 10:18:38 +02:00
Alexander Neumann 4d5c7a8749 backend/rclone: Make sure rclone terminates 2018-04-01 10:18:38 +02:00
Alexander Neumann fc0295016a Address code review comments 2018-04-01 10:18:38 +02:00
Alexander Neumann 99b62c11b8 backend/rclone: Stop rclone in case of errors 2018-04-01 10:18:38 +02:00
Alexander Neumann 6d9a029e09 backend/rclone: Prefix all error messages 2018-04-01 10:18:38 +02:00
Alexander Neumann 065fe1e54f backend/rclone: Skip test if binary is unavailable 2018-04-01 10:16:31 +02:00
Alexander Neumann 4dc0f24b38 backend/tests: Drain reader before returning error 2018-04-01 10:16:31 +02:00
Alexander Neumann fe99340e40 Add rclone backend 2018-04-01 10:16:31 +02:00
Alexander Neumann e377759c81 rest: Export Backend struct 2018-04-01 10:16:31 +02:00
Alexander Neumann cabbbd2b14 backend/rest: Export Content-Types 2018-04-01 10:16:31 +02:00
Alexander Neumann cf4cf94418 Move backend/sftp.StartForeground to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann 34f27edc03 Refactor SplitShellStrings 2018-04-01 10:16:31 +02:00
Alexander Neumann 345b6c4694 Move backend/sftp.SplitShellArgs to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann 594256bfa4 Merge pull request #1693 from restic/update-deps 
Update dependencies
 2018-03-30 17:19:08 +02:00
Alexander Neumann 673f0bbd6c Update vendored library github.com/cenkalti/backoff 2018-03-30 11:45:27 +02:00
Alexander Neumann d3f9c8b362 forget: Print policy 2018-03-30 10:24:26 +02:00
Alexander Neumann e5c929b793 Fix rest-server tests 
Since today, the rest-server needs to be explicitly told (via
`--no-auth`) that authentication is not necessary.
 2018-03-24 18:06:21 +01:00
Alexander Neumann 8206f85d2e Merge pull request #1552 from lawrencejones/use-auto-auth 
Automatically load Google auth
 2018-03-18 19:53:30 +01:00
Alexander Neumann 0279fd7212 Merge pull request #1669 from restic/make-tests-faster 
Reduce test runtime
 2018-03-13 19:50:34 +01:00
Alexander Neumann b9ada91054 Reduce data for TestCreateSnapshot 2018-03-11 21:42:39 +01:00
Alexander Neumann c67a8452f7 Disable polynomial check for chunker for tests 2018-03-11 20:59:40 +01:00
Alexander Neumann 5253ef218c Remove TestParallelSaveWithDuplication 2018-03-11 19:49:48 +01:00
Alexander Neumann 0923976909 Remove TestArchiverDuplication 2018-03-11 19:44:25 +01:00
Lawrence Jones 0dfdc11ed9
Automatically load Google auth 
This change removes the hardcoded Google auth mechanism for the GCS
backend, instead using Google's provided client library to discover and
generate credential material.

Google recommend that client libraries use their common auth mechanism
in order to authorise requests against Google services. Doing so means
you automatically support various types of authentication, from the
standard GOOGLE_APPLICATION_CREDENTIALS environment variable to making
use of Google's metadata API if running within Google Container Engine.
 2018-03-11 17:11:25 +00:00
Alexander Neumann e085713b35 Return the first password/key which works 
Closes #1663
 2018-03-11 14:12:21 +01:00
Alexander Neumann e77d8c64a7 Merge pull request #1661 from restic/fix-rest-content-length 
rest: Really set Content-Length HTTP header
 2018-03-10 20:34:30 +01:00
Alexander Neumann b3e1089cf9 Return error message for config decryption failure 
See #1663
 2018-03-09 21:05:35 +01:00
Alexander Neumann fcc9ce81ba rest: Really set Content-Length HTTP header 2018-03-09 20:21:34 +01:00
Nick Craig-Wood 04c4033695 backend/rest: check HTTP error response for List 
Before this change restic would attempt to JSON decode the error
message resulting in confusing `Decode: invalid character 'B' looking
for beginning of value` messages.  Afterwards it will return `List
failed, server response: 400 Bad Request (400)`
 2018-03-08 10:22:43 +00:00
Alexander Neumann d8dcbc89d1 lock: Ignore invalid lock file 
This commit fixes a bug introduced in
e9ea26884739a218bbab0248b634d2821be9c3ea: When an invalid lock is
encountered (e.g. if the file is empty), the code used to ignore that,
but now returns the error.

Now, invalid files are ignored for the normal lock check, and removed
when `restic unlock --remove-all` is run.

Closes #1652
 2018-03-05 20:19:57 +01:00
Alexander Neumann be0a5b7f06 Merge pull request #1649 from jasperla/solaris 
Minimal set of patches to get restic working on Solaris
 2018-03-05 20:00:17 +01:00
Jasper Lievisse Adriaanse 96311d1a2b Add support for illumos/Solaris 
This does come without xattr/fuse support at this point.

NB: not hooking up the integration tests as restic won't compile without
    cgo with Go < 1.10.
 2018-03-04 20:11:29 +00:00
Alexander Neumann da77f4a2e2 Merge pull request #1647 from duzvik/aws-session-token 
Change priority of AWS credential providers to accept AWS_SESSION_TOKEN
 2018-03-04 20:54:56 +01:00
denis.uzvik 6bb1bcce03 Change priority of AWS credential providers to accept AWS_SESSION_TOKEN 2018-03-04 19:58:27 +02:00
Alexander Neumann 929afc63d5 Use int64 for the length in the RewindReader 2018-03-04 10:40:42 +01:00
Alexander Neumann 99f7fd74e3 backend: Improve Save() 
As mentioned in issue [#1560](https://github.com/restic/restic/pull/1560#issuecomment-364689346)
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.

The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
 2018-03-03 15:49:44 +01:00
denis.uzvik 5873ab4031 Ignore s3 AccessDenied error, during creation of repository 2018-03-02 10:47:20 +02:00
Alexander Neumann c323f73bf9 Ignore files in the repo with invalid names 
Closes #1641
 2018-02-26 20:53:38 +01:00
Alexander Neumann aef5e03731 Merge pull request #1638 from restic/fix-list-retry 
backend/retry: return worker function error and abort
 2018-02-25 21:20:08 +01:00
Alexander Neumann dc1154c8ad Merge pull request #1556 from ifedorenko/check-subset 
Add --read-data-subset flag to check command
 2018-02-24 14:53:20 +01:00
Alexander Neumann 93210614f4 backend/retry: return worker function error and abort 
This is a bug fix: Before, when the worker function fn in List() of the
RetryBackend returned an error, the operation is retried with the next
file. This is not consistent with the documentation, the intention was
that when fn returns an error, this is passed on to the caller and the
List() operation is aborted. Only errors happening on the underlying
backend are retried.

The error leads to restic ignoring exclusive locks that are present in
the repo, so it may happen that a new backup is written which references
data that is going to be removed by a concurrently running `prune`
operation.

The bug was reported by a user here:
https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
 2018-02-24 13:26:13 +01:00
Toby Burress 08a5281bd4 Incorporate PR review comments. 2018-02-22 17:37:10 +00:00
Toby Burress cdb48a8970 Add tests for the eager-header refactor. 2018-02-22 01:14:04 +00:00
Toby Burress 4fd5f0b8a9 Refactor the eager-header reads for readability. 
This pulls the header reads into a function that works in terms of the
number of records requested.  This preserves the existing logic of
initially reading 15 records and then falling back if that fails.

In the event of a header with more than 15 records, it will read all
records, including the already-seen final 15 records.
 2018-02-22 00:45:40 +00:00
Igor Fedorenko 19e7803ac6 Fixed unexpected 'pack file cannot be listed' error 
Fixes #1633

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-02-20 21:28:57 -05:00
Igor Fedorenko 07d080830e Add --read-data-subset flag to check command 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-02-18 23:31:27 -05:00
Alexander Neumann be02008025 Merge pull request #1611 from qbit/master 
On OpenBSD only root can set sticky bit
 2018-02-18 12:57:54 +01:00
Alexander Neumann 29da86b473 Merge pull request #1623 from restic/backend-relax-restrictions 
backend: Relax requirement for new files
 2018-02-18 12:56:52 +01:00
Alexander Neumann b5062959c8 backend: Relax requirement for new files 
Before, all backend implementations were required to return an error if
the file that is to be written already exists in the backend. For most
backends, that means making a request (e.g. via HTTP) and returning an
error when the file already exists.

This is not accurate, the file could have been created between the HTTP
request testing for it, and when writing starts. In addition, apart from
the `config` file in the repo, all other file names have pseudo-random
names with a very very low probability of a collision. And even if a
file name is written again, the way the restic repo is structured this
just means that the same content is placed there again. Which is not a
problem, just not very efficient.

So, this commit relaxes the requirement to return an error when the file
in the backend already exists, which allows reducing the number of API
requests and thereby the latency for remote backends.
 2018-02-17 22:39:18 +01:00
Igor Fedorenko ab040d8811 Introduced repository.DownloadAndHash helper 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-02-16 21:13:11 -05:00
Igor Fedorenko d58ae43317 Reworked Backend.Load API to retry errors during ongoing download 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-02-16 21:12:14 -05:00
Aaron Bieber 99d88ad297
Disable the 'testSticky' test on OpenBSD. Only root can set sticky. 2018-02-11 07:46:31 -07:00
Alexander Neumann 514f1b8917 Relax timeout backend test 2018-02-10 12:53:38 +01:00
Igor Fedorenko aa333f4d49 Implement RetryBackend.List() 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-29 22:14:12 -05:00
Alexander Neumann 2369da158f Merge pull request #1592 from ncw/helpful-tests 
Make backend tests more helpful
 2018-01-28 10:09:35 +01:00
Nick Craig-Wood fb62da1748 Make backend tests more helpful 
* In TestList check that backend is empty first
  * Improve error message in TestBackend
 2018-01-27 21:36:35 +00:00
Alexander Neumann 5dc8d3588d GS: Use generic http transport 
During the development of #1524 I discovered that the Google Cloud
Storage backend did not yet use the HTTP transport, so things such as
bandwidth limiting did not work. This commit does the necessary magic to
make the GS library use our HTTP transport.
 2018-01-27 20:12:34 +01:00
Alexander Neumann c34db983d8 Read TLS client cert and key from the same file 2018-01-27 14:02:01 +01:00
Bryce Chidester e805b968b1 Support for TLS client certificate authentication 
This adds --tls-client-cert and --tls-client-key parameters and enables use
of that certificate/key pair when connecting to https servers.
 2018-01-27 13:18:22 +01:00
Alexander Neumann f99c95c766 archiver: Fix intermediate index upload 
A user discovered[1] that when the backup finishes during the upload of
an intermediate index, the upload is cancelled and the index never fully
saved, but the snapshot is saved and the backup finalizes without an
error. This lead to a situation where a snapshot references data that is
contained in the repo, but not referenced in any index, leading to
strange error messages.

This commit uses a dedicated context to signal the intermediate index
uploading routine to terminate after the last index has been uploaded.
This way, an upload running when the backup finishes is completed before
the routine terminates and the snapshot is saved.

[1] https://forum.restic.net/t/error-loading-tree-check-prune-and-forget-gives-error-b2-backend/406
 2018-01-26 22:01:07 +01:00
Alexander Neumann cccb2fc7e7 Merge pull request #1583 from restic/close-open-backend-files 
Close backend files in case of errors
 2018-01-26 21:57:28 +01:00
Alexander Neumann ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log 
Optimize debug logs
 2018-01-26 21:57:18 +01:00
Alexander Neumann e11a183578 Merge pull request #1588 from restic/fix-sftp-without-tty 
sftp: Allow running ssh without a tty
 2018-01-26 21:56:41 +01:00
Alexander Neumann 6eb2d76435 index: Lower parallel load to 4 2018-01-26 21:10:38 +01:00
Alexander Neumann f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann ca3c566a99 index: Check structure after saving a new index 2018-01-26 21:07:16 +01:00
Alexander Neumann 7719cf88d9 b2: Check timeout 2018-01-26 21:07:05 +01:00
Alexander Neumann 00e905ebe6 sftp: Allow running ssh without a tty 2018-01-26 19:21:14 +01:00
Alexander Neumann 909d9273cc Close backend files in case of errors 2018-01-25 21:05:57 +01:00
Alexander Neumann 663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Alexander Neumann ed99f53786 debug: Auto-shorten IDs passed as parameters 2018-01-25 20:49:26 +01:00
Igor Fedorenko abc4027083 Use errors.Cause in backend TestListCancel 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-25 08:53:50 -05:00
Alexander Neumann 9c55e8d69c Merge pull request #1549 from MJDSys/more_index_lookup_avoids 
More optimizations to avoid calling Index.Lookup()
 2018-01-24 20:53:30 +01:00
Alexander Neumann be205563be Merge pull request #1575 from ifedorenko/1567_optimize-repository-ListPack 
Optimize Repository.ListPack()
 2018-01-24 20:01:15 +01:00
Alexander Neumann fc6c341b26 Merge pull request #1574 from ifedorenko/1567_optimize-pack-readHeader 
Optimize pack readHeader() implementation
 2018-01-24 19:54:14 +01:00
Alexander Neumann f86c141b83 Merge pull request #1577 from EdwardBetts/patch-1 
Correct a spelling mistake.
 2018-01-24 19:44:49 +01:00
Edward Betts b5c23695c9
Correct a spelling mistake. 2018-01-24 12:09:41 +00:00
Igor Fedorenko 0084e42cb6 Optimize Repository.ListPack() 
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-23 22:39:51 -05:00
Matthew Dawson fe33c05a20
debug/log: Add benchmarks for calling the logging function 
Add some benchmarks for calling Log, both with a static string
along with calling the ID.Str and ID.String functions.
 2018-01-23 22:30:00 -05:00
Matthew Dawson 3789e55e20
repostiory/index: Remove logging from Lookup function. 
The logging in these functions double the time they take to execute.
However, it is only really useful on failures, which are better
reported by the calling functions.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              897           395           -55.96%
BenchmarkMasterIndexLookupMultipleIndex-6            2001          1090          -45.53%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       492           215           -56.30%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     1649          912           -44.69%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              1              -88.89%
BenchmarkMasterIndexLookupMultipleIndex-6            19             1              -94.74%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       6              0              -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     16             0              -100.00%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           96            -40.00%
BenchmarkMasterIndexLookupMultipleIndex-6            240           96            -60.00%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       48            0             -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     128           0             -100.00%
 2018-01-23 22:28:38 -05:00
Matthew Dawson 3a16148447
archiver/archiver: Use Index.Has() instead of Index.Lookup() in isKnownBlob 
Index.Has() is a faster then Index.Lookup() for checking if a blob exists
in the index.  As the returned data is never used, this avoids a ton
of allocations.
 2018-01-23 22:26:10 -05:00
Matthew Dawson df2c03a6a4
repository/master_index: Optimize Index.Lookup() 
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
 2018-01-23 22:25:56 -05:00
Matthew Dawson ebce4b2581
repository/index: Speed up benchmarks and tests 
When setting up the index used for benchmarking, use math/rand instead of
crypto/rand since the generated ids don't need to be evenly distributed,
and not be secure against guessing.  As such, use a different random id
function (only available during tests) that uses math/rand instead.
 2018-01-23 22:21:05 -05:00
Igor Fedorenko 953f3d55ee Optimize pack readHeader() implementation 
Load pack header length and 15 header entries with single backend
request. This eliminates separate header Load() request for most pack
files and significantly improves index.New() performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-23 22:02:25 -05:00
Alexander Neumann 7e6bfdae79 backend/rest: Implement REST API v2 2018-01-23 23:15:26 +01:00
Alexander Neumann 59782e347c repository: Fix repack test 
This reduces the chance of duplicate blobs, otherwise the tests fail
(make the contents of a blob depend on a pseudo-random number instead of
the size, sizes may be duplicate).
 2018-01-23 23:14:05 +01:00
Alexander Neumann e835abeceb backend/test: Reliably trigger timeout error 2018-01-23 23:14:05 +01:00
Alexander Neumann 8e812b7ac0 errors: Make Cause() unwrap *url.Error 2018-01-23 23:14:05 +01:00
Alexander Neumann 685ce719ad Add comments for List() 2018-01-23 23:14:05 +01:00
Alexander Neumann b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann e9ea268847 Change List() implementation for all backends 2018-01-21 21:15:09 +01:00
Alexander Neumann c4e9d5d11e backend: Add tests for new List() function 2018-01-21 18:35:37 +01:00
Alexander Neumann 52230b8f07 backend: Rework List() 
For a discussion see #1567
 2018-01-21 18:35:37 +01:00
Alexander Neumann 2130897ce0 rest: Add test for external server 2018-01-20 10:25:47 +01:00
Alexander Neumann 67da240068 rest: Refactor backend tests 2018-01-20 10:25:37 +01:00
Alexander Neumann 1046eabf95 rest: Remove unneeded tempdir 2018-01-20 10:13:04 +01:00
Alexander Neumann 200415e0a1 Merge pull request #1564 from restic/sftp-password-prompt 
sftp: Prompt for password, don't terminate on SIGINT
 2018-01-20 09:48:17 +01:00
Alexander Neumann bcb6881ffb Merge pull request #1548 from ifedorenko/checker-backend-Test 
checker: Optimize checker.Packs()
 2018-01-19 13:05:35 +01:00
Igor Fedorenko 231076fa4a checker: Optimize checker.Packs() 
Use result of single repository.List() to find both missing and
orphaned data packs. For 500GB repository this eliminates ~100K
repository.Test() calls and improves check time by >30M in my
environment (~45min before this change and ~7min after).

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-18 20:50:39 -05:00
Alexander Neumann 0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann c31a5e7e5c Add argument to Skipf() 2018-01-17 23:14:37 +01:00
Alexander Neumann 05958caf6e sftp: Prompt for password, don't terminate on SIGINT 
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
 2018-01-17 23:02:47 +01:00
Alexander Neumann 97565d68a2 fuse/mount: Add option for snapshot template 2018-01-15 14:47:17 +01:00
Alexander Neumann f49f5c5903 fuse: Don't return an error for EOF 2018-01-13 10:14:10 +01:00
Alexander Neumann d89f2e5226 fuse: Remove debug log message 2018-01-13 10:10:07 +01:00
Matthew Dawson 539599d1f1 repository/index: Optimize index.Has() 
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
 2018-01-08 21:46:17 +01:00
Alexander Neumann d77a326bb0 Add benchmark for Index.Has() 2018-01-08 21:10:30 +01:00
Alexander Neumann b8620429e2 restorer: Set directory mode as last step 
Closes #1512
 2018-01-07 15:13:24 +01:00
Alexander Neumann aab414b368 Add test case for #1512 2018-01-07 14:50:47 +01:00
Alexander Neumann 5f43e17918 Merge pull request #1535 from ifedorenko/check-retry 
fixed restic-check does not retry backend.Test failures
 2018-01-07 09:43:49 +01:00
Igor Fedorenko 8c550ca011 fixed restic-check does not retry backend.Test failures 
added missing RetryBackend.Test implementation

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-06 23:22:35 -05:00
Alexander Neumann 032621289b errors: Add "Fatal" prefix for Fatalf() 2018-01-06 23:27:47 +01:00
Alexander Neumann b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann aabc0ccaa7 fs: Make HasPathPrefix work with relative paths 2018-01-05 17:38:52 +01:00
Alexander Neumann 18ecd9df30 Improve limiting HTTP based backends 2017-12-29 12:43:49 +01:00
Alexander Neumann 3f7d85360a fuse: Only reload list of snapshots once per minute 2017-12-28 13:18:27 +01:00
Alexander Neumann 6c2b2a58ad backend: Retry deletes 2017-12-22 22:41:28 +01:00
Alexander Neumann bc04ce8e6b Merge pull request #1482 from restic/fix-1478 
backend: Only return top-level files for most dirs
 2017-12-20 18:06:47 +01:00
Alexander Neumann 7d8765a937 backend: Only return top-level files for most dirs 
Fixes #1478
 2017-12-14 19:14:16 +01:00
Alexander Neumann 81a04656c5 fuse: Cleanup node name 2017-12-13 20:21:18 +01:00
Harshavardhana 27ccea6371 Since upgrade to minio-go 4.0 remove workaround 
We added previously a code to fix the issue of chaining
credentials, we do not need this anymore since the
upstream minio-go already has this relevant change.
 2017-12-09 02:01:42 -08:00
Alexander Neumann 8b3b7bc5ef s3: Use context 2017-12-08 22:04:55 +01:00
Alexander Neumann 934ae1b559 Update to minio-go 4 2017-12-08 21:52:50 +01:00
Alexander Neumann 3a5e040b7e Merge pull request #1439 from armhold/propagate-context 
replace ad-hoc context.TODO() with gopts.ctx
 2017-12-08 20:27:36 +01:00
George Armhold 1695c8ed55 use global context for check, debug, dump, find, forget, init, key, 
list, mount, tag, unlock commands

gh-1434
 2017-12-06 07:02:55 -05:00
George Armhold 0dc31c03e1 remove check for context.Canceled 
gh-1434
 2017-12-06 05:38:29 -05:00
Alexander Neumann 82ded35706 node: Correctly compare times 2017-12-03 18:33:35 +01:00
Alexander Neumann 88607fc625 Correctly format all node types 2017-12-03 17:38:55 +01:00
Alexander Neumann 7092af6329 fuse: Handle sockets/fifos/devs correctly 
Closes #1463
 2017-12-03 17:25:00 +01:00
George Armhold 20d78ab0d9 add context to repo.Flush() call in test code 
gh-1434
 2017-12-03 07:32:50 -05:00
George Armhold be24237063 make retry code context-aware. 
detect cancellation in backend, so that retry code does not keep trying
once user has hit ^c

gh-1434
 2017-12-03 07:22:14 -05:00
George Armhold d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation 
can properly trickle down from cmd_*.

gh-1434
 2017-12-03 07:22:14 -05:00
Alexander Neumann 0b44c629f2 retry: Remove file after failed save 2017-11-30 22:05:14 +01:00
Alexander Neumann 410efe0694 Merge pull request #1454 from restic/fix-cache-dir 
Fix cache dir detection
 2017-11-29 18:43:07 +01:00
Alexander Neumann eddb8549ef backup: By default, do not save the access time 
This can be re-enabled with `--with-atime`.
 2017-11-28 21:31:35 +01:00
Alexander Neumann dc38265b54 Merge pull request #1436 from restic/remove-old-cache 
Remove old cache directories
 2017-11-27 21:37:05 +01:00
Alexander Neumann 0f5e38609f restore: Fix restore to relative path 2017-11-26 18:36:48 +01:00
Alexander Neumann 27d29b9853 restorer: Make sure node names are clean 2017-11-26 15:28:45 +01:00
Alexander Neumann 8a171731ba restorer: Add tests for invalid node names 2017-11-26 15:28:37 +01:00
Alexander Neumann 134abbd82b rest: Use client for creating the repository 
Before, creating a new repo via REST would use the defaut HTTP client,
which is not a problem unless the server uses HTTPS and a TLS
certificate which isn't signed by a CA in the system's CA store. In this
case, all commands work except the 'init' command, which fails with a
message like "invalid certificate".
 2017-11-25 20:56:40 +01:00
Alexander Neumann 1ebf0e8de8 Merge pull request #1437 from restic/fix-1292 
s3: Document and remove default prefix
 2017-11-25 11:34:26 +01:00
Alexander Neumann 47b326b7b5 Merge pull request #1423 from harshavardhana/creds 
Fix chaining of credentials for minio-go
 2017-11-24 21:57:52 +01:00
Alexander Neumann f79698dcdd Merge pull request #1410 from armhold/deadlock2 
unify behavior for max http connections across backends
 2017-11-24 21:32:56 +01:00
Alexander Neumann f4bab789b8 cache: Simplify cache dir creation 2017-11-24 20:53:26 +01:00
Alexander Neumann fa893ee477 cache: Add detection code for old cache dirs 2017-11-24 20:53:26 +01:00
Alexander Neumann 262b0cd9d4 s3: Remove default prefix "/restic" 2017-11-21 21:33:09 +01:00
Alexander Neumann e83ec17e95 s3: Correct comment 2017-11-20 22:21:39 +01:00