mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
3,040 Commits 7 Branches 49 Tags 76 MiB
Commit Graph

3040 Commits

Author SHA1 Message Date
Alexander Neumann a3f8e9dfa7 Add entry to CHANGELOG 2017-10-21 12:22:50 +02:00
Alexander Neumann 982810f7cc Merge pull request #1368 from TobyLL/master 
Ignore comments (lines starting with #) in the --files-from file
 2017-10-21 12:21:11 +02:00
Alexander Neumann 90b96d19cd Merge pull request #1365 from felix9/fix_1068 
Fix failure to detect some legacy s3 repos
 2017-10-21 12:19:58 +02:00
Alexander Neumann 6a52bb6f54 Merge pull request #1374 from restic/update-blazer 
Update github.com/kurin/blazer
 2017-10-21 11:26:49 +02:00
Alexander Neumann cacaa4393f Merge pull request #1373 from restic/check-gopkg-lock 
CI: Check the vendor dir and Gopkg.lock
 2017-10-21 11:14:30 +02:00
Alexander Neumann d63ab4e9a4 Merge pull request #1358 from prattmic/chunk_size 
gs: add option to set chunk size
 2017-10-21 11:13:48 +02:00
Alexander Neumann ca6daec8dd CI: Check the vendor dir and Gopkg.lock 2017-10-21 10:50:38 +02:00
Alexander Neumann c87f2420a6 Update github.com/kurin/blazer 2017-10-21 10:30:39 +02:00
Alexander Neumann f5bbbc52f4 Update issue template 2017-10-21 10:17:58 +02:00
TobyLL 9e3dde8ec7 Fix whitespace 2017-10-19 15:52:06 +01:00
TobyLL 9dba182e51 Ignore comments (lines starting with #) in the --files-from file 2017-10-19 15:48:22 +01:00
Felix Lee 944fc857eb Fix failure to detect some legacy s3 repos 
Sometimes s3 listobjects for a directory includes an entry for that
directory. The restic s3 backend doesn't expect that and returns
an error.

Symptom is:
  ReadDir: invalid key name restic/key/, removing prefix
     restic/key/ yielded empty string

I'm not sure when s3 does that; I'm unable to reproduce it myself.

But in any case, it seems correct to ignore that when it happens.

Fixes #1068
 2017-10-18 13:45:31 -07:00
Alexander Neumann 7507a658ac Merge pull request #1362 from restic/fix-backend-retry 
Fix Save() backend retry
 2017-10-18 07:45:56 +02:00
Michael Pratt 9fa4f5eb6b gs: disable resumable uploads 
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
 2017-10-17 21:12:04 -07:00
Alexander Neumann ce4d71d626 backend: Add partial read failure to error backend 2017-10-17 22:11:38 +02:00
Alexander Neumann 8e2ef3f38b cache: Store first, then store in the backend 
Store the file in the backend, then rewind the reader and store it
in the cache afterwards.
 2017-10-17 22:01:20 +02:00
Alexander Neumann 8dc952775e backend: Correctly retry Save() calls 
Make sure the given reader is an io.Seeker and rewind it properly each
time.
 2017-10-17 21:46:38 +02:00
Alexander Neumann 99b6163e27 Add 'dump' to manual 2017-10-16 20:24:47 +02:00
Alexander Neumann beaf55f1fc dump: Allow absolute paths 2017-10-16 20:22:01 +02:00
Alexander Neumann 980bb9059f Add entry to changelog 2017-10-16 20:15:19 +02:00
Alexander Neumann 0e7281eb71 Merge pull request #1346 from fawick/cmdCatFile 
Add dump sub-command for dumping files to stdout
 2017-10-16 20:13:44 +02:00
Alexander Neumann 0b6133d7b5 Merge pull request #1360 from mungomat/fuse_bugfix 
fuse: bugfix: ignore empty tags
 2017-10-16 20:07:05 +02:00
Tobias Klein b57ca64275 fuse: bugfix: ignore empty tags 2017-10-16 16:22:08 +02:00
Alexander Neumann faadbd734b Add entry to CHANGELOG 2017-10-14 20:29:47 +02:00
Alexander Neumann 88b0a93409 Merge pull request #1353 from restic/backend-retry 
Retry failed backend requests
 2017-10-14 20:28:57 +02:00
Alexander Neumann 4a995105a9 sftp: Fix Delete() 2017-10-14 16:08:15 +02:00
Alexander Neumann 7fe496f983 Ensure TestDelete runs last 2017-10-14 16:04:29 +02:00
Alexander Neumann e56370eb5b Remove Deleter interface 2017-10-14 16:04:29 +02:00
Alexander Neumann b8af7f63a0 backend test: Always remove files for TestList 2017-10-14 15:56:25 +02:00
Alexander Neumann 3eea555155 Add dependency cenkalti/backoff 2017-10-14 15:56:25 +02:00
Alexander Neumann 897c923cc9 Retry failed backend requests 2017-10-14 15:56:25 +02:00
Alexander Neumann 67193e3deb Improve error message when creating lock failed 2017-10-14 15:56:25 +02:00
Alexander Neumann 0e722efb09 backend: Add Delete() to restic.Backend interface 2017-10-14 15:56:25 +02:00
Alexander Neumann 3736f33ebf Merge pull request #1352 from tyll/manual_links 
Fix links to manual
 2017-10-14 14:54:16 +02:00
Fabian Wickborn d1d9c3f9d7 Renamed cmd_catfile.go to cmd_dump.go 2017-10-14 13:55:21 +02:00
Fabian Wickborn cd5cbe0910 Rename debug dump related variable and run function 2017-10-14 13:55:00 +02:00
Fabian Wickborn 814e992c0b Rename subcommand catfile to dump 2017-10-14 11:34:04 +02:00
Till Maas 660fe78735 Fix links to manual 2017-10-13 20:01:19 +02:00
Fabian Wickborn 87d084e18c Add subcommand dump 2017-10-12 20:24:34 +02:00
Alexander Neumann 9ce2a73fc5 Add issue and PR to the changelog entry 2017-10-12 20:20:06 +02:00
Alexander Neumann f2314b26ba Move 'dump' to 'debug dump' 2017-10-12 20:18:45 +02:00
Alexander Neumann 74dcf41f25 Disable 'dump' for non-debug builds 2017-10-12 20:14:48 +02:00
Alexander Neumann b6ba30186f Merge pull request #1336 from rmdashrf/rate-limit-backup 
Add basic rate limiting to backup
 2017-10-12 20:08:55 +02:00
rmdashrf 32637a0328 Basic rate limiting implementation. 
Added `--limit-upload` and `--limit-download` flags to rate limit
backups and restores.
 2017-10-11 20:01:20 -07:00
Alexander Neumann 0addd90e14 Merge pull request #1345 from harshavardhana/creds 
Refactor credentials management to support multiple mechanisms.
 2017-10-11 21:01:19 +02:00
Alexander Neumann 1b5ee5b10a Merge pull request #1343 from mungomat/fuse_speedup 
fuse mount: speedup
 2017-10-11 20:55:52 +02:00
Harshavardhana 042adeb5d0 Refactor credentials management to support multiple mechanisms. 
This PR adds the ability of chaining the credentials provider,
such that restic as a tool attempts to honor credentials from
multiple different ways.

Currently supported mechanisms are

 - static (user-provided)
 - IAM profile (only valid inside configured ec2 instances)
 - Standard AWS envs (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
 - Standard Minio envs (MINIO_ACCESS_KEY, MINIO_SECRET_KEY)

Refer https://github.com/restic/restic/issues/1341
 2017-10-09 12:51:39 -07:00
Tobias Klein 7e4ce0dacc fuse mount: speedup 2017-10-09 10:26:56 +02:00
Alexander Neumann 8ceb22fe8a Add entry to CHANGELOG 2017-10-08 09:48:22 +02:00
Alexander Neumann c5553ec855 Merge pull request #1276 from fawick/supply_ca_cert 
Add REST backend option to use CA root certificate
 2017-10-08 09:47:23 +02:00