Michael Eischer
623770eebb
repository: try to recover from invalid blob while repacking
...
If a blob that should be kept is invalid, Repack will now try to request
the blob using LoadBlob. Only return an error if that fails.
2022-07-30 17:37:07 +02:00
Michael Eischer
d0590b7841
prune: Add internal integrity check
...
After repacking every blob that should be kept must have been repacked.
We have seen a few cases in which a single blob went missing, which
could have been caused by a bitflip somewhere. This sanity check might
help catch some of these cases.
2022-07-30 17:37:07 +02:00
Michael Eischer
5cbde03eae
prune: split into smaller functions
2022-07-30 17:37:07 +02:00
Alexander Weiss
7643237da5
prune: separate collecting/printing/pruning
2022-07-30 17:37:07 +02:00
MichaelEischer
4ffd479ba4
Merge pull request #3773 from MichaelEischer/efficient-dir-json
...
Reduce memory usage for large directories/files
2022-07-23 17:47:32 +02:00
Michael Eischer
2ba14160de
Add changelog for the optimized tree serialization
2022-07-23 14:49:08 +02:00
Michael Eischer
4a10ebed15
archiver: reduce memory usage for large files
...
FutureBlob now uses a Take() method as a more memory-efficient way to
retrieve the futures result. In addition, futures are now collected
while saving the file. As only a limited number of blobs can be queued
for uploading, for a large file nearly all FutureBlobs already have
their result ready, such that the FutureBlob object just consumes
memory.
2022-07-23 14:45:07 +02:00
Michael Eischer
b817681a11
archiver: Incrementally serialize tree nodes
...
That way it is not necessary to keep both the Nodes forming a Tree and
the serialized JSON version in memory.
2022-07-23 14:45:07 +02:00
Michael Eischer
c206a101a3
archiver: unify FutureTree/File into futureNode
...
There is no real difference between the FutureTree and FutureFile
structs. However, differentiating both increases the size of the
FutureNode struct.
The FutureNode struct is now only 16 bytes large on 64bit platforms.
That way is has a very low overhead if the corresponding file/directory
was not processed yet.
There is a special case for nodes that were reused from the parent
snapshot, as a go channel seems to have 96 bytes overhead which would
result in a memory usage regression.
2022-07-23 14:45:07 +02:00
Michael Eischer
32f4997733
archiver: remove unused fileInfo from progress callback
2022-07-23 14:16:23 +02:00
Michael Eischer
dcb00fd2d1
archiver: cleanup Saver interface
2022-07-23 14:16:23 +02:00
Michael Eischer
79321a195c
archiver: remove dead attribute from FutureNode
2022-07-23 14:16:23 +02:00
MichaelEischer
049f4c4144
Merge pull request #3730 from MichaelEischer/stricter-check
...
Let `check` warn about legacy variants of the repo format version 1
2022-07-23 14:14:50 +02:00
Michael Eischer
3bf53da672
Add changelog for stricter checks
2022-07-23 11:21:26 +02:00
Michael Eischer
5a6f2f9fa0
Fix S3 legacy layout migration
2022-07-23 11:19:32 +02:00
Michael Eischer
04e49924fb
checker: Fix S3 legacy layout detection
2022-07-23 11:19:32 +02:00
Michael Eischer
768c890fcb
check: Deprecate --check-unused
...
Unused blobs are not a problem but rather expected to exist now that
prune by default does not remove every unused blob. However, the option
has caused questions from users whether a repository is damaged or not,
so just remove that option.
Note that the remaining code is left intact as it is still useful for
our test cases.
2022-07-23 11:19:32 +02:00
Michael Eischer
fcb3ddf181
check: Complain about usage of s3 legacy layout
2022-07-23 11:19:32 +02:00
Michael Eischer
8b8bd4e8ac
check: complain about mixed pack files
2022-07-23 11:19:32 +02:00
MichaelEischer
443cc49afd
Merge pull request #3830 from MichaelEischer/cleanup-repo
...
Extract Load/SaveTree/JSONUnpacked from repository
2022-07-23 10:46:13 +02:00
MichaelEischer
1f5369e072
Merge pull request #3831 from MichaelEischer/move-code
...
Move code out of the restic package and consolidate backend specific code
2022-07-23 10:33:05 +02:00
MichaelEischer
827ab02eea
Merge pull request #3661 from rgammans/azure_sas_support
...
add SAS authentication option for Azure repos
2022-07-23 10:32:03 +02:00
MichaelEischer
e9c39442fb
Merge pull request #3827 from MichaelEischer/backup-doc
...
backup: clarify usage string
2022-07-23 10:31:33 +02:00
Michael Eischer
9729e6d7ef
backend: extract readerat from restic package
2022-07-17 15:29:09 +02:00
Michael Eischer
c44b21d366
restorer: extract hardlinks index from restic package
2022-07-17 13:45:42 +02:00
Michael Eischer
8c11fc3ec9
crypto: move crypto buffer helpers
2022-07-17 13:42:23 +02:00
Michael Eischer
a0cef9f247
limiter: move to internal/backend
2022-07-17 13:40:15 +02:00
Michael Eischer
163ab9c025
mock: move to internal/backend
2022-07-17 13:40:06 +02:00
Michael Eischer
89d3ce852b
repository: extract Load/StoreJSONUnpacked
...
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
2022-07-17 13:22:00 +02:00
Michael Eischer
fbcbd5318c
repository: extract LoadTree/SaveTree
...
The repository has no real idea what a Tree is. So these methods never
belonged there.
2022-07-17 13:11:28 +02:00
MichaelEischer
d9ea1e9ee2
Merge pull request #3290 from aawsome/prune-handle-duplicates
...
prune: Handle duplicate blobs more efficiently
2022-07-17 11:51:54 +02:00
Michael Eischer
715d457aad
prune: code cleanups
2022-07-17 11:41:56 +02:00
Michael Eischer
9be1bd2acc
prune: handle very high duplication of some blobs
...
Suggested-By: Alexander Weiss <alex@weissfam.de>
2022-07-17 11:39:56 +02:00
Alexander Weiss
7478cbf70e
prune: Enhance treatment of duplicates
2022-07-17 00:22:23 +02:00
Michael Eischer
b2043e8198
backup: clarify usage string
...
Using the `--files-from` options it is possible to run `backup` without
specifying any source paths directly on the command line.
2022-07-17 00:03:22 +02:00
Michael Eischer
5639c41b6a
azure: Strip ? prefix from sas token
2022-07-16 23:55:18 +02:00
Roger Gammans
64a7ec5341
azure: add SAS authentication option
2022-07-16 23:55:18 +02:00
MichaelEischer
6cbeb4a9f9
Merge pull request #3825 from restic/rawtaz-doc-gdrive
...
doc: Add note about using rclone for Google Drive
2022-07-16 19:21:59 +02:00
rawtaz
f5c219f5a2
doc: Add note about using rclone for Google Drive
...
It wasn't clear that Google Cloud Storage and Google Drive are two different services and that one should use the rclone backend for the latter. This commit adds a note with this information.
2022-07-16 13:22:38 +02:00
MichaelEischer
d71b29221b
Merge pull request #3822 from JsBergbau/doc-max-compression
...
Added hint for --compression max in migration process
2022-07-12 23:02:45 +02:00
Michael Eischer
71ff6b77f0
doc: Rework hint to repack with max compression
2022-07-12 21:24:40 +02:00
MichaelEischer
6970d05d47
Merge pull request #3741 from lbausch/repository-wording
...
Wording: change repo to repository
2022-07-12 21:15:33 +02:00
MichaelEischer
3934480da4
Merge pull request #3819 from lbausch/restore-validate-patterns
...
restore: validate include/exclude patterns
2022-07-12 20:58:41 +02:00
MichaelEischer
71a0157c2c
Merge pull request #3820 from lbausch/patch-1
...
Fix wording in changelog template
2022-07-12 20:51:52 +02:00
Michael Eischer
2aad6f24b5
doc: update sample help output
2022-07-12 20:49:01 +02:00
Michael Eischer
ec4dfa3c66
Wording: replace further repo occurrences with repository
2022-07-12 20:48:01 +02:00
JsBergbau
8d3f04aefa
Added hint for --compression max in migration process
...
Added hint for --compression max in migration process. Since this is a onetime process users should be aware of this and consider this step.
2022-07-12 00:08:26 +02:00
Lorenz Bausch
b609523582
Add changelog entry
2022-07-09 22:24:56 +02:00
lbausch
ac96a4138d
Fix wording in changelog template
2022-07-09 22:13:54 +02:00
Lorenz Bausch
7e36ec279d
Test restore fails when using invalid patterns
2022-07-08 20:09:26 +02:00
