Michael Eischer
fd05037e1a
repository: recalibrate index batch allocation size
2022-04-30 11:34:10 +02:00
Michael Eischer
6fb408d90e
repository: implement pack compression
2022-04-30 11:34:10 +02:00
Michael Eischer
362ab06023
init: Add flag to specify created repository version
2022-04-30 10:07:42 +02:00
Michael Eischer
4b957e7373
repository: Implement index/snapshot/lock compression
...
The config file is not compressed as it should remain readable by older
restic versions such that these can return a proper error.
As the old format for unpacked data does not include a version header,
make use of a trick: The old data is always encoded as JSON. Thus it can
only start with '{' or '['. For any other value the first byte indicates
a versioned format. The version is set to 2 for now. Then the zstd
compressed data follows.
2022-04-30 10:07:42 +02:00
Michael Eischer
e597b99b55
repository: Reduce repack workers to prevent deadlock
...
As repack streams packs these occupy one backend connection. Uploading a
new pack also requires a backend connection. To prevent a deadlock
during repack when reaching the backend connections limit, simply limit
the repackWorker count to always leave one connection for uploading.
2022-04-23 11:28:18 +02:00
Alexander Neumann
a059ef90f8
Merge pull request #3702 from MichaelEischer/extend-config-error
...
Print used key name if config fails to load
2022-04-10 20:25:24 +02:00
Michael Eischer
c2aabb2686
Print used key name if config fails to load
2022-04-09 22:38:18 +02:00
Alexander Neumann
04e054465a
Merge pull request #3475 from MichaelEischer/local-sftp-conn-limit
...
Limit concurrent operations for local / sftp backend
2022-04-09 21:33:00 +02:00
Michael Eischer
7b9ae91e04
copy: Load snapshots before indexes
2022-04-09 12:27:25 +02:00
Michael Eischer
cd783358d3
local: Limit concurrent backend operations
...
Use a limit of 2 similar to the filereader concurrency in the archiver.
2022-04-09 12:21:38 +02:00
Michael Eischer
6408686973
repository: Simplify Blob equality check
2022-03-28 22:09:49 +02:00
Michael Eischer
243698680a
crypto: Use helpers for size calculations
2022-03-28 22:09:49 +02:00
Michael Eischer
f78bd14e28
repository: Remove pack implementation details from MasterIndex
2022-03-28 22:09:49 +02:00
Michael Eischer
dc3d77dacc
repository: make saveAndEncrypt private
2022-03-28 22:09:49 +02:00
Michael Eischer
6877e7edbb
repository: Rename LoadAndDecrypt to LoadUnpacked
...
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
Michael Eischer
537b4c310a
copy: Implement by reusing repack
...
The repack operation copies all selected blobs from a set of pack files
into new pack files. For prune the source and destination repositories
are identical. To implement copy, just use a different source and
destination repository.
2022-03-26 20:47:15 +01:00
Alexander Neumann
e682f7c0d6
Add tests for StreamPack
2022-03-21 21:15:03 +01:00
Michael Eischer
bba8ba7a5b
repository: cancel streampack context after error
2022-02-12 20:18:25 +01:00
Michael Eischer
47554a3428
repository: Fix error handling in repack
...
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer
930a00ad54
checker: reuse bufio reader
2022-02-12 20:18:25 +01:00
Michael Eischer
34ebafb8b6
repository: don't crash if blob size is too short
2022-02-12 20:18:25 +01:00
Michael Eischer
becebf5d88
repository: remove unused DownloadAndHash
2022-02-12 20:18:25 +01:00
Michael Eischer
f1e58e7c7f
checker: rewrite ReadData to stream packs
2022-02-12 20:18:25 +01:00
Michael Eischer
f40abd92fa
restorer: convert to use StreamPack
2022-02-12 20:18:25 +01:00
Michael Eischer
f00f690658
repository: stream packs during repacking
2022-02-12 20:18:25 +01:00
Michael Eischer
c4a2bfcb39
repository: Add StreamPacks function
...
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer
153e2ba859
repository: Implement lisiting blobs per pack file
2022-02-12 20:18:24 +01:00
greatroar
8d2996eaaa
Replace siphash by hash/maphash
...
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.
Benchmark results on linux/amd64, -benchtime=3s:
name old time/op new time/op delta
IndexHasUnknown-8 50.6ns ±10% 41.0ns ±19% -18.92% (p=0.000 n=9+10)
IndexHasKnown-8 52.6ns ±12% 41.5ns ±12% -21.13% (p=0.000 n=9+10)
IndexMapHash-8 3.64µs ± 1% 2.00µs ± 0% -45.09% (p=0.000 n=10+9)
IndexAlloc-8 700ms ± 1% 601ms ± 6% -14.18% (p=0.000 n=8+10)
IndexAllocParallel-8 205ms ± 5% 192ms ± 8% -6.18% (p=0.043 n=10+10)
MasterIndexAlloc-8 319ms ± 1% 279ms ± 5% -12.58% (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8 156ns ± 8% 147ns ± 6% -5.46% (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8 150ns ± 7% 142ns ± 8% -5.69% (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8 74.4ns ± 6% 72.0ns ± 9% ~ (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8 67.4ns ± 9% 65.5ns ± 7% ~ (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8 461ns ± 2% 445ns ± 2% -3.49% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8 408ns ±11% 378ns ± 5% -7.22% (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8 479ns ± 1% 437ns ± 4% -8.82% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8 406ns ± 8% 343ns ±15% -15.44% (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8 480ns ± 1% 455ns ± 5% -5.15% (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8 391ns ±18% 382ns ± 8% ~ (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8 71.0ns ± 8% 57.2ns ±11% -19.36% (p=0.000 n=9+10)
PackerManager-8 254ms ± 1% 254ms ± 1% ~ (p=0.285 n=15+15)
name old speed new speed delta
IndexMapHash-8 1.12GB/s ± 1% 2.05GB/s ± 0% +82.13% (p=0.000 n=10+9)
PackerManager-8 208MB/s ± 1% 207MB/s ± 1% ~ (p=0.281 n=15+15)
name old alloc/op new alloc/op delta
IndexMapHash-8 0.00B 0.00B ~ (all equal)
IndexAlloc-8 400MB ± 0% 400MB ± 0% ~ (p=1.000 n=9+10)
IndexAllocParallel-8 401MB ± 0% 401MB ± 0% +0.00% (p=0.000 n=10+10)
MasterIndexAlloc-8 258MB ± 0% 262MB ± 0% +1.42% (p=0.000 n=9+10)
PackerManager-8 73.1kB ± 0% 73.1kB ± 0% ~ (p=0.382 n=13+13)
name old allocs/op new allocs/op delta
IndexMapHash-8 0.00 0.00 ~ (all equal)
IndexAlloc-8 907k ± 0% 907k ± 0% -0.00% (p=0.000 n=10+10)
IndexAllocParallel-8 907k ± 0% 907k ± 0% +0.00% (p=0.009 n=10+10)
MasterIndexAlloc-8 327k ± 0% 317k ± 0% -3.06% (p=0.000 n=10+10)
PackerManager-8 744 ± 0% 744 ± 0% ~ (all equal)
2021-09-19 16:05:18 +02:00
Alexander Weiss
81876d5c1b
Simplify cache logic
2021-09-03 21:01:00 +02:00
Michael Eischer
9aa2eff384
Add plumbing to calculate backend specific file hash for upload
...
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
Ryan Hitchman
77bf148460
backup: add --dry-run/-n flag to show what would happen.
...
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.
Fixes #1542
Example usage:
$ restic -vv --dry-run . | grep add
new /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified /cmd/restic/global.go, saved in 0.000s (0 B added)
new /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
Alexander Neumann
aef3658a5f
Address review comments
2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9
errcheck: Add error check for MergeFinalIndexes()
2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee
errcheck: Add error checks
...
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Alexander Neumann
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading
...
Unify index loading
2021-01-28 13:50:42 +01:00
Michael Eischer
e2b0072441
check: add progress bar to the tree structure check
2021-01-28 11:10:50 +01:00
Michael Eischer
de99207046
repository: tweak comment for packs method
2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e
Count packs directly in RebuildIndexFiles
2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13
repository: Add test for ForAllIndexes
2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37
repository: move otherwise unused LoadIndex to tests
2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4
repository: deduplicate index loading implementation
2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972
check: extract parallel index loading
2020-12-22 22:36:18 +01:00
Alexander Neumann
36c5d39c2c
Fix issues reported by semgrep
2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile
...
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Weiss
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6
Harmonize naming
2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
Alexander Weiss
ce5d630681
Add MasterIndex.PackSize()
2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d
Return hdrSize in ListPack
2020-11-21 22:13:54 +01:00
Michael Eischer
1f43cac12d
check: Only track data blobs when unused blobs should be reported
...
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00