56836364a4
backend: pass context into every backend constructor
2023-06-17 15:15:57 +02:00
4df77e9f26
Merge pull request #4314 from MichaelEischer/restore-json
...
Add JSON support for the restore progress bar
2023-06-16 23:53:17 +02:00
191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc
...
Tune Go garbage collector
2023-06-16 23:24:39 +02:00
dd1ef13c1c
Merge pull request #4352 from MichaelEischer/pointerless-index
...
index: optimize go GC performance
2023-06-16 23:23:14 +02:00
b2ed42cec4
index: add basic hat test
2023-06-16 23:12:30 +02:00
07d1f8047e
restore: More descriptive field names for the JSON output
2023-06-09 12:07:55 +02:00
b2b0760eb0
restore: add test for json output
2023-06-09 12:03:33 +02:00
d54176ce5d
restore: add basic json progress
2023-06-09 12:03:33 +02:00
a9aff885d6
restore: reorganize progress bar code
...
The structure is now much more similar to that of the backup command.
2023-06-09 12:03:33 +02:00
c37d587f81
Use PreallocateFile in local backend
2023-06-09 11:49:06 +03:00
ffc6b3d887
Move PreallocateFile to fs package
2023-06-09 11:49:06 +03:00
0b908bb1fb
Address review comments
2023-06-08 20:24:21 +02:00
0372c7ef04
Improve the ETA displayed during backup
...
The ETA restic displays was based on a rate computed across the entire
backup operation. Often restic can progress at uneven rates. In the worst
case, restic progresses over most of the backup at a very high rate and
then finds new data to back up. The displayed ETA is then unrealistic and
never adapts.
Restic now estimates the transfer rate based on a sliding window, with the
goal of adapting to observed changes in rate. To avoid wild changes in the
estimate, several heuristics are used to keep the sliding window wide
enough to be relatively stable.
2023-06-08 20:05:56 +02:00
55c21846b1
Revert "index: remove redundant storage of indexmap size"
...
This reverts commit f1c388c623
2023-06-08 18:08:46 +02:00
609367195a
backend: Fix test compilation with Go 1.18 and 1.19
2023-06-07 22:31:15 +02:00
18eb1d3ab0
backend: test cleanup
2023-06-07 22:31:15 +02:00
32a6b66267
backend: add standardized Config.ApplyEnvironment
...
This removes the backend specific special cases while parsing the
configuration in `global.go`.
2023-06-07 22:31:15 +02:00
f903db492c
backend: let ParseConfig return a Config pointer
...
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.
An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
2023-06-07 22:31:15 +02:00
25a0be7f26
backend: fix linter warnings
2023-06-07 22:31:15 +02:00
a27b7f1370
backend: use generic instead of any type for test suite
2023-06-07 22:31:14 +02:00
fa361dbfbd
backend: use generic implementation for ParseConfig tests
2023-06-07 22:30:35 +02:00
5260d38980
backend: let ParseConfig return concrete type
2023-06-07 22:30:35 +02:00
2f7b4ceae1
backend: Move environment based configuration into backend
2023-06-07 22:30:35 +02:00
17446da5fd
Merge pull request #4343 from greatroar/cache
...
cache: Restructure New to remove redundant operations
2023-06-02 22:41:17 +02:00
eef0ee7a85
repository: trigger GC after loading the index
...
Loading the index requires some scratch space, thus make sure that this
memory does not factor into the targeted gc memory usage limit.
2023-06-02 21:56:14 +02:00
ac1dfc99bb
index: fix blocklist size
2023-06-02 19:39:12 +02:00
9a7056a479
index: implement indexmap.grow() without random access
2023-05-30 20:13:33 +02:00
fc05e35a08
index: let indexmap.Each iterate in allocation order
...
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.
As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
2023-05-30 20:12:36 +02:00
f1c388c623
index: remove redundant storage of indexmap size
2023-05-30 20:11:53 +02:00
12141afbad
index: Allow inlining of HAT
2023-05-30 20:11:14 +02:00
fed33295c3
index: store indexEntries in hashed array tree
...
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
2023-05-29 00:24:15 +02:00
b217f38ee7
index: Remove pointers from within indexentrys
...
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
2023-05-29 00:24:15 +02:00
0c1240360d
index: add garbage collection benchmark
...
Allocates an index and repeatedly triggers the GC.
2023-05-29 00:23:04 +02:00
ffca602315
repository: Fix panic in benchmarkLoadIndex
2023-05-28 23:55:47 +02:00
da419be43c
cache: Restructure New to remove redundant operations
...
New and its helpers used to create the cache directories several times
over. They now only do so once. The added test ensures that the cache is
produced in a consistent state when parts are deleted.
2023-05-27 10:32:08 +02:00
3e287afdbf
Merge pull request #4226 from Sohalt/gcs-region
...
gs: support other regions than us
2023-05-26 20:28:52 +02:00
ed5b2c2c9b
gs: support other regions than us
2023-05-26 19:54:42 +02:00
c3212ab6a6
test: use standard logging methods from testing for the test helpers
...
Use the logging methods from testing.TB to make use of tb.Helper(). This
allows the tests to log the filename and line number in which the test
helper was called. Previously the test helper was logged which is rarely
useful.
2023-05-26 19:34:57 +02:00
658aa4c0f7
Merge pull request #4339 from greatroar/ioctl
...
backend, termstatus: fix ioctl calls for 64-bit big-endian platforms
2023-05-26 19:34:14 +02:00
51dc80be5b
backend: Use IoctlSetPointerInt for tcsetpgrp
...
This function casts its argument to int32 before passing it to the
system call, so that big-endian CPUs read the lower rather than the
upper 32 bits of the pid.
This also gets rid of the last import of "unsafe" in the Unix build.
I changed syscall to x/sys/unix while I was at it, to remove one more
import line. The constants and types there are aliases for their syscall
counterparts.
2023-05-25 17:31:51 +02:00
ddbc0c1b37
termstatus: Fix IsProcessBackground for 64-bit big-endian Linux
...
Fixes #4223 . Verified with QEMU on linux/amd64.
2023-05-25 17:20:42 +02:00
16ba237d8b
azure: Fix missing rate limiting
2023-05-18 21:17:53 +02:00
22c9276719
rest: remove unused parameter
2023-05-18 21:17:53 +02:00
0058745881
test: use parameter instead of hardcoded constant
2023-05-18 21:17:53 +02:00
a719d10e22
termstatus: remove unused parameter from helpers
2023-05-18 21:17:53 +02:00
b0a01ae68a
ui/backup: remove unused parameter from ProgressPrinter interface
2023-05-18 21:17:53 +02:00
472bf5184f
Replace lots of unused parameters with _
...
The parameters are required by the implemented function signature or interface.
2023-05-18 21:17:53 +02:00
d1a5ec7839
Rename unused testing parameter to _
...
The parameter is an additional marker that the test helper must only be
used for tests.
2023-05-18 21:17:53 +02:00
1514593f22
Remove unused context or testing parameters
2023-05-18 21:17:53 +02:00
5e4e268bdc
Use _ as parameter name for unused Context
...
The context is required by the implemented interface.
2023-05-18 21:15:45 +02:00
2fa8b96843
Don't shadow builtins
2023-05-18 21:15:45 +02:00
7a01bd3b67
Remove unnecessary else block
2023-05-18 21:15:45 +02:00
319087c056
Remove redundant if ...; err != nil; return pattern
2023-05-18 21:15:45 +02:00
c832a492ac
fuse: use syscall errnos directly to fix deprecations
2023-05-18 21:15:45 +02:00
e01baeabba
Use either test or rtest to refer to internal test helpers
...
A single test file should not use both names.
2023-05-18 21:15:45 +02:00
5773b86d02
repository: Push all usage of errors.Fatal out of the package
...
As the `Fatal` error type only includes a string, it becomes impossible
to inspect the contained error. This is for a example a problem for the
fuse implementation, which must be able to detect context.Canceled
errors.
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-05-18 17:27:41 +02:00
a013014c24
backup: return normal error if --group-by cannot be parsed
...
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2023-05-18 17:23:31 +02:00
f9850b79b5
rest/sftp: Remove redundant fatal from error message
...
This caused restic to exit with error messages like
`Fatal: parsing repository location failed: Fatal: sftp path [...]`
`Fatal: create repository at rest:http://localhost:12345/ failed: Fatal: config file already exists`
2023-05-18 17:22:26 +02:00
2f518b7241
Merge pull request #4304 from MichaelEischer/unlimited-lock-refresh
...
lock: Do not limit backend concurrency for lock files
2023-05-13 21:40:29 +02:00
19ebc1b786
restore: Add basic test for progress bar accounting of hardlinks
2023-05-08 20:51:49 +02:00
23a122a901
restore: count files in the same way as the stats command
2023-05-08 20:51:49 +02:00
e77002f841
restore: correctly count hardlinks in progress bar
...
For hardlinked files, only the first instance of that file increases the
amount of bytes to restore. All later instances only increase the file
count but not the restore size.
2023-05-08 20:51:49 +02:00
d05f6211d1
lock: Do not limit backend concurrency for lock files
...
restic must be able to refresh lock files in time. However, large
uploads over slow connections can cause the lock refresh to be stuck
behind the large uploads and thus time out.
2023-05-08 19:04:46 +02:00
ee3c55ea3d
Merge pull request #2876 from aawsome/new-repair-command
...
Add repair command
2023-05-05 23:22:24 +02:00
db046c0acc
Merge pull request #4318 from MichaelEischer/status-output-truncation
...
Fix status output truncation
2023-05-05 23:09:59 +02:00
3e6a26e2e9
Merge pull request #4317 from MichaelEischer/debug-test-mount-failures
...
mount: enable debug logging for the flaky TestMount test
2023-05-05 23:08:03 +02:00
65c5e511a1
ui/termstatus: Add test for Terminal.SetStatus
2023-05-05 22:51:41 +02:00
6d10c655a0
termstatus: test status line sanitization
2023-05-05 22:51:41 +02:00
bb40e49e75
ui/termstatus: Fix truncation of status output
...
The last line was not truncated as expected
2023-05-05 22:45:52 +02:00
fefe15d7a1
Merge pull request #4316 from MichaelEischer/fix-check-modified-data
...
Fix flaky TestCheckerModifiedData test
2023-05-05 22:38:28 +02:00
1cb11ad8ad
mount: enable debug logging for the flaky TestMount test
...
The test case fails from time to time with an Input/Output error while
trying to access the snapshots directory.
2023-05-01 18:03:17 +02:00
90a663c94f
Merge pull request #4308 from greatroar/tar-header
...
dump: Report filename with tar.ErrFieldTooLong
2023-05-01 17:30:31 +02:00
88a7231217
report snapshot id if loading failed
2023-05-01 17:24:13 +02:00
c0627dc80d
check: Fix flaky TestCheckerModifiedData
...
The test had a 4% chance of not modified the data read from the
repository, in which case the test would fail. Change the data
manipulation to just modified each read operation.
2023-05-01 17:18:19 +02:00
1bd1f3008d
walker: extend TreeRewriter to support snapshot repairing
...
This adds support for caching already rewritten trees, handling of load
errors and disabling the check that the serialization doesn't lead to
data loss.
2023-05-01 15:20:24 +02:00
38dac78180
walker: restructure FilterTree into TreeRewriter
...
The more generic RewriteNode callback replaces the SelectByName and
PrintExclude functions. The main part of this change is a preparation to
allow using the TreeRewriter for the `repair snapshots` command.
2023-05-01 15:20:12 +02:00
bc2399fbd9
walker: recurse into directory based on node type
...
A broken directory might also not have a subtree.
2023-05-01 15:20:00 +02:00
1a9705fc95
walker: Simplify change detection in FilterTree
...
Now the rewritten tree is always serialized which makes sure that we
don't accidentally miss any relevant changes.
2023-05-01 15:19:48 +02:00
118d599d0a
Rename 'rebuild-index' to 'repair index'
...
The old name still works, but is deprecated.
2023-05-01 15:16:44 +02:00
1daf928a77
Merge pull request #4305 from MichaelEischer/stracktrace-for-windows
...
Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
2023-04-30 16:08:58 +02:00
37bab08181
Merge pull request #4302 from MichaelEischer/logging-cleanup
...
Tweak debug logs
2023-04-30 16:08:08 +02:00
6dc2324d2e
Merge pull request #4301 from MichaelEischer/less-restore-memory
...
restore: slightly reduce memory usage while restoring files
2023-04-30 16:07:43 +02:00
b50ff04cf3
dump: Report filename with tar.ErrFieldTooLong
...
Updates #4307 .
2023-04-27 11:45:41 +02:00
306a29980a
Print stacktrace in SIGINT handler if RESTIC_DEBUG_STACKTRACE_SIGINT set
...
The builtin mechanism to capture a stacktrace in Go is to send a SIGQUIT
to the running process. However, this mechanism is not avaiable on
Windows. Thus, tweak the SIGINT handler to dump a stacktrace if the
environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT` is set.
2023-04-23 15:50:40 +02:00
cdb0fb9c06
tweak debug logs
2023-04-23 11:38:06 +02:00
94cbc6392d
restore: slightly reduce memory usage while restoring files
...
The information which target files are contained in a pack file is no
longer necessary after processing a pack.
2023-04-23 11:33:21 +02:00
78a1757e5a
Cancel current command if cache becomes unusable
...
If the cache suddenly disappears, the current command will now fail.
2023-04-23 11:31:15 +02:00
51d823348d
Merge pull request #4286 from MichaelEischer/backend-cleanup-logging
...
Normalize backend logging and connection limiting
2023-04-23 11:24:26 +02:00
831f593b87
backend/sema: Add tests
2023-04-22 13:20:20 +02:00
ebba233a3a
backend/sema: rename constructor to NewBackend
2023-04-22 12:32:57 +02:00
f27750e270
backend/sema: rename type to connectionLimitedBackend
2023-04-21 22:53:09 +02:00
1dd873b706
Merge pull request #4293 from MichaelEischer/fix-groups-by
...
Fix parent snapshot selection for relative paths
2023-04-21 22:47:03 +02:00
913eab3361
Merge pull request #4234 from thndrbrrr/forget-opts-neg1-means-forever-issue-2565
...
restic forget --keep-* options will interpret -1 as "forever"
2023-04-14 23:18:47 +02:00
a06d927dce
Merge pull request #4291 from greatroar/widechars
...
ui/termstatus: Optimize Truncate
2023-04-14 22:48:34 +02:00
fd3ed9e2f4
Merge pull request #4192 from greatroar/quote
...
ui/termstatus: Quote funny filenames
2023-04-14 22:39:09 +02:00
6042df075f
migrations: Fix S3 backend detection
2023-04-14 22:32:16 +02:00
c934c99d41
gs: replace usage of context.Background()
2023-04-14 22:32:15 +02:00
616926d2c1
gs: use IsNotExist to check error
2023-04-14 22:32:15 +02:00
05abc6d6f5
backend: deduplicate implementation of Delete() method
2023-04-14 22:32:15 +02:00
45244fdf68
backend: remove parameter validation tests
...
These parameter validations have been factored out into
SemaphoreBackend.
2023-04-14 22:32:15 +02:00
Michael Eischer