Commit Graph

538 Commits

Author SHA1 Message Date
Alexander Neumann a350625554
Merge pull request #3524 from MichaelEischer/atomic-sftp
sftp: Implement atomic uploads
2022-03-21 11:08:22 +01:00
Alexander Neumann 32e61f2620
Update changelog/unreleased/issue-1106
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-03-21 11:04:04 +01:00
Alexander Neumann 8388f66c4c
Merge pull request #3668 from greatroar/symlink-size
Report symlink sizes from FUSE mount
2022-03-21 11:02:32 +01:00
Alexander Neumann 29a5778626 Improve wording 2022-03-20 13:46:16 +01:00
Michael Eischer 53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Vincent Bernat 2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
greatroar fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
Michael Eischer 7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
MichaelEischer ad4f4dbc7a
Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar 63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
gum3ng dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
Michael Eischer 5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
MichaelEischer cc90f2ba6b
Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
MichaelEischer d8f58fb7bf
Merge pull request #3592 from vgt/jsondiff
Add json output for diff command
2022-02-07 21:33:30 +01:00
Michael Eischer aaa7f94139 Add changelog for O_NOATIME 2022-02-06 15:00:37 +01:00
Michael Eischer 5f34ad523f diff: fix test failure and add remark on quiet to changelog 2022-02-05 23:02:07 +01:00
Aneesh Agrawal 058dfc20da Avoid choosing parent snapshot newer than time of current snapshot
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.

However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.

Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
David Vogt 77c850148a Add json output for diff command 2022-01-23 23:22:26 +01:00
MichaelEischer df89aa0087
Merge pull request #3623 from invine/issue-3464
Skip lock file creation on forget with --no-lock and --dry-run
2022-01-23 18:12:34 +01:00
Pavel Frolov 792523b28b [issue 3464] skip lock creation in case of dry-run 2022-01-23 06:44:41 +03:00
rawtaz 7f6fc78f95
Merge pull request #3544 from restic/fix-b2-delete-retry
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
rawtaz abfbacf3d3
Merge pull request #3591 from MichaelEischer/prune-fix-max-repack
prune: Handle --max-repack-size=0 as expected
2022-01-13 03:53:20 +01:00
MichaelEischer 2c3e5d943d
Merge pull request #3593 from DarkKirb/parallelize-restic-copy
Parallelize blob upload/download for restic copy
2021-12-29 22:31:54 +01:00
Charlotte 🦝 Delenk e2bb384a60
Parallelize blob upload/download for restic copy
Currently restic copy will copy each blob from every snapshot serially,
which has performance implications on high-latency backends such as b2.

This commit introduces 8x parallelism for blob downloads/uploads which
can improve restic copy operations up to 8x for repositories with many
small blobs on b2.

This commit also addresses the TODO comment in the copyTree function.

Related work:

A more thorough improvement of the restic copy performance can be found
in PR #3513
2021-12-29 18:59:09 +01:00
MichaelEischer e5985e0d63
Merge pull request #3602 from cqjjjzr/fix-rclone-sigint
fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever (#3601)
2021-12-29 16:56:37 +01:00
Charlie Jiang a5b0e0bef4 fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
Michael Eischer 64e733f3d6 tweak copy --no-lock changelog 2021-12-27 18:22:25 +01:00
Michael Eischer 0cfdb82ea4 prune: Handle --max-repack-size=0 as expected
Previously the flag was ignored and `--max-repack-size=1` had to be
used.
2021-12-27 15:48:56 +01:00
Kyle Brennan d5ed5da85c
update changelog for copy --no-lock 2021-12-03 12:16:40 -08:00
Manuel González 212b2f651f
Add file mode in symbolic notation to `ls --json`
This aligns `restic ls --json` with `restic find --json`, utilizing the same
naming.
2021-11-16 19:45:50 +02:00
Michael Eischer 2ea998f70e Add PR link to changelog 2021-11-13 22:25:54 +01:00
Garry McNulty e8fa3855e7 Update blazer
If a request fails with "x509: certificate signed by unknown authority",
the B2 backend now returns the error without retrying the request.

Closes #3556
Closes #2355
2021-11-13 22:25:54 +01:00
MichaelEischer f1cfb97237
Merge pull request #3514 from phcreery/rclone_timeout
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer cb81ee9396
Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
phcreery 43d173b042 rclone: add timeout option and documentation 2021-11-07 17:49:21 +01:00
kitone 0425a30420 add changelog issue-3382 2021-11-04 15:13:48 +01:00
Gautam Menghani 836fbb9133 [#issue 3490] Support for specifying file size in read-data-subset 2021-11-02 15:25:46 +05:30
Michael Eischer 257740b0cc b2: Successful delete if file does not exist
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
Garry McNulty 708d7a2574 s3: Add warning if key ID or secret is empty
Also add debug message if no credential types are available.

Closes #2388
2021-10-06 23:13:40 +01:00
MichaelEischer a5e103a212
Merge pull request #3522 from greatroar/dump-lru
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
Uli Martens 1ebcb1d097 Add changelog entry for PR #3508 2021-09-24 15:45:09 +02:00
Sam Lucidi 897d8e662c Add --insecure-tls flag to disable SSL cert verification
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
MichaelEischer 1827b16ade
Merge pull request #3519 from greatroar/maphash
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar 8b758c78a3 Require Go 1.14 to build 2021-09-19 16:18:19 +02:00
MichaelEischer 58efe21eca
Merge pull request #3264 from amozoss/upstream-master
Refactor backup progress
2021-09-19 14:54:42 +02:00
Michael Eischer e62d4f622f Reword changelog 2021-09-19 14:41:07 +02:00
greatroar bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
Michael Eischer 503d4c3e2f Add changelog 2021-09-12 16:00:49 +02:00
Alexander Neumann bf9c8771a4
Merge pull request #3482 from MichaelEischer/changelog-3429
Add changelog for #3246 and #3429
2021-09-08 09:15:40 +02:00
Michael Eischer 5e84f38f31 Add changelog for #3246 2021-09-07 21:18:11 +02:00
Alexander Neumann 8fe122d675
Merge pull request #3488 from MichaelEischer/rebuild-broken-index
Fix `rebuild-index` for damaged index
2021-09-07 17:00:23 +02:00
Alexander Weiss bce87922c0 Improve recover
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
2021-09-03 21:36:57 +02:00
Michael Eischer 194ed19557 Add changelog 2021-08-22 18:29:58 +02:00
Lorenz Bausch 4d56b34096
Fix typo in changelog 2021-08-21 12:16:04 +02:00
Michael Eischer 1fab5892b5 Add changelog for #3429 2021-08-16 17:42:16 +02:00
greatroar ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
Michael Eischer 65908647e3 Add changelog 2021-08-04 21:51:53 +02:00
Michael Eischer 702cff636f Add use case to changelog 2021-08-04 21:19:29 +02:00
Alexander Weiss 780e11b7e2 Adapt changelog 2021-08-04 21:19:29 +02:00
erin 4126435663 resolve rawtaz's review comments
make majority of suggestions from review by @rawtaz verbatim, with one clarification on my part in changelog
2021-08-04 21:19:29 +02:00
Ryan Hitchman 77bf148460 backup: add --dry-run/-n flag to show what would happen.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.

Fixes #1542

Example usage:

$ restic -vv --dry-run . | grep add
new       /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified  /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified  /cmd/restic/global.go, saved in 0.000s (0 B added)
new       /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new       /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified  /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified  /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified  /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified  /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified  /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
Alexander Neumann b19cd8c50f Prepare changelog for 0.12.1 2021-08-03 11:45:19 +02:00
Leo R. Lundgren cb844e7136 Polish changelog entries 2021-08-03 00:01:09 +02:00
Alexander Neumann b7fe1fe6b4 Reword changelog entries 2021-08-03 00:01:09 +02:00
David le Blanc 326fefcd80 Allow --tag and --keep-tag to match untagged snapshots 2021-08-02 23:06:20 +02:00
MichaelEischer 5571c3f7fd
Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
Alexander Neumann 77551597b2
Merge pull request #3416 from torfason/keep-hourly-within
Keep hourly within
2021-07-30 10:36:41 +02:00
Alexander Neumann 92f293cd0b
Merge pull request #3427 from MichaelEischer/find-packs-from-index
find: List missing pack files based on the index
2021-07-30 10:33:02 +02:00
Magnus Thor Torfason 74ebc650ab forget: Add --keep-within-hourly (and friends)
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.

This adds the following flags/parameters to restic forget:
  --keep-within-hourly duration
  --keep-within-daily duration
  --keep-within-weekly duration
  --keep-within-monthly duration
  --keep-within-yearly duration

Includes following changes:
  - Add tests for --keep-within-hourly (and friends)
  - Add documentation for --keep-within-hourly (and friends)
  - Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
Michael Eischer 7d28006e2e Add changelog 2021-07-10 22:39:01 +02:00
Michael Eischer 495831d53c add release binaries for linux/s390x 2021-07-10 00:10:24 +02:00
Michael Eischer a81f34ae47 Add changelog 2021-07-06 21:04:34 +02:00
Michael Eischer 43b82d69b4 Add changelog 2021-06-29 21:27:00 +02:00
Michael Eischer 409306db2b Add changelog for #3426 2021-06-20 14:46:37 +02:00
MichaelEischer 5dad45f005
Merge pull request #3312 from soraxas/master
Bump cobra and add completions for fish
2021-05-23 13:58:27 +02:00
greatroar d7322a5f36 restic ls --json: print "size":0 for empty files 2021-05-21 21:06:00 +02:00
Tin Lai 9cc1ecdd45
bump cobra and add completions for fish
Signed-off-by: Tin Lai <oscar@tinyiu.com>
2021-05-21 13:47:52 +10:00
Alexander Neumann 226cd8d4d1
Merge pull request #3310 from MichaelEischer/copy-unstable-tree
`copy` raw tree blobs
2021-05-18 09:36:51 +02:00
Alexander Neumann 4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Alexander Neumann d7b5061aa5
Merge pull request #3394 from MichaelEischer/apple-m1-build
Build darwin/arm64 binaries for Apple M1
2021-05-17 09:56:44 +02:00
Michael Eischer 55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
Michael Eischer af6f6fba15 Build darwin/arm64 binaries for Apple M1 2021-05-14 23:04:45 +02:00
MichaelEischer 7349c6d338
Merge pull request #3167 from renard/limit-snapshots-list
Add option to limit snapshots list
2021-05-13 20:26:33 +02:00
Sébastien Gross 2a92b68e65 cmd/snapshots: Add option to limit snapshots list
This patch adds a `--latest` option to limit snapshots list to the n
last snapshots. It is very similar to the `--last` one but does not
limit to one entry. It also deprecates the `--last` flag usage in
favor of `--latest 1`

Output example:

    $ restic snapshots --latest 2
    repository 0d3eb989 opened successfully, password is correct
    ID        Time                 Host        Tags        Paths
    ------------------------------------------------------------
    5a33bdcc  2020-12-14 12:30:00  local                   /home
    73887d8e  2020-12-15 12:30:00  local                   /home
    ------------------------------------------------------------
    2 snapshots

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2021-05-13 20:18:23 +02:00
MichaelEischer 64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
Michael Eischer f65e1211d9 Add changelog entry 2021-04-11 20:02:09 +02:00
Michael Eischer 88731d8c28 Add changelog 2021-03-27 17:08:35 +01:00
greatroar dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
Hendrik Luup 5592c17e4a Display 'created new cache in ' message only if output is terminal 2021-03-22 11:31:08 +02:00
MichaelEischer c9b4fadd91
Merge pull request #3294 from Achilleshiel/fix-copy-repofile
Add RepositoryFile2 Option for secondary repository
2021-03-08 22:50:31 +01:00
Wouter Horlings 9ccdba9df6 Add repositoryFile2 option
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.

This also fixes the conflict with the `--repository-file` and `--repo2`
flag.

Tests are added for the initSecondaryGlobalOpts function.

This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.

With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.

In total the tests checks wether the following are passed correct:
   - Password
   - PasswordFile
   - Repo
   - RepositoryFile

The following situation must return an error to pass the test:
   - no Repo or RepositoryFile defined
   - Repo and RepositoryFile defined both
2021-03-08 22:41:13 +01:00
Michael Eischer f9c581f219 Tweak changelog 2021-03-08 22:11:34 +01:00
Juergen Hoetzel 18fccb5995 backup: In case of error also Wait() for terminal goroutine
This prevents a race condition where the final summary message can get lost.
2021-03-08 22:11:34 +01:00
MichaelEischer 814a399e4c
Merge pull request #3283 from dennypage/master
Treat an empty password as a fatal error for repository init.
2021-02-28 00:49:21 +01:00
Michael Eischer 13730e3844 Add link to pr to changelog 2021-02-28 00:40:52 +01:00
Denny Page 0f41e99ea7 Treat an empty password as a fatal error for repository init. 2021-02-28 00:40:00 +01:00
Michael Eischer 9e852af5be check: Fix crash of --read-data-subset=x% on empty repository
Rounding up to at least one pack file should only be done if at least a
pack file exists.
2021-02-27 16:05:36 +01:00
Michael Eischer a293fd9aef local: Ignore files in intermediate folders
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
Michael Eischer 65bd2a9a49 backup: Correctly handle --quiet flag
The quiet flag changes the backup output to assume a non-interactive
terminal. However, the output progress interval was not set to 0 by
default.
2021-02-15 22:14:58 +01:00