mirror
/
borg
mirror of https://github.com/borgbackup/borg.git
1
0
Fork 0
Code Issues Releases Wiki Activity
8,438 Commits 9 Branches 132 Tags 68 MiB
Commit Graph

236 Commits

Author SHA1 Message Date
TW d0d21e7928
Merge pull request #7197 from ThomasWaldmann/valid-archive-names 
text attributes: validate more strictly, fixes #2290
 2022-12-15 19:24:17 +01:00
Thomas Waldmann 61e5115d1d
fix fso.stats updating 
fso is None in dry_run mode.
 2022-12-15 18:10:36 +01:00
Thomas Waldmann 19e25043b5
create: --list --dry-run output for directories, fixes #7209 
Without the status being set no output was generated in
dry-run mode, confusing users about whether borg would back
up directories (in non-dry-run mode).

- == item not backed up just because of dry-run mode
x == item excluded
 2022-12-15 17:37:51 +01:00
Thomas Waldmann b9dc2be661
remove support for bpath placeholder 
this needs to get done differently later, maybe base64 for json output.
 2022-12-12 19:39:52 +01:00
Thomas Waldmann 1e2741ad3d
remove support for barchive placeholder 
not needed any more as the archive name is now validated pure text.
 2022-12-12 19:10:58 +01:00
Thomas Waldmann 4b54b5565e
use comment validator for cli args and borg transfer 2022-12-12 18:48:01 +01:00
Thomas Waldmann ebff12ad41
transfer: validate archive names before transfer 2022-12-12 17:06:13 +01:00
Thomas Waldmann 1f859c9f17
refactor: get archive timestamps via archive_ts_now() 2022-12-04 10:55:17 +01:00
Thomas Waldmann 17cc72ec28 fix args.paths related argparsing, fixes #6994 
argparse: the default action is "store" and that overwrote an already
existing list in args.paths (e.g. from --pattern="R someroot") when it
started to process the positional PATH args.

with "extend" it now extends the existing args.paths with the list of
positional PATH arguments (which can be 0..N elements long, nargs="*").

note: "extend" is new since python 3.8, thus this can only be backported
to 1.2-maint, but not to 1.1-maint.
 2022-11-04 23:29:58 +01:00
nyuszika7h 54ec166219
tar_filter: recognize .tar.zst as zstd (#7093) 
tar_filter: recognize .tar.zst as zstd
 2022-10-22 14:46:12 +02:00
Franco Ayala 2ed7f317d3
Adding performance statistics to borg create (#6991) 
- file status A/M/E counters
- chunking time
- hashing time
- rx_bytes / tx_bytes

Note: the sleep() in the test is needed due to timestamp granularity on linux being much more coarse than expected (uses the system timer, 100Hz or 250Hz).
 2022-10-19 21:40:02 +02:00
TW 349b4905ce
Merge pull request #7074 from ThomasWaldmann/fix-nfiles-master 
archive.save(): always use metadata from stats, fixes #7072
 2022-10-08 19:59:09 +02:00
Thomas Waldmann bf3f9b5e5b benchmark: fixed TypeError in compression benchmarks 2022-10-05 22:11:44 +02:00
Thomas Waldmann 2e83d18d35 archive.save(): always use metadata from stats, fixes #7072 
e.g. nfiles, size, etc.

fixes:
- checkpoint archives did not have this metadata yet
- borg import-tar did not have this metadata yet
 2022-10-03 23:25:02 +02:00
Thomas Waldmann 5d8d21df4d rcompress docs / examples / markup fixes 2022-10-02 15:50:41 +02:00
TW 58c375678a
Merge pull request #7055 from ThomasWaldmann/borg-lock-wait-env-var 
BORG_LOCK_WAIT=n env var to set default for --lock-wait option, fixes #5279
 2022-09-27 19:41:14 +02:00
Thomas Waldmann e57ff29f22 BORG_LOCK_WAIT=n env var to set default for --lock-wait option, fixes #5279 2022-09-27 17:09:19 +02:00
Thomas Waldmann 9455c1e278 rcompress: do a repo-wide (re)compression 
reads all chunks in on-disk order and recompresses them if they are not already using
the desired compression type and level (and obfuscation level).

supports SIGINT/ctrl-c and --checkpoint-interval (default: 1800s).

this is a borg command that compacts when committing (without this, it would have
a huge space usage). it commits/compacts every checkpoint interval or when
pressing ctrl-c / receiving SIGINT.
 2022-09-27 16:50:19 +02:00
TW 78b1301b98
Merge pull request #7028 from ThomasWaldmann/match-archives 
implement pattern support for --match-archives, fixes #6504
 2022-09-27 11:38:27 +02:00
Thomas Waldmann c4e54ca44e repository.scan: use same end_segment within same scan 
achieved by putting it into the state that is now used instead of the marker.
 2022-09-19 21:14:25 +02:00
Thomas Waldmann 49a4884cfe repository.scan: do not use chunkid as marker, but (segment, offset) 
when using .scan(limit, marker), we used to use the last chunkid from
the previously returned scan result to remember how far we got and
from where we need to continue.

as this approach used the repo index to look up the respective segment/offset,
it was problematic if the code using scan was re-writing the chunk to
a new segment/offset, updating the repo index (e.g. when recompressing a chunk)
and basically destroying the memory about from where we need to continue
scanning.

thus, directly returning (segment, offset) as marker is easier and solves this issue.
 2022-09-19 12:03:13 +02:00
Thomas Waldmann 4493d396e6 implement pattern support for --match-archives, fixes #6504 
also:
- rename --glob-archives option to --match-archives (short: -a, unchanged)
- globbing patterns now need sh: prefix
- regex patterns need re: prefix
- "identical" match "patterns" use an id: prefix
- new default style is id: pattern (--glob-archives used sh: glob pattern)
- source code: glob -> match, GLOB -> PATTERN
 2022-09-16 15:10:13 +02:00
Thomas Waldmann d3a2d831b7 refactor replace_placeholders, fixes #6966 
fix replacing placeholders in archive name, --comment and --glob-archives values (even if overridden by other options like
--timestamp).

add test.
 2022-09-09 23:28:34 +02:00
Thomas Waldmann f647fa9f08 transfer: check if correct upgrader class is used for v1 repos 2022-09-09 21:20:35 +02:00
Thomas Waldmann cf333cef91 upgrader fixes for new api 2022-09-07 20:27:03 +02:00
Thomas Waldmann 1e156ca02b fix upgrader 2022-09-07 19:23:11 +02:00
Thomas Waldmann fa986a9f19 repoobj: add a layer to format/parse repo objects 
borg < 2:

obj = encrypted(compressed(data))

borg 2:

obj = enc_meta_len32 + encrypted(msgpacked(meta)) + encrypted(compressed(data))

handle compr / decompr in repoobj

move the assert_id call from decrypt to RepoObj.parse

also:
- for AEADKeyBase, add a dummy assert_id (not needed here)
- only test assert_id for other if not AEADKeyBase instance
- remove test_getting_wrong_chunk. assert_id is called elsewhere
  and is not needed any more anyway with the new AEAD crypto.
- only give manifest (includes key, repo, repo_objs)
- only return manifest from Manifest.load (includes key, repo, repo_objs)
 2022-09-04 00:49:38 +02:00
Thomas Waldmann 732ecb29fe update prune docs, -P is gone, -a replaces it 2022-08-14 22:01:14 +02:00
Thomas Waldmann 66e74e0471 archiver modules: rename to *_cmd[s] 
this was already required for list_cmd (due to "list"
being a builtin), now renamed all else also.
 2022-08-13 22:59:48 +02:00
Thomas Waldmann 403ff07dde move nanorst module to borg.helpers 2022-08-13 22:10:13 +02:00
Thomas Waldmann 9beaced33c move manifest module from helpers to borg.manifest 2022-08-13 21:55:12 +02:00
Thomas Waldmann ded384929d move tar related code to borg.archiver.tar 2022-08-13 21:31:41 +02:00
Thomas Waldmann 22804f05f9 move prune related code to borg.archiver.prune 2022-08-13 20:59:18 +02:00
Thomas Waldmann 4ab07b6acb create/recreate/import-tar --timestamp: default to local timezone 
if you want to give UTC, just append: +00:00
 2022-08-13 19:17:10 +02:00
Thomas Waldmann ade08ce842 use timezones 
- timezone aware timestamps
- str representation with +HHMM or +HH:MM
- get rid of to_locatime
- fix with_timestamp
- have archive start/end time always in local time with tz or as given
- idea: do not lose tz information

then we know when a backup was made and even from
which timezone it was made. if we want to compute
utc, we can do that using these infos.

this makes a quite nice archives list, with timestamps
as expected (in local time with timezone info).

at some places we just enforce utc, like for the
repo manifest timestamp or for the transaction log,
these are usually not looked at by the user.
 2022-08-13 18:31:22 +02:00
Thomas Waldmann c51a0626de remove remainders of borg key change-algorithm 
most was already removed previously, just this was overlooked.
 2022-08-11 17:42:27 +02:00
Thomas Waldmann a18a9df364 run "black" 2022-08-11 06:34:09 +02:00
Thomas Waldmann 252b145833 remove now unused argument() 2022-08-11 06:30:16 +02:00
Thomas Waldmann 8562940b0d with_repository: fake mechanism not needed 
was only used by borg create --dry-run, but that
internally does not use the repo anyway.
 2022-08-11 06:29:34 +02:00
Thomas Waldmann 8bebf65bc2 with_repository: invert_fake arg is not used 2022-08-11 06:19:23 +02:00
Thomas Waldmann 73d901a22e with_repository: "exclusive" param is always bool 2022-08-11 06:12:07 +02:00
Thomas Waldmann e54f109afa --nobsdflags does not exist any more 
it was replaced by --noflags.
 2022-08-11 06:00:12 +02:00
Thomas Waldmann 4d570497be ctrl-c must not kill other subprocesses, fixes #6912 
There are some other places with subprocesses:

- borg create --content-from-command
- borg create --paths-from-command
- (de)compression filter process of import-tar / export-tar
 2022-08-06 23:37:38 +02:00
Thomas Waldmann fb74fdb710 massively increase per archive metadata stream size limit, fixes #1473 
implemented by introducing one level of indirection, the limit is now
very high, so it is not practically relevant any more.

we always use the indirection (storing the metadata stream chunk ids list not
directly into the archive item, but into some repo objects referenced by the new
ArchiveItem.item_ptrs list).

thus, the code behaves the same for all archive sizes.
 2022-08-06 19:01:41 +02:00
Thomas Waldmann 4ec17d969c rename --copy-ae-key into --copy-crypt-key 2022-08-04 10:32:24 +02:00
Thomas Waldmann 17e75f3eea docs: transfer: talk of "related repo" 2022-08-03 18:33:02 +02:00
Thomas Waldmann bcd7ab3dec docs: rcreate: explain "related repo" 2022-08-03 18:20:10 +02:00
Thomas Waldmann 3ee69bc7ba Key: crypt_key instead of enc_key + enc_hmac_key, fixes #6611 2022-08-03 12:04:23 +02:00
TW 7fde480719
Merge pull request #6921 from ThomasWaldmann/new-ae-key 
rcreate --copy-ae-key: copy AE key from key of other repo, fixes #6710
 2022-08-01 15:30:58 +02:00
TW 1744c92967
Merge pull request #6919 from ThomasWaldmann/fix-delete-master 
delete: remove unused option --keep-security-info
 2022-08-01 15:12:06 +02:00
Thomas Waldmann 9878956140 rcreate --copy-ae-key: copy AE key from key of other repo, fixes #6710 
default: create new, random authenticated encryption key.
 2022-08-01 14:57:30 +02:00
Thomas Waldmann a8dda5cb0c transfer: check whether ID hash method and chunker secret are same, fixes #6893 
also: add PlaintextKey and AuthenticatedKey support to uses_same_id_hash function.
 2022-08-01 13:27:47 +02:00
Thomas Waldmann 2c2afe4123 delete: remove unused option --keep-security-info 
this option is only useful for deleting repos.
 2022-08-01 11:41:39 +02:00
Thomas Waldmann 0deb4352c1 prune/delete --checkpoint-interval=1800 and ctrl-c/SIGINT support, fixes #6284 
manifest, repo and cache are committed every checkpoint interval.

also, when ctrl-c is pressed, finish deleting the current archive, commit and then terminate.
 2022-07-30 17:24:40 +02:00
Thomas Waldmann ad7c7159b2 Merge branch 'master' into borg2 2022-07-27 13:49:50 +02:00
Thomas Waldmann 7bc7f01342 remove remainders of attic legacy 
we expect that everybody has upgraded to borg
using borg 1.2.x or older, thus we do not need
to care about attic repos any more in borg2.
 2022-07-13 16:55:29 +02:00
Thomas Waldmann ea6b373f85 move common options definition to archiver.common 2022-07-09 15:13:13 +02:00
Thomas Waldmann b2fb334af0 sort build_parser calls / mixins 2022-07-09 15:13:13 +02:00
Thomas Waldmann 70b54a696f move create command to archiver.create 2022-07-09 15:13:13 +02:00
Thomas Waldmann c871dbd8a5 move recreate command to archiver.recreate 2022-07-09 15:13:13 +02:00
Thomas Waldmann ae6ef77495 move extract command to archiver.extract 2022-07-09 15:13:13 +02:00
Thomas Waldmann e05f7971da move build_filter/build_matcher to archiver.common 2022-07-09 15:13:13 +02:00
Thomas Waldmann e57f3bb424 move info command to archiver.info 2022-07-09 15:13:13 +02:00
Thomas Waldmann 4d5a89ebc3 move rinfo command to archiver.rinfo 2022-07-09 15:13:13 +02:00
Thomas Waldmann add73745c6 move rlist command to archiver.rlist 2022-07-09 15:13:13 +02:00
Thomas Waldmann b959ff913a move list command to archiver.list_cmd 
note: can't be named "archiver.list" due to collision with builtin type "list".
 2022-07-09 15:13:13 +02:00
Thomas Waldmann 382b37ef60 move list command to archiver.list 2022-07-09 15:13:13 +02:00
Thomas Waldmann 350a8fe252 move delete command to archiver.delete 2022-07-09 15:13:13 +02:00
Thomas Waldmann 06fe8d3e64 move rename command to archiver.rename 2022-07-09 15:13:13 +02:00
Thomas Waldmann a31332ddb5 move rdelete command to archiver.rdelete 2022-07-09 15:13:13 +02:00
Thomas Waldmann 94aec46a01 move rcreate command to archiver.rcreate 2022-07-09 15:13:12 +02:00
Thomas Waldmann ff411b5d98 move serve command to archiver.serve 2022-07-09 15:13:12 +02:00
Thomas Waldmann 68772484ce move mount/umount/borgfs command to archiver.mount 2022-07-09 15:13:12 +02:00
Thomas Waldmann 2ff2927aa3 move check command to archiver.check 2022-07-09 15:13:12 +02:00
Thomas Waldmann df57d33b5f move compact command to archiver.compact 2022-07-09 15:13:12 +02:00
Thomas Waldmann 9fb224db6b move diff command to archiver.diff 2022-07-09 15:13:12 +02:00
Thomas Waldmann ea03562b11 move prune command to archiver.prune 2022-07-09 15:13:12 +02:00
Thomas Waldmann 94e6477e51 move config command to archiver.config 2022-07-09 15:13:12 +02:00
Thomas Waldmann 8d9e47e374 move transfer command to archiver.transfer 2022-07-09 15:13:12 +02:00
Thomas Waldmann 6d60976c51 move help commands to archiver.help 2022-07-09 15:13:12 +02:00
Thomas Waldmann f906d9d246 move locking commands to archiver.locks 2022-07-09 15:13:12 +02:00
Thomas Waldmann af2ee7aeb1 move key commands to archiver.keys 2022-07-09 15:13:12 +02:00
Thomas Waldmann 5ea9fb73db move benchmark commands to archiver.benchmarks 2022-07-09 15:13:12 +02:00
Thomas Waldmann 27b63e7c48 move tar commands to archiver.tar 2022-07-09 15:13:12 +02:00
Thomas Waldmann 890d5950e7 move debug commands to archiver.debug 2022-07-09 15:13:12 +02:00
Thomas Waldmann 227ecb5b8e transform archiver module into a package 2022-07-07 22:40:27 +02:00