Commit Graph

7217 Commits

Author SHA1 Message Date
Thomas Waldmann c8830cde44 update docs 2022-09-08 22:26:23 +02:00
Thomas Waldmann 5afe94883a segment entry payload, metadata length: 16bit is enough
guess we will not have much more than size, csize, psize, ctype, clevel.
2022-09-08 21:52:19 +02:00
Thomas Waldmann b28d6ee657 recompress: only read metadata to check for ctype/clevel 2022-09-08 20:47:40 +02:00
Thomas Waldmann 06eab6a228 RepositoryCache: cache complete and meta-only chunks separately 2022-09-08 19:38:18 +02:00
Thomas Waldmann 106abbe4d9 new read_data param for repository.get() and .get_many()
True (default): return full chunk (client can decrypt meta and data)

False: return enough so client can decrypt only the meta
2022-09-08 18:31:56 +02:00
Thomas Waldmann 74ffceabf4 simplify: read_data param of io.read() is not used (yet) 2022-09-08 13:17:59 +02:00
Thomas Waldmann b64427c480 simplify: iter_objects always returns (..., size, data)
data might be None (if read_data is False).

also removed the include_data argument, not needed any more.
2022-09-08 12:48:39 +02:00
Thomas Waldmann e827f98c45 transfer: add test
bit hard to test the 1.2 -> 2.0 transfer, but we can at least
test the 2.0 -> 2.0 "NoOp" transfer.
2022-09-07 21:10:54 +02:00
Thomas Waldmann cf333cef91 upgrader fixes for new api 2022-09-07 20:27:03 +02:00
Thomas Waldmann 4c9ed2a6c6 refactor compressors to new api
legacy: add/remove ctype/clevel bytes prefix of compressed data

new: use a separate metadata dict

compressors: use an int as ID, not a len 1 bytestring
2022-09-07 19:23:47 +02:00
Thomas Waldmann 1e156ca02b fix upgrader 2022-09-07 19:23:11 +02:00
Thomas Waldmann b6cbf045ff add a test for borg 1 -> 2 repo objects transformation 2022-09-05 22:17:51 +02:00
Thomas Waldmann 754c583799 decompression dispatching based on meta dict 2022-09-05 22:14:40 +02:00
Thomas Waldmann fa986a9f19 repoobj: add a layer to format/parse repo objects
borg < 2:

obj = encrypted(compressed(data))

borg 2:

obj = enc_meta_len32 + encrypted(msgpacked(meta)) + encrypted(compressed(data))

handle compr / decompr in repoobj

move the assert_id call from decrypt to RepoObj.parse

also:
- for AEADKeyBase, add a dummy assert_id (not needed here)
- only test assert_id for other if not AEADKeyBase instance
- remove test_getting_wrong_chunk. assert_id is called elsewhere
  and is not needed any more anyway with the new AEAD crypto.
- only give manifest (includes key, repo, repo_objs)
- only return manifest from Manifest.load (includes key, repo, repo_objs)
2022-09-04 00:49:38 +02:00
TW 6c6f10df1e
Merge pull request #6986 from ThomasWaldmann/flags-repo-api
repository api: flags support, fixes #6982
2022-08-22 18:45:21 +02:00
Thomas Waldmann 90ca04f535 repository api: flags support, fixes #6982
- .list: only return IDs for objects where flags & mask == value.
- .flags(_many) (new) to set/query flags
2022-08-22 18:16:59 +02:00
TW 204ecece09
Merge pull request #6985 from ThomasWaldmann/minor-fixes
minor cleanup: fix typos and punctuation in comments
2022-08-22 11:17:43 +02:00
Thomas Waldmann 15d56b2ff7 minor cleanup: fix typos and punctuation in comments 2022-08-21 11:36:27 +02:00
TW 709d162ee7
Merge pull request #6974 from ThomasWaldmann/fixes-master
misc fixes
2022-08-17 08:31:58 +02:00
Thomas Waldmann 29ae701ec9 fix test_recreate_timestamp
it did not work in timezone utc-0800.
2022-08-17 08:07:14 +02:00
Thomas Waldmann 732ecb29fe update prune docs, -P is gone, -a replaces it 2022-08-14 22:01:14 +02:00
TW 66ace47316
Merge pull request #6972 from ThomasWaldmann/update-changes-master
update CHANGES
2022-08-14 01:22:06 +02:00
Thomas Waldmann e078337c7c update CHANGES 2022-08-14 00:08:51 +02:00
TW 34a67d2872
Merge pull request #6971 from ThomasWaldmann/formatting-fixes
fix spacing (found by coala)
2022-08-13 23:56:21 +02:00
Thomas Waldmann ac4f96ec9f fix spacing (found by coala) 2022-08-13 23:37:07 +02:00
TW eb8f1c20c5
Merge pull request #6970 from ThomasWaldmann/refactor-archiver-cmds
archiver modules: rename to *_cmd[s]
2022-08-13 23:25:43 +02:00
Thomas Waldmann 96139ae3c9 adapt pep8 ignores 2022-08-13 23:03:29 +02:00
Thomas Waldmann 66e74e0471 archiver modules: rename to *_cmd[s]
this was already required for list_cmd (due to "list"
being a builtin), now renamed all else also.
2022-08-13 22:59:48 +02:00
TW fe6cd9e008
Merge pull request #6969 from ThomasWaldmann/refactor-master
refactor, move stuff around
2022-08-13 22:45:33 +02:00
Thomas Waldmann 9367ad2f07 adapt pep8 ignores 2022-08-13 22:24:20 +02:00
Thomas Waldmann 403ff07dde move nanorst module to borg.helpers 2022-08-13 22:10:13 +02:00
Thomas Waldmann ecd7bce5b8 move shellpattern module to borg.helpers 2022-08-13 22:03:44 +02:00
Thomas Waldmann 578639b35e move lrucache module to borg.helpers 2022-08-13 22:02:04 +02:00
Thomas Waldmann 9beaced33c move manifest module from helpers to borg.manifest 2022-08-13 21:55:12 +02:00
Thomas Waldmann ded384929d move tar related code to borg.archiver.tar 2022-08-13 21:31:41 +02:00
TW 900398f927
Merge pull request #6968 from ThomasWaldmann/cleanup-prune-master
move prune related code to borg.archiver.prune
2022-08-13 21:28:33 +02:00
Thomas Waldmann 22804f05f9 move prune related code to borg.archiver.prune 2022-08-13 20:59:18 +02:00
TW 3715e327a6
Merge pull request #6964 from ThomasWaldmann/timestamps
Timestamps with timezones
2022-08-13 20:23:31 +02:00
Thomas Waldmann 29db66c8bc transfer: fix archive timestamps to have +00:00 2022-08-13 19:50:49 +02:00
Thomas Waldmann 4ab07b6acb create/recreate/import-tar --timestamp: default to local timezone
if you want to give UTC, just append: +00:00
2022-08-13 19:17:10 +02:00
Thomas Waldmann b4933eb337 get rid of dateutil 2022-08-13 18:31:22 +02:00
Thomas Waldmann ade08ce842 use timezones
- timezone aware timestamps
- str representation with +HHMM or +HH:MM
- get rid of to_locatime
- fix with_timestamp
- have archive start/end time always in local time with tz or as given
- idea: do not lose tz information

then we know when a backup was made and even from
which timezone it was made. if we want to compute
utc, we can do that using these infos.

this makes a quite nice archives list, with timestamps
as expected (in local time with timezone info).

at some places we just enforce utc, like for the
repo manifest timestamp or for the transaction log,
these are usually not looked at by the user.
2022-08-13 18:31:22 +02:00
Thomas Waldmann bab68a8d25 use py37+ datetime.isoformat / .fromisoformat
since python 3.7, .isoformat() is usable IF timespec != "auto"
is given ("auto" [default] would be as evil as before, sometimes
formatting with, sometimes without microseconds).

also since python 3.7, there is now .fromisoformat().
2022-08-11 21:18:56 +02:00
TW b392a60c08
Merge pull request #6962 from ThomasWaldmann/fixes-master
misc fixes / cleanups
2022-08-11 18:38:32 +02:00
Thomas Waldmann c51a0626de remove remainders of borg key change-algorithm
most was already removed previously, just this was overlooked.
2022-08-11 17:42:27 +02:00
Thomas Waldmann a7bc1f88c1 constants: reorder stuff, add comments 2022-08-11 17:36:57 +02:00
Thomas Waldmann a18a9df364 run "black" 2022-08-11 06:34:09 +02:00
Thomas Waldmann 252b145833 remove now unused argument() 2022-08-11 06:30:16 +02:00
Thomas Waldmann 8562940b0d with_repository: fake mechanism not needed
was only used by borg create --dry-run, but that
internally does not use the repo anyway.
2022-08-11 06:29:34 +02:00
Thomas Waldmann 8bebf65bc2 with_repository: invert_fake arg is not used 2022-08-11 06:19:23 +02:00