remove a lot of complexity from the code that was just there to
support legacy borg versions < 1.0.9 which did not TAM authenticate
the manifest.
since then, borg writes TAM authentication to the manifest,
even if the repo is unencrypted.
if the repo is unencrypted, it did not check the somehow pointless
authentication that was generated without any secret, but
if we add that fake TAM, we can also verify the fake TAM.
if somebody explicitly switches off all crypto, they can not
expect authentication.
for everybody else, borg now always generates the TAM and also
verifies it.
rebuild_refcounts verifies and recreates the TAM.
Now it re-uses the salt, so that the archive ID does not change
just because of a new salt if the archive has still the same data.
list: shows either "verified" or "none", depending on
whether a TAM auth tag could be verified or was
missing (old archives from borg < 1.0.9).
when loading an archive, we now try to verify the archive
TAM, but we do not require it. people might still have
old archives in their repos and we want to be able to
list such repos without fatal exceptions.
This is an emergency workaround for authenticated repos
if the user has lost the borg key.
We can't compute the TAM key without the borg key, so just
skip all the TAM stuff.
while on macOS the new and old security dir location is the same path,
this is not the case on e.g. Linux, it could move from .config/borg/security to
.local/share/borg/security .
See #5760.
not needed for borg2 repos (we derive a new session key for each borg
invocation and start counting from 0).
also not needed for borg 1.x repos because we only read them (borg transfer)
and won't write new encrypted data to them.
borg < 2:
obj = encrypted(compressed(data))
borg 2:
obj = enc_meta_len32 + encrypted(msgpacked(meta)) + encrypted(compressed(data))
handle compr / decompr in repoobj
move the assert_id call from decrypt to RepoObj.parse
also:
- for AEADKeyBase, add a dummy assert_id (not needed here)
- only test assert_id for other if not AEADKeyBase instance
- remove test_getting_wrong_chunk. assert_id is called elsewhere
and is not needed any more anyway with the new AEAD crypto.
- only give manifest (includes key, repo, repo_objs)
- only return manifest from Manifest.load (includes key, repo, repo_objs)
also added some .pyi files needed to check the cython code (taken from #5703 and updated).
fixed "syntax error" in key.py.
all mypy complaints not fixed yet.
These are legacy crypto modes based on AES-CTR mode:
(repokey|keyfile)[-blake2]
New crypto modes with session keys and AEAD ciphers:
(repokey|keyfile)[-blake2]-(aes-ocb|chacha20-poly1305)
Tests needed some changes:
- most used repokey/keyfile, changed to new modes
- some nonce tests removed, the new crypto code does not generate
the repo side nonces any more (were only used for AES-CTR)
see ticket and borg.helpers.msgpack docstring.
this changeset implements the full migration to
msgpack 2.0 spec (use_bin_type=True, raw=False).
still needed compat to the past is done via want_bytes decoder in borg.item.
the check only considered old key -> new key changes, but
new key to new key is of course also fine.
e.g. repokey-aes-ocb -> repokey-aes-ocb (both use hmac-sha256
as id hash)
it potentially will ask for the passphrase for the key of OTHERREPO.
for the newly created repo, it will use the same passphrase.
it will copy: enc_key, enc_hmac_key, id_key, chunker_seed.
keeping the id_key (and id algorithm) and the chunker seed (and chunker
algorithm and parameters) is desirable for deduplication.
the id algorithm is usually either HMAC-SHA256 or BLAKE2b.
keeping the enc_key / enc_hmac_key must be implemented carefully:
A) AES-CTR -> AES-CTR is INSECURE due to nonce reuse, thus not allowed.
B) AES-CTR -> AEAD with session keys is secure.
C) AEAD with session keys -> AEAD with session keys is secure.
AEAD modes with session keys: AES-OCB and CHACHA20-POLY1305.
when migrating from repokey to keyfile, we just store an empty key into the repo config,
because we do not have a "delete key" RPC api. thus, empty key means "there is no key".
here we fix load_key, so that it does not behave differently for no key and empty key:
in both cases, it just returns an empty value.
additionally, we strip the value we get from the config, so whitespace does not matter.
All callers now check for the repokey not being empty, otherwise RepoKeyNotFoundError
is raised.
Argon2 the second part: implement encryption/decryption of argon2 keys
borg init --key-algorithm=argon2 (new default, older pbkdf2 also still available)
borg key change-passphrase: keep key algorithm the same
borg key change-location: keep key algorithm the same
use env var BORG_TESTONLY_WEAKEN_KDF=1 to resource limit (cpu, memory, ...) the kdf when running the automated tests.
we already have .decrypt(id, data, ...).
i changed .encrypt(chunk) to .encrypt(id, data).
the old borg crypto won't really need or use the id,
but the new AEAD crypto will authenticate the id in future.
encrypt used to "patch" the IV into the header,
decrypt used to fetch it from there.
encrypt now takes the header just "as is" and
also decrypt expects that the IV is already set.
Thomas Waldmann