see ticket and borg.helpers.msgpack docstring.
this changeset implements the full migration to
msgpack 2.0 spec (use_bin_type=True, raw=False).
still needed compat to the past is done via want_bytes decoder in borg.item.
the check only considered old key -> new key changes, but
new key to new key is of course also fine.
e.g. repokey-aes-ocb -> repokey-aes-ocb (both use hmac-sha256
as id hash)
it potentially will ask for the passphrase for the key of OTHERREPO.
for the newly created repo, it will use the same passphrase.
it will copy: enc_key, enc_hmac_key, id_key, chunker_seed.
keeping the id_key (and id algorithm) and the chunker seed (and chunker
algorithm and parameters) is desirable for deduplication.
the id algorithm is usually either HMAC-SHA256 or BLAKE2b.
keeping the enc_key / enc_hmac_key must be implemented carefully:
A) AES-CTR -> AES-CTR is INSECURE due to nonce reuse, thus not allowed.
B) AES-CTR -> AEAD with session keys is secure.
C) AEAD with session keys -> AEAD with session keys is secure.
AEAD modes with session keys: AES-OCB and CHACHA20-POLY1305.
when migrating from repokey to keyfile, we just store an empty key into the repo config,
because we do not have a "delete key" RPC api. thus, empty key means "there is no key".
here we fix load_key, so that it does not behave differently for no key and empty key:
in both cases, it just returns an empty value.
additionally, we strip the value we get from the config, so whitespace does not matter.
All callers now check for the repokey not being empty, otherwise RepoKeyNotFoundError
is raised.
Argon2 the second part: implement encryption/decryption of argon2 keys
borg init --key-algorithm=argon2 (new default, older pbkdf2 also still available)
borg key change-passphrase: keep key algorithm the same
borg key change-location: keep key algorithm the same
use env var BORG_TESTONLY_WEAKEN_KDF=1 to resource limit (cpu, memory, ...) the kdf when running the automated tests.
we already have .decrypt(id, data, ...).
i changed .encrypt(chunk) to .encrypt(id, data).
the old borg crypto won't really need or use the id,
but the new AEAD crypto will authenticate the id in future.
encrypt used to "patch" the IV into the header,
decrypt used to fetch it from there.
encrypt now takes the header just "as is" and
also decrypt expects that the IV is already set.
also:
cleanup class structure: less inheritance, more mixins.
define type bytes using the 4:4 split
upper 4 bits are ciphersuite:
0 == legacy AES-CTR based stuff
1+ == new AEAD stuff
lower 4 bits are keytype:
legacy: a bit mixed up, as it was...
new stuff: 0=keyfile 1=repokey, ...
"passphrase" encryption mode repos can not be created since borg 1.0.
back then, users were advised to switch existing repos of that type
to repokey mode using the "borg key migrate-to-repokey" command.
that command is still available in borg 1.0, 1.1 and 1.2, but not
any more in borg >= 1.3.
while we still might see the PassphraseKey.TYPE byte in old repos,
it is handled by the RepoKey code since borg 1.0.
If we create a new repo (and a new keyfile key, create=True),
there must not already exist a keyfile at the path/filename
where we want to write the new one.
In other use cases (e.g. if we overwrite a keyfile due
to the user changing their passphrase, create=False),
of course overwriting at the same path/fname is desired.
Running 'borg key import' on a keyfile repository with the BORG_KEY_FILE
environment variable set works correctly if the BORG_KEY_FILE file
already exists. However, the command crashes if the BORG_KEY_FILE file
does not exist:
$ BORG_KEY_FILE=newborgkey borg key import /home/strager/borg-backups/straglum borgkey
Local Exception
Traceback (most recent call last):
[snip]
File "[snip]/borg/crypto/key.py", line 713, in sanity_check
with open(filename, 'rb') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '[snip]/newborgkey'
Platform: Linux straglum 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.11 Python: CPython 3.7.7 msgpack: 0.5.6
PID: 15306 CWD: /home/strager/Projects/borg
sys.argv: ['[snip]/borg', 'key', 'import', '/home/strager/borg-backups/straglum', 'borgkey']
SSH_ORIGINAL_COMMAND: None
Make 'borg key import' not require the BORG_KEY_FILE file to already
exist.
This commit does not change the behavior of 'borg key import' without
BORG_KEY_FILE. This commit also does not change the behavior of 'borg
key import' on a repokey repository.
I want to change the key lookup logic for the 'borg key import' command.
Extract methods out of the KeyfileKey.find_key and
KeyfileKey.get_new_target to make this future change possible without
duplicating code.
This commit should not change behavior.
wrap msgpack to avoid future upstream api changes making troubles
or that we would have to globally spoil our code with extra params.
make sure the packing is always with use_bin_type=False,
thus generating "old" msgpack format (as borg always did) from
bytes objects.
make sure the unpacking is always with raw=True,
thus generating bytes objects.
note:
safe unicode encoding/decoding for some kinds of data types is done in Item
class (see item.pyx), so it is enough if we care for bytes objects on the
msgpack level.
also wrap exception handling, so borg code can catch msgpack specific
exceptions even if the upstream msgpack code raises way too generic
exceptions typed Exception, TypeError or ValueError.
We use own Exception classes for this, upstream classes are deprecated
if interactive passphrase query fails and the env vars are not set,
show a clear error message about this.
users often do 'BORG_PASSPHRASE=secret', forgetting the 'export'.
or they use sudo (and not sudo -E).
in both cases, the env vars won't be available for the borg process.
Thomas Waldmann