If not set, it will default to "legacy" (always return 2 for errors).
This commit only changes the Error exception class and its subclasses.
The more specific exit codes need to be defined via .exit_mcode in the subclasses.
Also: use ERROR loglevel for these (not WARNING).
A different amount of index entries was already logged as error
and led to "error_found = True" in repository.check.
Different values in the rebuilt index vs. the on-disk index were
only logged on warning level, but did not lead to error_found = True.
Guess there is no reason why these should not be errors and lead to
error_found = True, so this was fixed in this commit.
Minor related change: change report_error function args, so it can be
called like logger.error - including giving a format AND args.
The intention of LockRoster.modify(key, REMOVE) is to remove self.id.
Using set.discard will just ignore it if self.id is not present there anymore.
Previously, using set.remove triggered a KeyError that has been frequently
seen in tracebacks of teardowns involving Repository.__del__ and Repository.__exit__.
I added a REMOVE2 op to serve one caller that needs to get the KeyError if
self.id was not present.
Thanks to @herrmanntom for the workaround!
When borg invokes a system command, it needs to prepare the environment
for that. This is especially important when using a pyinstaller-made
borg fat binary that works with a modified env var LD_LIBRARY_PATH -
system commands may crash with that.
borg already had calls to prepare_subprocess_env at some places (e.g.
when invoking ssh for the remote repo connection), but they were
missing for:
borg create --content-from-command ...
borg create --paths-from-command ...
before this fix, borg check --repair just created an
empty shadow index, which can lead to incomplete
entries if entries are added later.
and such incomplete (but present) entries can lead to
compact_segments() resurrecting old PUTs by accidentally
dropping related DELs.
get_args() exception handling before this fix only dealt with
subclasses of "Error", but we have to expect other exceptions
there, too.
In any case, if we have some fatal exception here, we must
terminate with rc 2.
ArgumentTypeError: emit a short error message - usually this is
a user error, invoking borg in a wrong way.
Other exceptions: full info and traceback.
for the other compression methods, this is done in
the base class, but the zlib legacy does not call
that method as it also removes the header bytes,
which zlib legacy does not have.
also: do a small optimisation in borg check:
if the type of the repo object is not ROBJ_ARCHIVE_META, we
can skip the object, it can not contain valid archive meta data.
if the type is correct, this is already a sufficient check, so
we can be quite sure that there will be valid archive metadata
in the object.
writing: put type into repoobj metadata
reading: check wanted type against type we got
repoobj metadata is encrypted and authenticated.
repoobj data is encrypted and authenticated, also (separately).
encryption and decryption of both metadata and data get the
same "chunk ID" as AAD, so both are "bound" to that (same) ID.
a repo-side attacker can neither see cleartext metadata/data,
nor successfully tamper with it (AEAD decryption would fail).
also, a repo-side attacker could not replace a repoobj A with a
differently typed repoobj B without borg noticing:
- the metadata/data is cryptographically bound to its ID.
authentication/decryption would fail on mismatch.
- the type check would fail.
thus, the problem (see CVEs in changelog) solved in borg 1 by the
manifest and archive TAMs is now already solved by the type check.
For many use cases, the repo-wide "rcompress" is more efficient.
Also, recreate --recompress calls add_chunk with overwrite=True,
which is unsupported with the AdHocCache.
remove a lot of complexity from the code that was just there to
support legacy borg versions < 1.0.9 which did not TAM authenticate
the manifest.
since then, borg writes TAM authentication to the manifest,
even if the repo is unencrypted.
if the repo is unencrypted, it did not check the somehow pointless
authentication that was generated without any secret, but
if we add that fake TAM, we can also verify the fake TAM.
if somebody explicitly switches off all crypto, they can not
expect authentication.
for everybody else, borg now always generates the TAM and also
verifies it.
rebuild_refcounts verifies and recreates the TAM.
Now it re-uses the salt, so that the archive ID does not change
just because of a new salt if the archive has still the same data.
list: shows either "verified" or "none", depending on
whether a TAM auth tag could be verified or was
missing (old archives from borg < 1.0.9).
when loading an archive, we now try to verify the archive
TAM, but we do not require it. people might still have
old archives in their repos and we want to be able to
list such repos without fatal exceptions.
This part of the archive checker recreates the Archive
items (always, just in case some missing chunks needed
repairing).
When loading the Archive item, we now verify the TAM.
When saving the (potentially modified) Archive item,
we now (re-)generate the TAM.
Archives without a valid TAM are dropped rather than TAM-authenticated
when saving them. There shouldn't be any archives without a valid TAM:
- borg writes an archive TAM since long (1.0.9)
- users are expected to TAM-authenticate archives created
by older borg when upgrading to borg 1.2.5.
Also:
Archive.set_meta: TAM-authenticate new archive
This is also used by Archive.rename and .recreate.
In these tests, we only compare paths, but we do not
need to create these paths for that. By not trying to
create them, we can avoid permission issues, e.g. under
fakeroot.
- master branch has different free space requirements from 1.2-maint,
so we now use a 700MB filesystem
- used pytest.mark.parametrize for the test passes, kind of a progress
display
- fix bug in rcreate call, encryption arg is needed
- fix bug in lock file cleanup
- added repo space cleanup
- updated docstring with current linux instructions (ubuntu)
- stopped using the "reserved" files, the "input" files are good enough
to get some space freed.
-
Thomas Waldmann