writing: put type into repoobj metadata
reading: check wanted type against type we got
repoobj metadata is encrypted and authenticated.
repoobj data is encrypted and authenticated, also (separately).
encryption and decryption of both metadata and data get the
same "chunk ID" as AAD, so both are "bound" to that (same) ID.
a repo-side attacker can neither see cleartext metadata/data,
nor successfully tamper with it (AEAD decryption would fail).
also, a repo-side attacker could not replace a repoobj A with a
differently typed repoobj B without borg noticing:
- the metadata/data is cryptographically bound to its ID.
authentication/decryption would fail on mismatch.
- the type check would fail.
thus, the problem (see CVEs in changelog) solved in borg 1 by the
manifest and archive TAMs is now already solved by the type check.
macFUSE supports a volname mount option to give what
finder displays on desktop / in directory list.
if the user did not specify it, we make something up,
because otherwise it would be "macFUSE Volume 0 (Python)".
Replacing the internals should make the implementation faster
and simpler since the order tracking is done by the `OrderedDict`.
Furthermore, this commit adds type hints to `LRUCache` and
renames the `upd` method to `replace` to make its use more clear.
we now just treat that one .borg_part file we might have inside
checkpoint archives as a normal file.
people can recognize via the file name it is a partial file.
nobody cares for statistics of checkpoint files and the final
archive now does not contain any partial files any more, thus
no needs to maintain statistics about count and size of part
files.
borg < 2:
obj = encrypted(compressed(data))
borg 2:
obj = enc_meta_len32 + encrypted(msgpacked(meta)) + encrypted(compressed(data))
handle compr / decompr in repoobj
move the assert_id call from decrypt to RepoObj.parse
also:
- for AEADKeyBase, add a dummy assert_id (not needed here)
- only test assert_id for other if not AEADKeyBase instance
- remove test_getting_wrong_chunk. assert_id is called elsewhere
and is not needed any more anyway with the new AEAD crypto.
- only give manifest (includes key, repo, repo_objs)
- only return manifest from Manifest.load (includes key, repo, repo_objs)
borg now has the chunks list in every item with content.
due to the symmetric way how borg now deals with hardlinks using
item.hlid, processing gets much simpler.
but some places where borg deals with other "sources" of hardlinks
still need to do some hardlink management:
borg uses the HardLinkManager there now (which is not much more
than a dict, but keeps documentation at one place and avoids some
code duplication we had before).
item.hlid is computed via hardlink_id function.
support hardlinked symlinks, fixes#2379
as we use item.hlid now to group hardlinks together,
there is no conflict with the item.source usage for
symlink targets any more.
2nd+ hardlinks now add to the files count as did the 1st one.
for borg, now all hardlinks are created equal.
so any hardlink item with chunks now adds to the "file" count.
ItemFormatter: support {hlid} instead of {source} for hardlinks
it does not make sense to request versions view if you only
look at 1 archive, but the code shall not crash in that case
as it did, but give a clear error msg.
this is different default behaviour than in borg < 1.2:
default (numeric_owner=False) is to use the user/group name from the archive,
look up the local uid / gid and then use that for the FUSE fs.
when --numeric-owner is given (numeric_owner=True), then the uid/gid
from the archive is directly used (as it was the default behaviour in
borg < 1.2).
this was implemented like this (changing the default behaviour) to make
borg mount and borg extract behave more similar considering usage of
user/group numeric archived ids or archived names mapped to corresponding
numeric local system ids.
also, both now use the same function to get the uid/gid from the item.
fuse:
- add user and group name entries to default_dir
- also: set internal_dict(!) of new Item with data from Item.as_dict()
- add a daemonizing() ctx manager
The foreground borg mount process (local repo) survives until the lock
migration (performed by the background) is finished, so the lock to be
migrated is never stale, and the race condition is gone.
- add a test case revealing that locking is not safe during daemonization (borg mount)
- amend printing in testsuite.archiver
setting it to 255 for now (as seen on Linux / ext4),
better than the default 0.
the attribute is present since llfuse 1.3.0, which is
the minimum requirement currently anyway.
Fix FUSE low linear read speed on large files
This patch is a quick fix for an inefficient seek algorithm used in read() function:
1) On each read call chunk list for given file is regenerated.
2) Finding needed chunk takes a lot of time for large offsets.
For 64 Gb files read speed drops to 16 MB/s, for 128 Gb - to 8 MB/s, and so on.
After applying this patch read speed becomes 270 MB/s, and does not depend on read position inside file.
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
"default_permissions" is now enforced by borg by default to let the
kernel check uid/gid/mode based permissions.
"ignore_permissions" can be given to not enforce "default_permissions".
note: man mount.fuse explicitly tells about the security issue:
default_permissions
By default FUSE doesn't check file access permissions, ...
This option enables permission checking, restricting access
based on file mode.
This option is usually useful together with the allow_other
mount option.
We consider this a pitfall waiting for someone to fall into and this is
why we chose to change the default behaviour for borg.
wrap msgpack to avoid future upstream api changes making troubles
or that we would have to globally spoil our code with extra params.
make sure the packing is always with use_bin_type=False,
thus generating "old" msgpack format (as borg always did) from
bytes objects.
make sure the unpacking is always with raw=True,
thus generating bytes objects.
note:
safe unicode encoding/decoding for some kinds of data types is done in Item
class (see item.pyx), so it is enough if we care for bytes objects on the
msgpack level.
also wrap exception handling, so borg code can catch msgpack specific
exceptions even if the upstream msgpack code raises way too generic
exceptions typed Exception, TypeError or ValueError.
We use own Exception classes for this, upstream classes are deprecated
previous commit did not yet support hardlinks correctly, if the
hardlink master was excluded somehow.
added some tests for this, also refactored related tests slightly.
borg mount [options] repo_or_archive mountpoint path [paths...]
paths: you can just give some "root paths" (like for borg extract) to
only partially populate the FUSE filesystem.
Similar for these exclusion group options:
--exclude
--exclude-from
--pattern
--patterns-from
--strip-components
Thomas Waldmann