borg init calls this. If there is a PermissionError, it is
usually fs permission issue at path or its parent directory.
Don't give a traceback, but rather an error msg and a specific exit code.
this is a fwd port from 1.4-maint. as we don't have nonce files
any more in master, only the generally useful stuff has been ported.
- add Error / ErrorWithTraceback exception classes to RPC layer.
- add hex_to_bin helper
previously, this was handled in RPCError handler and always resulted in rc 2.
now re-raise Lock Exceptions locally, so it gives rc 2 (legacy) or 7x (modern).
writing: put type into repoobj metadata
reading: check wanted type against type we got
repoobj metadata is encrypted and authenticated.
repoobj data is encrypted and authenticated, also (separately).
encryption and decryption of both metadata and data get the
same "chunk ID" as AAD, so both are "bound" to that (same) ID.
a repo-side attacker can neither see cleartext metadata/data,
nor successfully tamper with it (AEAD decryption would fail).
also, a repo-side attacker could not replace a repoobj A with a
differently typed repoobj B without borg noticing:
- the metadata/data is cryptographically bound to its ID.
authentication/decryption would fail on mismatch.
- the type check would fail.
thus, the problem (see CVEs in changelog) solved in borg 1 by the
manifest and archive TAMs is now already solved by the type check.
shutting down logging is problematic as it is global
and we do multi-threaded execution, e.g. in tests.
thus, rather just flush the important loggers and keep
them alive.
server (listening) side:
borg serve --socket # default location
borg serve --socket=/path/to/socket
client side:
borg -r socket:///path/to/repo create ...
borg --socket=/path/to/socket -r socket:///path/to/repo ...
served connections:
- for ssh: proto: one connection
- for socket: proto: many connections (one after the other)
The socket has user and group permissions (770).
skip socket tests on win32, they hang infinitely, until
github CI terminates them after 60 minutes.
socket tests: use unique socket name
don't use the standard / default socket name, otherwise tests
running in parallel would interfere with each other by using
the same socket / the same borg serve process.
write a .pid file, clean up .pid and .sock file at exit
add stderr print for accepted/finished socket connection
- tears down logging (so no new log output is generated afterwards)
- sends all queued log output
- then returns
also: make stdin_fd / stdout_fd instance variables
Instead, install a handler that sends the LogRecord dicts to a queue.
That queue is then emptied in the borg serve main loop and
the LogRecords are sent msgpacked via stdout to the client,
similar to the RPC results.
On the client side, the LogRecords are recreated from the
received dicts and fed into the clientside logging system.
As we use msgpacked LogRecord dicts, we don't need JSON for
this purpose on the borg serve side any more.
On the client side, the LogRecords will then be either formatted
as normal text or as JSON log output (by the clientside log
formatter).
this used to call get_base_dir (and would have needed
legacy=True now to work like expected).
rather implemented the desired behaviour locally and
got rid of the legacy call (which was a bit strange
anyway as it also considered BORG_BASE_DIR, which is
unexpected when resolving ~).
in the sysinfo function, there is a way to suppress
all sysinfo output via an env var and just return an
empty string.
so we can expect it is always in unpacked, but it
might be the empty string.
log output:
always expect json, remove $LOG format support.
we keep limited support for unstructured format also,
just not to lose anything from remote stderr.
rpc format:
ancient borg used tuples in the rpc protocol,
but recent ones use easier-to-work-with dicts.
version info:
we expect dicts with server/client version now.
not needed for borg2 repos (we derive a new session key for each borg
invocation and start counting from 0).
also not needed for borg 1.x repos because we only read them (borg transfer)
and won't write new encrypted data to them.
this option did not change behaviour since longer,
we only had kept it for API compatibility.
as a borg2 repo server won't have old clients talking to it,
we can safely remove this everywhere now.
legacy: add/remove ctype/clevel bytes prefix of compressed data
new: use a separate metadata dict
compressors: use an int as ID, not a len 1 bytestring
borg < 2:
obj = encrypted(compressed(data))
borg 2:
obj = enc_meta_len32 + encrypted(msgpacked(meta)) + encrypted(compressed(data))
handle compr / decompr in repoobj
move the assert_id call from decrypt to RepoObj.parse
also:
- for AEADKeyBase, add a dummy assert_id (not needed here)
- only test assert_id for other if not AEADKeyBase instance
- remove test_getting_wrong_chunk. assert_id is called elsewhere
and is not needed any more anyway with the new AEAD crypto.
- only give manifest (includes key, repo, repo_objs)
- only return manifest from Manifest.load (includes key, repo, repo_objs)
there was no way to tell the repository version for a remote repo.
borg 2 needs that to reject doing most operations with an old repo,
except the stuff needed for borg transfer.
see ticket and borg.helpers.msgpack docstring.
this changeset implements the full migration to
msgpack 2.0 spec (use_bin_type=True, raw=False).
still needed compat to the past is done via want_bytes decoder in borg.item.
add remote upload buffer (--remote-buffer)
- added new option --remote-buffer
- allow to_send to grow to selected size
- don't grow if wait is specified
- fill pipe on any command (including 'async_response')
- add new option to docs
- create EfficientBytesQueue to prevent recreation of buffer each time we send something
- add tests for EfficientBytesQueue
The umask value is NOT transmitted from client to server any more,
so the borg client can not influence the borg server umask any more.
If one wants to have a specific umask on the server side, one needs to
use a ssh forced command in .ssh/authorized_keys file.
OTOH, as the default value is 077 (in general, for client as well as for
the server) and the server does not take the value from the client any more,
there usually should be no need to give it on the server side, IF you are
happy with the default value.
Thomas Waldmann