an acd_cli (amazon cloud drive fuse filesystem) user had "borg init" crash in the line below.
by adding the assertion we tell that we do not expect the transaction_id to be None there,
so it is easier to differentiate from a random coding error.
we will not get() objects that have a segment entry larger than MAX_OBJECT_SIZE.
thus we should never produce such entries.
also: introduce repository.MAX_DATA_SIZE that gives the max payload size.
old clients use self.exclusive = None and do a read->write lock upgrade when needed.
new clients use self.exclusive = True/False and never upgrade.
replay fakes an old client by setting self.exclusive = None to get a lock upgrade if needed.
if we don't, we try to upgrade the lock.
this is to support old clients talking to a new server and also
to avoid bad consequences from coding mistakes for new clients.
Empty index file is most likely a result from an unclean
shutdown in the middle of write, e.g. on ext4 with delayed
allocation enabled (default).
Ignoring such a file would get it recreated by other parts of code,
where as not ignoring it leads to an exception about
not being able to read enough bytes from the index.
this commit fixes#1195
Signed-off-by: Oleg Drokin <green@linuxhacker.ru>
(Remote)Repository.close() is not a public API anymore, but a private
API. It shall not be used from within other classes than Repository
or it's tests. The proper way is to use a context manager now. However,
for RPC/Remote compatibility with Borg 1.0 it is kept and unchanged.
Repositories are not opened by __init__ now anymore, it is done
by binding it to a context manager. (This SHOULD be compatible both ways
with remote, since opening the repo is handled by a RepositoryServer method)
Decorators @with_repository() and @with_archive simplify
context manager handling and remove unnecessary indentation.
Backported to 1.0-maint
New repository config item, repository.append_only, causes Borg to never
delete or append to existing data files. Hints and indices are handled
as before, old ones are deleted, because they can get quite large, but
are automatically reconstructed: no need to keep them.
When append_only is activated a file /path/to/repo/transactions/<NUMBER>
will be created for every commit.
Deleting all segments <NUMBER+1> and higher will rollback to that commit.
Note that this only influences Borg behaviour. Since repository config
can't be altered remotely (except for repository.key) this can't be
disabled when accessed remotely over SSH with "borg serve" as the
forced command.
This is only a feature to support the use case, and does not replace
appropriate file system permissions or monitoring.
Resolves#809
as soon as one target segment is full, it is a good time to commit it and remove the source segments
that are already completely unused (because they were transferred int the target segment).
so, for compact_segments(save_space=True), the additional space needed should be about 1 segment size.
note: we can't just do that at the end of one source segment as this might create very small
target segments, which is not wanted.
The read_msgpack and write_msgpack functions were only used in one place
each. Since msgpack is read and written in lots of places, having
functions with these generic names is confusing. Also, the helpers
module is quite a mess, so reducing its size seems to be a good idea.
due to borg's architecture, breaking the repo lock needs first creating a repository object.
this would usually try to get a lock and then block if there already is one.
thus I added a flag to open without trying to create a lock.
this was making us require mock, which is really a test component and
shouldn't be part of the runtime dependencies. furthermore, it was
making the imports and the code more brittle: it may have been
possible that, through an environment variable, backups could be
corrupted because mock libraries would be configured instead of real
once, which is a risk we shouldn't be taking.
finally, this was used only to build docs, which we will build and
commit to git by hand with a fully working borg when relevant.
see #384.
added try/finally (the code in between was just indented, no
other code changes) to make sure it sets self.index back to None,
even if the code crashes e.g. due to an IntegrityError caused
by an incomplete segment caused by a disk full condition.
also, in prepare_txn, create an empty in-memory index if transaction_id
is None, which is required by the Repository.check code to work correctly.
If the index is not empty there, it will miscalculate segment usage
(self.segments).
subclasses of "Error": do not show traceback
(this is used when a failure is expected and has rather trivial reasons and usually
does not need debugging)
subclasses of "ErrorWithTraceback": show a traceback
(this is for severe and rather unexpected stuff, like consistency / corruption issues
or stuff that might need debugging)
I reviewed all the Error subclasses whether they fit into the one or other class.
Also: fixed docstring typo, docstring formatting
instead of applying this only to usage generation, use it as a generic
mechanism to disable loading of Cython code.
it may be incomplete: there may be other places where Cython code is
loaded that is not checked, but that is sufficient to build the usage
docs. the environment variable used is documented as such in the
docs/usage.rst.
we also move the check to a helper function and document it
better. this has the unfortunate side effect of moving includes
around, but I can't think of a better way.
this is such a crude hack it is totally embarrassing....
the proper solution would probably be to move the `build_parser()`
function out of `Archiver` completely, but this is such an undertaking
that i doubt it is worth doing since we're looking at switching to
click anyways.
the main problem in moving build_parser() out is that it references
`self` all the time, so it *needs* an archiver context that it can
reuse. we could make the function static and pass self in there by
hand, but it seems like almost a worse hack... and besides, we would
need to load the archiver in order to do that, which would break usage
all over again...
Marian Beermann