borg1 needed this due to its transactional / rollback behaviour:
if there was uncommitted stuff in the repo, next repo opening automatically
rolled back to last commit. thus we needed checkpoint archives to reference
chunks and commit the repo.
borg2 does not do that anymore, unused chunks are only removed when the
user invokes borg compact.
thus, if a borg create gets interrupted, the user can just run borg create
again and it will find some chunks are already in the repo, making progress
even if borg create gets frequently interrupted.
This was an implementation specific "in on-disk order" list method that made sense
with borg 1.x log-like segment files only.
But we now store objects separately, so there is no "in on-disk order" anymore.
This was used for an implementation detail of the borg 1.x
repository code, dumping uncommitted objects. Not needed any more.
Also remove local repository method scan_low_level, it was only used by --ghost.
Tests were a bit tricky as there is validation on 2 layers now:
- repository3 does an xxh64 check, finds most corruptions already
- on the archives level, borg also does an even stronger cryptographic check
Dummy returns all-zero stats from that call.
Problem was that these values can't be computed from the chunks cache
anymore. No correct refcounts, often no size information.
Also removed hashindex.ChunkIndex.summarize (previously used by the above mentioned
.stats() call) and .stats_against (unused) for same reason.
Lots of low-level code written back then to optimize runtime of some
functions.
We'll solve this differently by doing less stats, esp. if it is expensive to compute.
Note: this is the default cache implementation in borg 1.x,
it worked well, but there were some issues:
- if the local chunks cache got out of sync with the repository,
it needed an expensive rebuild from the infos in all archives.
- to optimize that, a local chunks.archive.d cache was used to
speed that up, but at the price of quite significant space needs.
AdhocCacheWithFiles replaced this with a non-persistent chunks cache,
requesting all chunkids from the repository to initialize a simplified
non-persistent chunks index, that does not do real refcounting and also
initially does not have size information for pre-existing chunks.
We want to move away from precise refcounting, LocalCache needs to die.
much faster and easier now, similar to what borg delete --force --force used to do.
considering that speed, no need for checkpointing anymore.
--stats does not work that way, thus it was removed. borg compact now shows some stats.
Features:
- exclusive and non-exclusive locks
- acquire timeout
- lock auto-expiry (after 30mins of inactivity), lock refresh
- use tz-aware datetimes (in utc timezone) in locks
Also:
- document lock acquisition rules in the src
- increased default BORG_LOCK_WAIT to 10s
- better document with-lock test
Stale locks are ignored and automatically deleted.
Default: stale == 30 Minutes old.
lock.refresh() can be called frequently to avoid that an acquired lock becomes stale.
It does not do much if the last real refresh was recently.
After stale/2 time it checks and refreshes the locks in the store.
Update the repository3 code to call refresh frequently:
- get/put/list/scan
- inside check loop
borg transfer is primarily a general purpose archive transfer function
from borg2 to related borg2 repos.
but for upgrades from borg 1.x, we also need to support:
- rcreate with a borg 1.x "other repo"
- transfer with a borg 1.x "other repo"
It uses xxh64 hashes of the meta and data parts to verify their validity.
On a server with borg, this can be done server-side without the borg key.
The new RepoObj header has meta_size, data_size, meta_hash and data_hash.
Simplify the repository a lot:
No repository transactions, no log-like appending, no append-only, no segments,
just using a key/value store for the individual chunks.
No locking yet.
Also:
mypy: ignore missing import
there are no library stubs for borgstore yet, so mypy errors without that option.
pyproject.toml: install borgstore directly from github
There is no pypi release yet.
use pip install -e . rather than python setup.py develop
The latter is deprecated and had issues installing the "borgstore from github" dependency.
test the healing more thoroughly:
- preservation of correct chunks list in .chunks_healthy
- check that .chunks_healthy is removed after healing
- check that doing another borg check --repair run does not find
something to heal, again.
also did a datatype consistency fix for item.chunks_healthy list
members: they are now post processed in the same way as item.chunks,
so they have type ChunkListEntry rather than simple tuple.
it needs to be like this to support a ~/.pypirc like this,
containing a separate upload token for the borgbackup project:
[distutils]
index-servers =
borgbackup
...
[borgbackup]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-...(token)...
