Note: this is the default cache implementation in borg 1.x,
it worked well, but there were some issues:
- if the local chunks cache got out of sync with the repository,
it needed an expensive rebuild from the infos in all archives.
- to optimize that, a local chunks.archive.d cache was used to
speed that up, but at the price of quite significant space needs.
AdhocCacheWithFiles replaced this with a non-persistent chunks cache,
requesting all chunkids from the repository to initialize a simplified
non-persistent chunks index, that does not do real refcounting and also
initially does not have size information for pre-existing chunks.
We want to move away from precise refcounting, LocalCache needs to die.
much faster and easier now, similar to what borg delete --force --force used to do.
considering that speed, no need for checkpointing anymore.
--stats does not work that way, thus it was removed. borg compact now shows some stats.
Features:
- exclusive and non-exclusive locks
- acquire timeout
- lock auto-expiry (after 30mins of inactivity), lock refresh
- use tz-aware datetimes (in utc timezone) in locks
Also:
- document lock acquisition rules in the src
- increased default BORG_LOCK_WAIT to 10s
- better document with-lock test
Stale locks are ignored and automatically deleted.
Default: stale == 30 Minutes old.
lock.refresh() can be called frequently to avoid that an acquired lock becomes stale.
It does not do much if the last real refresh was recently.
After stale/2 time it checks and refreshes the locks in the store.
Update the repository3 code to call refresh frequently:
- get/put/list/scan
- inside check loop
borg transfer is primarily a general purpose archive transfer function
from borg2 to related borg2 repos.
but for upgrades from borg 1.x, we also need to support:
- rcreate with a borg 1.x "other repo"
- transfer with a borg 1.x "other repo"
It uses xxh64 hashes of the meta and data parts to verify their validity.
On a server with borg, this can be done server-side without the borg key.
The new RepoObj header has meta_size, data_size, meta_hash and data_hash.
Simplify the repository a lot:
No repository transactions, no log-like appending, no append-only, no segments,
just using a key/value store for the individual chunks.
No locking yet.
Also:
mypy: ignore missing import
there are no library stubs for borgstore yet, so mypy errors without that option.
pyproject.toml: install borgstore directly from github
There is no pypi release yet.
use pip install -e . rather than python setup.py develop
The latter is deprecated and had issues installing the "borgstore from github" dependency.
test the healing more thoroughly:
- preservation of correct chunks list in .chunks_healthy
- check that .chunks_healthy is removed after healing
- check that doing another borg check --repair run does not find
something to heal, again.
also did a datatype consistency fix for item.chunks_healthy list
members: they are now post processed in the same way as item.chunks,
so they have type ChunkListEntry rather than simple tuple.
it needs to be like this to support a ~/.pypirc like this,
containing a separate upload token for the borgbackup project:
[distutils]
index-servers =
borgbackup
...
[borgbackup]
repository = https://upload.pypi.org/legacy/
username = __token__
password = pypi-...(token)...
