1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2025-01-19 14:02:55 +00:00
Commit graph

8750 commits

Author SHA1 Message Date
Thomas Waldmann
4fe6d7fd19
src: renamed R*MixIn archiver classes to Repo*MixIn, fixed imports 2024-09-08 15:37:41 +02:00
Thomas Waldmann
f8d07e7d16
src: renamed r* archiver modules to repo_*, fixed imports 2024-09-08 15:33:11 +02:00
TW
ea08e49210
Merge pull request #8332 from ThomasWaldmann/use-borgstore
use borgstore and other big changes
2024-09-08 15:16:24 +02:00
Thomas Waldmann
3794e32890
--append-only and --storage-quota are not supported (yet?) 2024-09-08 12:32:35 +02:00
Thomas Waldmann
b50ed04ffc
build_usage / build_man 2024-09-08 12:02:17 +02:00
Thomas Waldmann
7442cbf8cf
update CHANGES 2024-09-08 12:01:10 +02:00
Thomas Waldmann
682aedba50
check --repair --undelete-archives: bring archives back from the dead
borg delete and borg prune do a quick and dirty archive deletion,
just removing the archives directory entry for them.

--undelete-archives can still find the archive metadata objects
by completely scanning the repository and re-create missing
archives directory entries.

but only until borg compact would remove all unused data.

if only the manifest is missing or corrupted, do not run that
scan, it is not required for the manifest anymore.
2024-09-08 12:01:06 +02:00
Thomas Waldmann
bc1f90b641
check: do not create addtl. archives dir entries if we already have one
if the manifest file is missing, check generated *.1 *.2 ... archives although an entry for the correct name and id was already
present. BUG!

this is because if the manifest is lost, that does not imply
anymore that the complete archives directory is also lost, as it
did in borg 1.x.

Also improved log messages a bit.
2024-09-08 12:01:05 +02:00
Thomas Waldmann
a509a0c463
locking: no traceback on lock timeout (expected) 2024-09-08 12:01:03 +02:00
Thomas Waldmann
0e183b225d
shared locking for many borg commands
not for check and compact, these need an exclusive lock.

to try parallel repo access on same machine, same user,
one needs to use a non-locking cache implementation:

export BORG_CACHE_IMPL=adhoc

this is slow due the missing files cache in that implementation,
but unproblematic because no caches/indexes are persisted.
2024-09-08 12:01:01 +02:00
Thomas Waldmann
8412168555
check: only write to repo if --repair is given
old borg just didn't commit the transaction and
thus caused a transaction rollback if not in
repair mode.

we can't do that anymore, thus we must avoid
modifying the repo if not in repair mode.
2024-09-08 12:01:00 +02:00
Thomas Waldmann
ef7dd76da1
manifest: no read-modify-write for borgstore archives list
previously, borg always read all archives entries, modified the
list in memory, wrote back to the repository (similar as borg 1.x
did).

now borg works directly with archives/* in the borgstore.
2024-09-08 12:00:58 +02:00
Thomas Waldmann
b56c81bf62
manifest.archives: refactor api
Archives was built with a dictionary-like api,
but in future we want to go away from a
read-modify-write archives list.
2024-09-08 12:00:57 +02:00
Thomas Waldmann
b82ced274f
refactor: move archives related code from Manifest to Archives class 2024-09-08 12:00:54 +02:00
Thomas Waldmann
60e88efa94
repository: catch store backend exception, re-raise as repo exception 2024-09-08 12:00:53 +02:00
Thomas Waldmann
ace97fadec
docs: updates / removing outdated stuff 2024-09-08 12:00:50 +02:00
Thomas Waldmann
b14c050f69
rspace: manage reserved space in repository 2024-09-07 22:05:05 +02:00
Thomas Waldmann
1cd2f4dca3
locking: deal with potential auto-expire during suspend 2024-09-07 22:05:04 +02:00
Thomas Waldmann
7bf0f47fea
check repository: implement --max-duration and checkpoints, fixes #6039 2024-09-07 22:05:02 +02:00
Thomas Waldmann
60a592d50f
with-lock: refresh repo lock while subprocess is running, fixes #8347
otherwise the lock might become stale and could get
killed by any other borg process.

note: ThreadRunner class written by PyCharm AI and
only needed small enhancements. nice.
2024-09-07 22:05:00 +02:00
Thomas Waldmann
dc9fff9953
locking: ignore+delete locks of dead processes 2024-09-07 22:04:58 +02:00
Thomas Waldmann
86dc673649
compact: fix dsize computation if wanted chunks are present in repo 2024-09-07 22:04:56 +02:00
Thomas Waldmann
551834acfc
rcompress: not supported for legacy repos 2024-09-07 22:04:55 +02:00
Thomas Waldmann
e2aa9d56d0
build_chunkindex_from_repo: reduce code duplication 2024-09-07 22:04:53 +02:00
Thomas Waldmann
07ab6e02f4
hashindex types: remove some unused stuff 2024-09-07 22:04:52 +02:00
Thomas Waldmann
15c70397c1
ChunkIndex: remove unused .merge method
LocalCache used this to assemble a new overall chunks index from multiple
chunks.archive.d's single-archive chunks indexes.
2024-09-07 22:04:50 +02:00
Thomas Waldmann
ddf6812703
ChunkIndex: remove .incref method 2024-09-07 22:04:49 +02:00
Thomas Waldmann
ccc84c7a4e
cache: renamed .chunk_incref -> .reuse_chunk, boolean .seen_chunk
reuse_chunk is the complement of add_chunk for already existing chunks.

It doesn't do refcounting anymore.

.seen_chunk does not return the refcount anymore, but just whether the chunk exists.

If we add a new chunk, it immediately sets its refcount to MAX_VALUE, so
there is no difference anymore between previously existing chunks and new
chunks added. This makes the stats even more useless, but we have less complexity.
2024-09-07 22:04:47 +02:00
Thomas Waldmann
f9d2e6827b
ArchiveChecker: don't do precise refcounting here
That's the job of borg compact and not needed inside borg check.
check only needs to know if a chunk is present in the repo.
2024-09-07 22:04:46 +02:00
Thomas Waldmann
e9c42a7d6b
ArchiveChecker: .rebuild_refcounts -> .rebuild_archives 2024-09-07 22:04:44 +02:00
Thomas Waldmann
266e6caa80
ArchiveChecker: remove unused possibly_superseded code
We don't care about unused or superseded repo objects any more here,
borg compact will deal with them.
2024-09-07 22:04:43 +02:00
Thomas Waldmann
bafbf625e7
ArchiveChecker.verify_data: simplify / optimize
.init_chunks has just built self.chunks using repository.list(), so don't
call that again, but just iterate over self.chunks.

also some other changes, making the code much simpler.
2024-09-07 22:04:41 +02:00
Thomas Waldmann
ef47666627
cache/hashindex: remove decref method, don't try to remove chunks on exceptions
When the AdhocCache(WithFiles) queries chunk IDs from the repo to build the chunks
index, it won't know their refcount and thus all chunks in the index have their
refcount at the MAX_VALUE (representing "infinite") and that would never decrease
nor could that ever reach zero and get the chunk deleted from the repo.

Only completely new chunks first written in the current borg run have a valid
refcount.

In some exception handlers, borg tried to clean up chunks that won't be used
by an item by decref'ing them. That is either:
- pointless due to refcount being at MAX_VALUE
- inefficient, because the user might retry the backup and would need to
  transmit these chunks to the repo again.

We'll just rely on borg compact ONLY to clean up any unused/orphan chunks.
2024-09-07 22:04:40 +02:00
Thomas Waldmann
d27b7a7981
cache: remove transactions, load files/chunks cache on demand 2024-09-07 22:04:38 +02:00
Thomas Waldmann
57268909f8
upgrade black to 24.x 2024-09-07 22:04:36 +02:00
Thomas Waldmann
a40978ae1b
blacken the code 2024-09-07 22:04:35 +02:00
Thomas Waldmann
ec1d89f477
compact: better stats
- compression factor
- dedup factor
- repo size

All values are approx. values without considering overheads.
2024-09-07 22:04:34 +02:00
Thomas Waldmann
c67cf07522
Repository.list: return [(id, stored_size), ...]
Note: LegacyRepository still returns [id, ...] and so does RemoteRepository.list,
if the remote repo is a LegacyRepository.

also: use LIST_SCAN_LIMIT
2024-09-07 22:03:56 +02:00
Thomas Waldmann
a15cd1e493
repository: remove __len__ and __contains__ 2024-09-07 22:01:21 +02:00
Thomas Waldmann
1a382a8bcf
set repository._location only 2024-09-07 22:01:19 +02:00
Thomas Waldmann
3408e942de
add sftp: / borgstore to quickstart docs 2024-09-07 22:01:18 +02:00
Thomas Waldmann
22b68b0199
add sftp: to repository url format docs 2024-09-07 22:01:16 +02:00
Thomas Waldmann
ec8a127b01
BORG_REPO env var: behave the same when unset or when set to empty string 2024-09-07 22:01:14 +02:00
Thomas Waldmann
7714b6542a
support sftp: repositories via borgstore 2024-09-07 22:01:12 +02:00
Thomas Waldmann
05739aaa65
refactor: rename repository/locking classes/modules
Repository -> LegacyRepository
RemoteRepository -> LegacyRemoteRepository
borg.repository -> borg.legacyrepository
borg.remote -> borg.legacyremote

Repository3 -> Repository
RemoteRepository3 -> RemoteRepository
borg.repository3 -> borg.repository
borg.remote3 -> borg.remote

borg.locking -> borg.fslocking
borg.locking3 -> borg.storelocking
2024-09-07 22:01:11 +02:00
Thomas Waldmann
84bd2b20d5
rcreate: refer to borgstore rather than filesystem directory 2024-09-07 22:01:09 +02:00
Thomas Waldmann
15e759c506
rcompress: fix help and comments
no "on-disk order" anymore.
2024-09-07 22:01:08 +02:00
Thomas Waldmann
8455c95003
info: do not output deduplicated_size
No precise refcounting, can't compute that inexpensively.
2024-09-07 22:01:06 +02:00
Thomas Waldmann
0b85b1a8bd
parseformat: remove dsize and unique_chunks placeholder
We don't have precise refcounts, thus we can't compute these.
2024-09-07 22:01:04 +02:00
Thomas Waldmann
c5023da729
transfer: rather talk of presence than refcount 2024-09-07 22:01:02 +02:00