mirror/borg
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2024-12-24 08:45:13 +00:00
Code Issues Releases Wiki Activity

avoid orphan content chunks on BackupOSError, fixes #6709

Browse source 
if we run into some issue reading an input file, e.g. an I/O error,
the BackupOSError exception raised due to that will skip the current
file and no archive item will be created for this file.

But we maybe have already added some of its content chunks to the repo,
we have either written them as new chunks or incref'd some identical chunk
in the repo.

Added an exception handler that decrefs (and deletes if refcount reaches 0)
these chunks again before re-raising the exception, so the repo is in a
consistent state again and we do not have orphaned content chunks in the repo.
This commit is contained in:
Thomas Waldmann 2023-02-01 01:32:59 +01:00
parent 9b7647c89d
commit ffe32316a5
No known key found for this signature in database
GPG key ID: 243ACFA951F78E01
1 changed files with 25 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
src/borg/archive.py
View file

@ -1300,6 +1300,7 @@ def chunk_processor(chunk):
# to get rid of .chunks_healthy, as it might not correspond to .chunks any more. # to get rid of .chunks_healthy, as it might not correspond to .chunks any more.
if self.rechunkify and "chunks_healthy" in item: if self.rechunkify and "chunks_healthy" in item:
del item.chunks_healthy del item.chunks_healthy
try:
for chunk in chunk_iter: for chunk in chunk_iter:
cle = chunk_processor(chunk) cle = chunk_processor(chunk)
item.chunks.append(cle) item.chunks.append(cle)
@ -1307,6 +1308,23 @@ def chunk_processor(chunk):
if show_progress: if show_progress:
stats.show_progress(item=item, dt=0.2) stats.show_progress(item=item, dt=0.2)
self.maybe_checkpoint(item) self.maybe_checkpoint(item)
except BackupOSError:
# something went wrong (e.g. an I/O error while reading a source file), try to avoid orphan content chunks:
# case A: "no checkpoint archive has been created yet":
# we have incref'd (written) some chunks, no commit yet, no file item for these chunks yet.
# -> item.chunks has a list of orphaned content chunks, we need to decref them.
# case B: "some checkpoint archives have been created already":
# at the time we commit them, everything is fine and consistent:
# we have incref'd (written) some chunks, created a part file item referencing them, committed.
# directly after commit, we have removed the part file item, but kept chunks in the repo, kept refcounts.
# maybe we have incref'd (written) some more chunks after the commit, no file item for these chunks yet.
# -> item.chunks has a list of orphaned content chunks, we need to decref them.
# So, cases A and B need same treatment.
for chunk in item.chunks:
cache.chunk_decref(chunk.id, stats, wait=False)
# now that we have cleaned up the chunk references, we can re-raise the exception
# this will skip THIS processing of this file, but continue with the next one.
raise
class FilesystemObjectProcessors: class FilesystemObjectProcessors: