implemented by introducing one level of indirection, the limit is now
very high, so it is not practically relevant any more.
we always use the indirection (storing the metadata stream chunk ids list not
directly into the archive item, but into some repo objects referenced by the new
ArchiveItem.item_ptrs list).
thus, the code behaves the same for all archive sizes.
Since compression type identification has been split into type and
level, the graphic needed a slight update.
Unfortunately, I don't have access to Visio, so I converted this to odg.
While writing my own out-of-band decoder, I had a hard time figuring out
how to unpack the manifest. From the description, I was only able to
read that the manifest is msgpack'd, but I had not been able to figure
out that it's also going through the same encryption+compression logic
as all other things do.
This should make it a little clearer and provide the necessary
information to understand how the compression works.
in the finished == true message, these are missing:
- message
- current / total
- info
This is to be somewhat consistent with #6683 by only providing a
minimal set of values for the finished case.
The finished messages is primarily intended for cleanup purposes,
e.g. clearing the progress display.
borg now has the chunks list in every item with content.
due to the symmetric way how borg now deals with hardlinks using
item.hlid, processing gets much simpler.
but some places where borg deals with other "sources" of hardlinks
still need to do some hardlink management:
borg uses the HardLinkManager there now (which is not much more
than a dict, but keeps documentation at one place and avoids some
code duplication we had before).
item.hlid is computed via hardlink_id function.
support hardlinked symlinks, fixes#2379
as we use item.hlid now to group hardlinks together,
there is no conflict with the item.source usage for
symlink targets any more.
2nd+ hardlinks now add to the files count as did the 1st one.
for borg, now all hardlinks are created equal.
so any hardlink item with chunks now adds to the "file" count.
ItemFormatter: support {hlid} instead of {source} for hardlinks
"passphrase" encryption mode repos can not be created since borg 1.0.
back then, users were advised to switch existing repos of that type
to repokey mode using the "borg key migrate-to-repokey" command.
that command is still available in borg 1.0, 1.1 and 1.2, but not
any more in borg >= 1.3.
while we still might see the PassphraseKey.TYPE byte in old repos,
it is handled by the RepoKey code since borg 1.0.
docs: clarify on-disk order and size of log entry fields
The order of the fields of a log entry on disk is CRC32 first, the docs had the size first.
I tried to make this list similar to the HashIndex struct description.
the movement of the start of the hashing window stops at (file_size - window_size), thus THAT would be the factor in that formula, not just file_size.
for medium and big files, window_size is much smaller than file_size, so guess we can just say "approximately" for the general case.
a file map can be:
- created internally inside chunkify by calling sparsemap, which uses
SEEK_DATA / SEEK_HOLE to determine data and hole ranges inside a
seekable sparse file.
Usage: borg create --sparse --chunker-params=fixed,BLOCKSIZE ...
BLOCKSIZE is the chunker blocksize here, not the filesystem blocksize!
- made by some other means and given to the chunkify function.
this is not used yet, but in future this could be used to only read
the changed parts and seek over the (known) unchanged parts of a file.
sparsemap: the generate range sizes are multiples of the fs block size.
the tests assume 4kiB fs block size.
We know use only "target chunk size" when speaking of the chunk size
that is expected to happen most of the time. This removes statistical
and mathematical inacurracies that could be troublesome for mathematical
people.
Fixes#5336
rephrase some warnings, fixes#5164
borg check --repair and borg recreate are now present in the code since rather long, so they are not experimental any more.
borg recreate might be used wrongly (e.g. accidentally excluding everything / not matching anything when recreating an archive). added some warning words in the docs, but it will not ask for confirmation any more.
borg check: there might be kinds of corruption borg check --repair can not fix and it might make things even worse while trying to fix. so this will still ask for confirmation, just with different wording.
locking: fix ExclusiveLock race condition bug, fixes#4923
- ExclusiveLock is now based on os.rename instead of os.mkdir.
- catch FileNotFoundError observed under race condition in ExclusiveLock.release()
and .kill_stale_lock()
- added TestExclusiveLock.test_race_condition() which reveals issue #4923
- updated docs
- locking: use "raise LockTimeout from None" for prettier traceback
Co-authored-by: Thomas Portmann <thomas@portmann.org>
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
Thomas Waldmann