borg now has the chunks list in every item with content.
due to the symmetric way how borg now deals with hardlinks using
item.hlid, processing gets much simpler.
but some places where borg deals with other "sources" of hardlinks
still need to do some hardlink management:
borg uses the HardLinkManager there now (which is not much more
than a dict, but keeps documentation at one place and avoids some
code duplication we had before).
item.hlid is computed via hardlink_id function.
support hardlinked symlinks, fixes#2379
as we use item.hlid now to group hardlinks together,
there is no conflict with the item.source usage for
symlink targets any more.
2nd+ hardlinks now add to the files count as did the 1st one.
for borg, now all hardlinks are created equal.
so any hardlink item with chunks now adds to the "file" count.
ItemFormatter: support {hlid} instead of {source} for hardlinks
export-tar: just msgpack and b64encode all item metadata and
put that into a BORG specific PAX header.
this is *additional* to the standard tar metadata.
import-tar: when detecting the BORG specific PAX header, just get
all metadata from there (and ignore the standard tar
metadata).
"passphrase" encryption mode repos can not be created since borg 1.0.
back then, users were advised to switch existing repos of that type
to repokey mode using the "borg key migrate-to-repokey" command.
that command is still available in borg 1.0, 1.1 and 1.2, but not
any more in borg >= 1.3.
while we still might see the PassphraseKey.TYPE byte in old repos,
it is handled by the RepoKey code since borg 1.0.
While doing some doc updates I needed a way to test them - to build
the documentation and inspect the output. I ran into an issue:
running python setup.py build_man was throwing exceptions:
1. The import-tar parser had a None description causing:
File "/home/user/borg/setup_docs.py", line 451, in write_heading
write(char * len(header))
TypeError: object of type 'NoneType' has no len()
2. There was no docs/usage/import-tar.rst causing an exception too
add remote upload buffer (--remote-buffer)
- added new option --remote-buffer
- allow to_send to grow to selected size
- don't grow if wait is specified
- fill pipe on any command (including 'async_response')
- add new option to docs
- create EfficientBytesQueue to prevent recreation of buffer each time we send something
- add tests for EfficientBytesQueue
a file map can be:
- created internally inside chunkify by calling sparsemap, which uses
SEEK_DATA / SEEK_HOLE to determine data and hole ranges inside a
seekable sparse file.
Usage: borg create --sparse --chunker-params=fixed,BLOCKSIZE ...
BLOCKSIZE is the chunker blocksize here, not the filesystem blocksize!
- made by some other means and given to the chunkify function.
this is not used yet, but in future this could be used to only read
the changed parts and seek over the (known) unchanged parts of a file.
sparsemap: the generate range sizes are multiples of the fs block size.
the tests assume 4kiB fs block size.
rephrase some warnings, fixes#5164
borg check --repair and borg recreate are now present in the code since rather long, so they are not experimental any more.
borg recreate might be used wrongly (e.g. accidentally excluding everything / not matching anything when recreating an archive). added some warning words in the docs, but it will not ask for confirmation any more.
borg check: there might be kinds of corruption borg check --repair can not fix and it might make things even worse while trying to fix. so this will still ask for confirmation, just with different wording.
locking: fix ExclusiveLock race condition bug, fixes#4923
- ExclusiveLock is now based on os.rename instead of os.mkdir.
- catch FileNotFoundError observed under race condition in ExclusiveLock.release()
and .kill_stale_lock()
- added TestExclusiveLock.test_race_condition() which reveals issue #4923
- updated docs
- locking: use "raise LockTimeout from None" for prettier traceback
Co-authored-by: Thomas Portmann <thomas@portmann.org>
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
Thomas Waldmann