add remote upload buffer (--remote-buffer)
- added new option --remote-buffer
- allow to_send to grow to selected size
- don't grow if wait is specified
- fill pipe on any command (including 'async_response')
- add new option to docs
- create EfficientBytesQueue to prevent recreation of buffer each time we send something
- add tests for EfficientBytesQueue
a file map can be:
- created internally inside chunkify by calling sparsemap, which uses
SEEK_DATA / SEEK_HOLE to determine data and hole ranges inside a
seekable sparse file.
Usage: borg create --sparse --chunker-params=fixed,BLOCKSIZE ...
BLOCKSIZE is the chunker blocksize here, not the filesystem blocksize!
- made by some other means and given to the chunkify function.
this is not used yet, but in future this could be used to only read
the changed parts and seek over the (known) unchanged parts of a file.
sparsemap: the generate range sizes are multiples of the fs block size.
the tests assume 4kiB fs block size.
Using "package" ansible module make the ansible playbook able to run
against most unix OSes. Pacman module only works with Arch and derivatives.
Also : changing state from "latest" to "present". Ansible should not be
a way to keep your system up-to-date : it's a configuration management
system and, as such, should not change anything if neither the playbook
nor the machine changed its state (idempotency).
Reference : https://github.com/ansible/ansible-lint/blob/master/lib/ansiblelint/rules/PackageIsNotLatestRule.py#L24
We know use only "target chunk size" when speaking of the chunk size
that is expected to happen most of the time. This removes statistical
and mathematical inacurracies that could be troublesome for mathematical
people.
Fixes#5336
rephrase some warnings, fixes#5164
borg check --repair and borg recreate are now present in the code since rather long, so they are not experimental any more.
borg recreate might be used wrongly (e.g. accidentally excluding everything / not matching anything when recreating an archive). added some warning words in the docs, but it will not ask for confirmation any more.
borg check: there might be kinds of corruption borg check --repair can not fix and it might make things even worse while trying to fix. so this will still ask for confirmation, just with different wording.
locking: fix ExclusiveLock race condition bug, fixes#4923
- ExclusiveLock is now based on os.rename instead of os.mkdir.
- catch FileNotFoundError observed under race condition in ExclusiveLock.release()
and .kill_stale_lock()
- added TestExclusiveLock.test_race_condition() which reveals issue #4923
- updated docs
- locking: use "raise LockTimeout from None" for prettier traceback
Co-authored-by: Thomas Portmann <thomas@portmann.org>
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
The umask value is NOT transmitted from client to server any more,
so the borg client can not influence the borg server umask any more.
If one wants to have a specific umask on the server side, one needs to
use a ssh forced command in .ssh/authorized_keys file.
OTOH, as the default value is 077 (in general, for client as well as for
the server) and the server does not take the value from the client any more,
there usually should be no need to give it on the server side, IF you are
happy with the default value.
Running 'borg key import' on a keyfile repository with the BORG_KEY_FILE
environment variable set works correctly if the BORG_KEY_FILE file
already exists. However, the command crashes if the BORG_KEY_FILE file
does not exist:
$ BORG_KEY_FILE=newborgkey borg key import /home/strager/borg-backups/straglum borgkey
Local Exception
Traceback (most recent call last):
[snip]
File "[snip]/borg/crypto/key.py", line 713, in sanity_check
with open(filename, 'rb') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '[snip]/newborgkey'
Platform: Linux straglum 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.11 Python: CPython 3.7.7 msgpack: 0.5.6
PID: 15306 CWD: /home/strager/Projects/borg
sys.argv: ['[snip]/borg', 'key', 'import', '/home/strager/borg-backups/straglum', 'borgkey']
SSH_ORIGINAL_COMMAND: None
Make 'borg key import' not require the BORG_KEY_FILE file to already
exist.
This commit does not change the behavior of 'borg key import' without
BORG_KEY_FILE. This commit also does not change the behavior of 'borg
key import' on a repokey repository.
general.rst and man_intro.rst both included usage_general.rst.inc, which resulted in three Sphinx warning "WARNING: duplicate label".
To prevent this we move all sections of usage_general into own include files and add a second usage_general file without the three labels.
[DOCS] #4941 – How important is Borg config?
Add one FAQ with for $HOME/.config/borg, describing its content and how to take care of it.
Change "Default to" to "Defaults to".
Move TMPDIR to "Directories and files" section.
Add link to new FAQ for Borg config directory.
Add link to cache internals.
I used `grep -Pnr '.{100}' *.rst` to find lines >100 characters long and
reflowed them where reasonable. Line length in the docs isn't too important (it
doesn't matter once they're compiled), but it's nice not to have super-long
lines in one's editor.
Because some of the JSON blocks in frontends.rst included non-JSON
contents (user replies, etc.) the blocks didn't parse/highlight and
`make html` complained:
borg/docs/internals/frontends.rst:28: WARNING: Could not lex literal_block as "json".
There's no easy way to enable and disable highlighting just for
specific lines, but individually unsetting the highlight language per
block suppresses the warnings.
- Added "Will move with BORG_CONFIG_DIR variable unless specified." to BORG_SECURITY_DIR info.
- put BORG_SECURITY_DIR immediately below BORG_CONFIG_DIR (and moved BORG_CACHE_DIR up before them).
Reiterate that 'file cache names are absolute'
Drive home the point that relative source names does not save you from re-chunking if absolute pathnames change.
Also: mention bind mount solution.
remove WSL autodetection. if WSL still has this problem, you need to
set BORG_WORKAROUNDS=basesyncfile in the borg process environment to
work around it.
(cherry picked from commit beb948fc71)
docs about remote temp directory usage
Updated doc to cover use of temp directory on remote, in particular how to handle changing the directory used for systems such as QNAP with tiny default /tmp directories.
fixup by tw: do not mention specific brands, specific amounts
this is not QNAP specific, but may happen as well on other limited
systems. the amount of tmp files can also vary widely.
Fix various code blocks in the docs
- rst markup: put codeblock markup on separate line to make it better visible / separate it from normal text colons.
- borg help texts in archiver.py: put codeblock markup instead of colon - that way it looks like a single colon when using the cli help and also works as code block markup.
add a push-style example to borg-create docs
When I browsed the readthedocs and manpages while setting up Borg for
myself for the first time, I saw multiple hints that pushing backups via
SSH is supported (e.g. the existence of the `borg serve` command and the
`--rsh` option), but I never found a clear example. I only found
<https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls>
after being pointed to it by @ThomasWaldmann.
Therefore, an explicit example in the `borg create` documentation seems
appropriate. I put it next to the pull-style example using sshfs which
serves a similar usecase.
It was broken because there's already a different reference named
github in `docs/global.rst.inc`. Fixed by using an anonymous hyperlink
reference, per https://stackoverflow.com/a/14067756/278488
Add "SSH Configuration" section
Add "SSH Configuration" section to "borg serve" documentation, to outline ssh/sshd configuration to prevent borg serve keeping a lock on a repo in the event the ssh connection is abnormally disconnected.
In response to issues #3988, #636 and #4485 (and probably others).
