- add DecidingCompressor class providing generic decide(),
decide_compress() and compress() suited to work using a typical
private _decide() implementation that eventually generates compressed
data as a side-effect of the actual decision process
- the new decide_compress() method returns both the actually used
compressor (which may be NONE) and the compressed data to allow
the auto compressor to use that information instead of having to
figure out whether the LZ4 compressor decided to use NONE or LZ4
compression
- derive LZ4, LZMA and ZSTD classes from DecidingCompressor and
have them fall back to NONE if the compressed data turns out bigger
than the original data
- leave ZLIB as is, as it does not use an ID header and compatibility
being the only reason to use it instead of above algorithms anyway
- adapt testsuite to cope with incompressible test data not being
compressed
- add tests to verify that incompressible data is not compressed to
larger size compressed data
Fix FUSE low linear read speed on large files
This patch is a quick fix for an inefficient seek algorithm used in read() function:
1) On each read call chunk list for given file is regenerated.
2) Finding needed chunk takes a lot of time for large offsets.
For 64 Gb files read speed drops to 16 MB/s, for 128 Gb - to 8 MB/s, and so on.
After applying this patch read speed becomes 270 MB/s, and does not depend on read position inside file.
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
general.rst and man_intro.rst both included usage_general.rst.inc, which resulted in three Sphinx warning "WARNING: duplicate label".
To prevent this we move all sections of usage_general into own include files and add a second usage_general file without the three labels.
[DOCS] #4941 – How important is Borg config?
Add one FAQ with for $HOME/.config/borg, describing its content and how to take care of it.
Change "Default to" to "Defaults to".
Move TMPDIR to "Directories and files" section.
Add link to new FAQ for Borg config directory.
Add link to cache internals.
I used `grep -Pnr '.{100}' *.rst` to find lines >100 characters long and
reflowed them where reasonable. Line length in the docs isn't too important (it
doesn't matter once they're compiled), but it's nice not to have super-long
lines in one's editor.
Because some of the JSON blocks in frontends.rst included non-JSON
contents (user replies, etc.) the blocks didn't parse/highlight and
`make html` complained:
borg/docs/internals/frontends.rst:28: WARNING: Could not lex literal_block as "json".
There's no easy way to enable and disable highlighting just for
specific lines, but individually unsetting the highlight language per
block suppresses the warnings.
if the rebuilt index size matched the on-disk index size AND there
was a difference in e.g. 1 key, the old code only output the key/value
for one index, but not what is present in the other index.
we already had better code in the branch for different index sizes,
so just use that for both cases.
additionally we tell when the index size matches (new) because we
also tell if there is a mismatch.
