1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2024-12-27 02:08:54 +00:00
Commit graph

4086 commits

Author SHA1 Message Date
Marian Beermann
b2953357ed recreate: add --recompress flag, avoid weirdo use of args.compression 2017-04-04 15:11:15 +02:00
enkore
f878678b0c Merge pull request #2364 from enkore/f/autocomp
Refactor compression decision stuff
2017-04-04 00:28:03 +02:00
Marian Beermann
2ff75d58f2 remove Chunk() 2017-04-04 00:16:15 +02:00
Marian Beermann
69fb9bd403 remove --compression-from 2017-04-04 00:16:15 +02:00
Marian Beermann
929f2760dd change global compression default to lz4 as well
To be consistent with --compression defaults.
2017-04-04 00:16:15 +02:00
Marian Beermann
0847c3f9a5 Unify ComprSpec and CompressionSpec; don't instanciate Compressors right away 2017-04-04 00:16:14 +02:00
Marian Beermann
d1826cca92 Rename CompressionDecider1 -> CompressionDecider 2017-04-03 21:31:28 +02:00
Marian Beermann
0c7410104c Rename Chunk.meta[compress] => Chunk.meta[compressor] 2017-04-03 21:31:28 +02:00
Marian Beermann
88647595ac compress: docs 2017-04-03 21:31:28 +02:00
Marian Beermann
5a20fc08de key: compress(chunk) return data 2017-04-03 21:31:28 +02:00
Marian Beermann
a27f585eaa refactor CompressionDecider2 into a meta Compressor 2017-04-03 21:31:28 +02:00
Marian Beermann
bb6b4fde93 BORG_HOSTNAME_IS_UNIQUE=yes by default. 2017-04-01 21:28:41 +02:00
TW
d79da81d22 Merge pull request #2372 from RonnyPfannschmidt/travis
enhance travis setuptools_scm situation
2017-04-01 18:47:27 +02:00
Ronny Pfannschmidt
132f0006d3 enhance travis setuptools_scm situation
* add setuptools_scm to the development requirements
* print the own version at install time
* unshallow the repo and fetch all tags
2017-04-01 18:25:45 +02:00
enkore
74a4c4f78d Merge pull request #2371 from ThomasWaldmann/minor-extract-refactor
extract: remove duplicate code
2017-04-01 16:58:09 +02:00
Thomas Waldmann
bdbcbf7bb8 extract: remove duplicate code
anything at <path> gets nuked already a few lines above, if possible.
2017-04-01 16:56:21 +02:00
enkore
30f5e6efbb Merge pull request #2351 from ThomasWaldmann/refactor-extract
Refactor extract
2017-03-30 16:03:55 +02:00
Thomas Waldmann
d4e27e2952 extract: small bugfix and refactoring for parent dir creation
make_parent(path) helper to reduce code duplication.
also use it for directories although makedirs can also do it.

bugfix: also create parent dir for device files, if needed.
2017-03-28 23:22:25 +02:00
Thomas Waldmann
ceaf4a8fcf extract: small bugfix and optimization for hardlink masters
if a hardlink master is not in the to-be-extracted subset, the "x"
status was not displayed for it.

also, the matcher was called twice for matching items.
2017-03-28 22:02:54 +02:00
enkore
22b56aebdc Merge pull request #2349 from enkore/f/clarify-license-text
LICENSE: use canonical formulation
2017-03-28 15:45:14 +02:00
Marian Beermann
9d8b7ca0f5 LICENSE: use canonical formulation
("copyright holders and contributors" instead of "author")
2017-03-28 14:11:35 +02:00
enkore
d8d7c68a9a Merge pull request #2348 from ThomasWaldmann/rel110b4
Release 1.1.0b4
2017-03-28 10:33:31 +02:00
Thomas Waldmann
963949812d vagrant: increase memory for parallel testing
considering that we run pytest-xdist with 4 parallel processes,
we need a bit more RAM to avoid the OOM killer and MemoryError.

so, 1GiB for 32bit, 1.5GiB for 64bit VMs.
2017-03-27 12:38:19 +02:00
Thomas Waldmann
38860b3f53 lz4 compress: lower max. buffer size, exception handling
on the wheezy32 test machine, a test testing with corrupted data crashed
with a MemoryError when it tried to get a ~800MB large buffer.
MemoryError is now transformed to DecompressionError, so it gets handled
better.

Also, the bound for giving up is now much lower: 1GiB -> 128MiB.
2017-03-27 12:08:54 +02:00
Thomas Waldmann
85bfcd439c ran setup.py build_man 2017-03-27 01:58:19 +02:00
Thomas Waldmann
acd3da62f4 add docstring to do_benchmark_crud 2017-03-27 01:57:52 +02:00
Thomas Waldmann
251983a741 ran setup.py build_usage 2017-03-27 01:45:45 +02:00
Thomas Waldmann
5bc17148e1 patterns help: mention path full-match in intro 2017-03-27 01:45:04 +02:00
Thomas Waldmann
72abd12e3c update CHANGES with release date 2017-03-27 01:35:32 +02:00
TW
d9462d406c Merge pull request #2327 from ThomasWaldmann/update-master-changes
update CHANGES (master / 1.1.0b4)
2017-03-27 01:26:06 +02:00
Thomas Waldmann
60ae95801d update CHANGES (master / 1.1.0b4) 2017-03-27 01:25:20 +02:00
TW
f2e9e862d8 Merge pull request #2334 from ThomasWaldmann/precise-pattern
add PathFullPattern / optimization for it
2017-03-27 01:15:24 +02:00
Thomas Waldmann
cb6bfdf4d6 add docs for path full-match patterns 2017-03-27 00:31:08 +02:00
Thomas Waldmann
93feb75411 optimize PathFullPattern matching for O(1) time
For a borg create run using a patterns file with 15.000 PathFullPattern excludes
that excluded almost all files in the input data set:
- before this optimization: ~60s
- after this optimization: ~1s
2017-03-27 00:31:08 +02:00
Thomas Waldmann
ebd928795e add PathFullPattern
not really a pattern (as in potentially having any variable parts) - it just does a full,
precise match, after the usual normalizations.

the reason for adding this is mainly for later optimizations, e.g. via set membership check,
so that a lot of such PathFullPatterns can be "matched" within O(1) time.
2017-03-27 00:31:08 +02:00
TW
6dab80218b Merge pull request #2344 from ThomasWaldmann/fix-size-symlink
fix symlink item fs size computation
2017-03-26 23:53:16 +02:00
Thomas Waldmann
90dd0e8eca fix symlink item fs size computation
a symlink has a 'source' attribute, so it was confused with a hardlink
slave here. see also issue #2343.

also, a symlink's fs size is defined as the length of the target path.
2017-03-26 23:11:14 +02:00
TW
4fe9d067da Merge pull request #2255 from ThomasWaldmann/async-errors
async exception processing
2017-03-26 18:15:12 +02:00
Thomas Waldmann
a9088135aa RemoteRepository: shutdown with timeout 2017-03-26 17:41:41 +02:00
Thomas Waldmann
945880af47 implement async_response, add wait=True for add_chunk/chunk_decref
Before this changeset, async responses were:
- if not an error: ignored
- if an error: raised as response to the arbitrary/unrelated next command

Now, after sending async commands,  the async_response command must be used
to process outstanding responses / exceptions.

We are avoiding to pile up lots of stuff in cases of high latency, because we do NOT
first wait until ALL responses have arrived, but we just can begin to process responses.
Calls with wait=False will just return what we already have received.
Repeated calls with wait=True until None is returned will fetch all responses.

Async commands now actually could have non-exception non-None results, but
this is not used yet. None responses are still dropped.

The motivation for this is to have a clear separation between a request
blowing up because it (itself) failed and failures unrelated to that request /
to that line in the sourcecode.

also: fix processing for async repo obj deletes

exception_ignored is a special object used that is "not None" (as None is used to signal
"finished with processing async results") but also not a potential async response result value.

Also:

added wait=True to chunk_decref() and add_chunk()

this makes async processing explicit - the default is synchronous and you only
need to be careful and do extra steps for async processing if you explicitly
request async by calling with wait=False (usually for speed reasons).

to process async results, use async_response, see above.
2017-03-26 17:33:19 +02:00
enkore
713889dbfd Merge pull request #2340 from jdchristensen/auto-threshold
With --compression auto,C, only use C if lz4 achieves at least 3% compression
2017-03-26 04:59:42 +02:00
Dan Christensen
48652a65a6 With --compression auto,C, only use C if lz4 achieves at least 3% compression 2017-03-25 19:57:54 -04:00
TW
f0a04a9db2 Merge pull request #2338 from ThomasWaldmann/normalize-once-only
optimize / simplify path normalization
2017-03-25 23:44:06 +01:00
Thomas Waldmann
1a376ae1f1 PatternMatcher: only normalize the path once
not N times for N patterns.
2017-03-25 23:16:05 +01:00
Thomas Waldmann
126e782998 path normalization: rather use function than decorator
less and less complex code, more flexible usage.
2017-03-25 22:49:39 +01:00
enkore
83de89d186 Merge pull request #2336 from enkore/issue/2335
Remove Python API docs
2017-03-25 16:04:57 +01:00
Marian Beermann
dfdf590445 docs: typos 2017-03-25 15:42:00 +01:00
Marian Beermann
475d53d9ef docs: kill api page 2017-03-25 15:41:11 +01:00
TW
e949dfcb54 Merge pull request #2332 from ThomasWaldmann/fix-auto-compression
use immutable data structure for the compression spec, fixes #2331
2017-03-25 15:24:58 +01:00
TW
d81311e21d Merge pull request #2333 from fmikker/feature/faq-backing-up-root-partition
Added FAQ section about backing up root partition
2017-03-25 01:01:02 +01:00