make_parent(path) helper to reduce code duplication.
also use it for directories although makedirs can also do it.
bugfix: also create parent dir for device files, if needed.
if a hardlink master is not in the to-be-extracted subset, the "x"
status was not displayed for it.
also, the matcher was called twice for matching items.
considering that we run pytest-xdist with 4 parallel processes,
we need a bit more RAM to avoid the OOM killer and MemoryError.
so, 1GiB for 32bit, 1.5GiB for 64bit VMs.
on the wheezy32 test machine, a test testing with corrupted data crashed
with a MemoryError when it tried to get a ~800MB large buffer.
MemoryError is now transformed to DecompressionError, so it gets handled
better.
Also, the bound for giving up is now much lower: 1GiB -> 128MiB.
For a borg create run using a patterns file with 15.000 PathFullPattern excludes
that excluded almost all files in the input data set:
- before this optimization: ~60s
- after this optimization: ~1s
not really a pattern (as in potentially having any variable parts) - it just does a full,
precise match, after the usual normalizations.
the reason for adding this is mainly for later optimizations, e.g. via set membership check,
so that a lot of such PathFullPatterns can be "matched" within O(1) time.
a symlink has a 'source' attribute, so it was confused with a hardlink
slave here. see also issue #2343.
also, a symlink's fs size is defined as the length of the target path.
Before this changeset, async responses were:
- if not an error: ignored
- if an error: raised as response to the arbitrary/unrelated next command
Now, after sending async commands, the async_response command must be used
to process outstanding responses / exceptions.
We are avoiding to pile up lots of stuff in cases of high latency, because we do NOT
first wait until ALL responses have arrived, but we just can begin to process responses.
Calls with wait=False will just return what we already have received.
Repeated calls with wait=True until None is returned will fetch all responses.
Async commands now actually could have non-exception non-None results, but
this is not used yet. None responses are still dropped.
The motivation for this is to have a clear separation between a request
blowing up because it (itself) failed and failures unrelated to that request /
to that line in the sourcecode.
also: fix processing for async repo obj deletes
exception_ignored is a special object used that is "not None" (as None is used to signal
"finished with processing async results") but also not a potential async response result value.
Also:
added wait=True to chunk_decref() and add_chunk()
this makes async processing explicit - the default is synchronous and you only
need to be careful and do extra steps for async processing if you explicitly
request async by calling with wait=False (usually for speed reasons).
to process async results, use async_response, see above.
