Commit Graph

3801 Commits

Author SHA1 Message Date
TW 6dab80218b Merge pull request #2344 from ThomasWaldmann/fix-size-symlink
fix symlink item fs size computation
2017-03-26 23:53:16 +02:00
Thomas Waldmann 90dd0e8eca fix symlink item fs size computation
a symlink has a 'source' attribute, so it was confused with a hardlink
slave here. see also issue #2343.

also, a symlink's fs size is defined as the length of the target path.
2017-03-26 23:11:14 +02:00
TW 4fe9d067da Merge pull request #2255 from ThomasWaldmann/async-errors
async exception processing
2017-03-26 18:15:12 +02:00
Thomas Waldmann a9088135aa RemoteRepository: shutdown with timeout 2017-03-26 17:41:41 +02:00
Thomas Waldmann 945880af47 implement async_response, add wait=True for add_chunk/chunk_decref
Before this changeset, async responses were:
- if not an error: ignored
- if an error: raised as response to the arbitrary/unrelated next command

Now, after sending async commands,  the async_response command must be used
to process outstanding responses / exceptions.

We are avoiding to pile up lots of stuff in cases of high latency, because we do NOT
first wait until ALL responses have arrived, but we just can begin to process responses.
Calls with wait=False will just return what we already have received.
Repeated calls with wait=True until None is returned will fetch all responses.

Async commands now actually could have non-exception non-None results, but
this is not used yet. None responses are still dropped.

The motivation for this is to have a clear separation between a request
blowing up because it (itself) failed and failures unrelated to that request /
to that line in the sourcecode.

also: fix processing for async repo obj deletes

exception_ignored is a special object used that is "not None" (as None is used to signal
"finished with processing async results") but also not a potential async response result value.

Also:

added wait=True to chunk_decref() and add_chunk()

this makes async processing explicit - the default is synchronous and you only
need to be careful and do extra steps for async processing if you explicitly
request async by calling with wait=False (usually for speed reasons).

to process async results, use async_response, see above.
2017-03-26 17:33:19 +02:00
enkore 713889dbfd Merge pull request #2340 from jdchristensen/auto-threshold
With --compression auto,C, only use C if lz4 achieves at least 3% compression
2017-03-26 04:59:42 +02:00
Dan Christensen 48652a65a6 With --compression auto,C, only use C if lz4 achieves at least 3% compression 2017-03-25 19:57:54 -04:00
TW f0a04a9db2 Merge pull request #2338 from ThomasWaldmann/normalize-once-only
optimize / simplify path normalization
2017-03-25 23:44:06 +01:00
Thomas Waldmann 1a376ae1f1 PatternMatcher: only normalize the path once
not N times for N patterns.
2017-03-25 23:16:05 +01:00
Thomas Waldmann 126e782998 path normalization: rather use function than decorator
less and less complex code, more flexible usage.
2017-03-25 22:49:39 +01:00
enkore 83de89d186 Merge pull request #2336 from enkore/issue/2335
Remove Python API docs
2017-03-25 16:04:57 +01:00
Marian Beermann dfdf590445 docs: typos 2017-03-25 15:42:00 +01:00
Marian Beermann 475d53d9ef docs: kill api page 2017-03-25 15:41:11 +01:00
TW e949dfcb54 Merge pull request #2332 from ThomasWaldmann/fix-auto-compression
use immutable data structure for the compression spec, fixes #2331
2017-03-25 15:24:58 +01:00
TW d81311e21d Merge pull request #2333 from fmikker/feature/faq-backing-up-root-partition
Added FAQ section about backing up root partition
2017-03-25 01:01:02 +01:00
Fredrik Mikker d01a8f54b6 Added FAQ section about backing up root partition
Using borgbackup to backup the root partition works fine, just
remember to exclude non-essential directories.

Signed-off-by: Fredrik Mikker <fredrik@mikker.se>
2017-03-24 16:49:07 +01:00
Thomas Waldmann 2414cd4df7 use immutable data structure for the compression spec, fixes #2331
the bug was compr_args.update(compr_spec), helpers.py:2168 - that mutated
the compression spec dict (and not just some local one, but the compr spec
dict parsed from the commandline args).

so a change that was intended just for 1 chunk changed the desired
compression level on the archive scope.

I refactored the stuff to use a namedtuple (which is immutable, so such
effects can not happen again).
2017-03-24 03:09:55 +01:00
TW 1077ccbc37 Merge pull request #2321 from ThomasWaldmann/revert-bigint-removal
Revert bigint removal
2017-03-21 19:04:30 +01:00
Thomas Waldmann 1b008f725c fixup: remove unneeded imports 2017-03-21 02:42:51 +01:00
Thomas Waldmann b27cc37e85 safe_timestamp: arg is always an int
the Item object already does the bigint_to_int decode when accessing .mtime/.atime/.ctime
2017-03-21 02:40:50 +01:00
Thomas Waldmann 3665cc3024 bigint conversion: add compatibility note 2017-03-21 02:27:20 +01:00
Thomas Waldmann f708183743 Revert "don't do "bigint" conversion for nanosecond mtime"
This reverts commit 8b2e7ec680.

We still need the bigint stuff for borg 1.0 compatibility.

# Conflicts:
#	src/borg/cache.py
2017-03-21 02:21:32 +01:00
Thomas Waldmann 04dba76fc9 Mostly revert "clean imports, remove unused code"
This reverts commit b7eaeee266.

We still need the bigint stuff for compatibility to borg 1.0 archives.

# Conflicts:
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/helpers.py
#	src/borg/key.py
2017-03-21 02:05:38 +01:00
TW 16e6e3d989 Merge pull request #2288 from ThomasWaldmann/patterns-style-default
support setting the pattern style default in patterns file
2017-03-17 14:09:47 +01:00
TW 10d4c97cad Merge pull request #2309 from ThomasWaldmann/fix-2304
clamp (nano)second values to unproblematic range, fixes #2304
2017-03-16 20:31:39 +01:00
Thomas Waldmann b7a17a6db7 clamp (nano)second values to unproblematic range, fixes #2304
filesystem -> clamp -> archive (create)
2017-03-16 20:31:05 +01:00
Thomas Waldmann 42371181fc support switching the pattern style default in patterns file 2017-03-16 19:37:48 +01:00
TW ae650792e7 Merge pull request #2311 from enkore/f/ratelimit-faq
faq: mention --remote-ratelimit in bandwidth limit question
2017-03-16 18:57:56 +01:00
Marian Beermann 988e452bc0 faq: mention --remote-ratelimit in bandwidth limit question 2017-03-16 15:13:03 +01:00
enkore 883a7eefb2 Archive: allocate zeros when needed (#2308)
fixes huge memory usage of mount (8 MiB × number of archives)
2017-03-15 17:08:07 +01:00
TW e3a0c4e32d Merge pull request #2302 from ThomasWaldmann/fix-caskroom-link
fix caskroom link, fixes #2299
2017-03-15 02:54:47 +01:00
Thomas Waldmann b287fba22e fix caskroom link, fixes #2299 2017-03-15 01:22:21 +01:00
Marian Beermann d1738ec315 docs/security: reiterate that RPC in Borg does no networking 2017-03-13 16:29:32 +01:00
Marian Beermann 86867e5171 docs/security: counter tracking
Copied from #2266
2017-03-13 14:50:56 +01:00
Milkey Mouse 2117861738 Securely erase config file, fixes #2257
The SaveFile code, while ensuring atomicity, did not allow for secure
erasure of the config file (containing the old encrypted key). Now it
creates a hardlink to the file, lets SaveFile do its thing, and writes
random data over the old file (via the hardlink). A secure erase is
needed because the config file can contain the old key after changing
one's password.
2017-03-12 12:18:02 +01:00
Thomas Waldmann 23f6a82f1b fix borg key/debug/benchmark crashing without subcommand, fixes #2240 2017-03-12 11:55:43 +01:00
Thomas Waldmann a842001385 fix error msg, it is --keep-within, not --within 2017-03-12 11:53:42 +01:00
Marian Beermann 00f98d8ad1 docs/development: update merge remarks 2017-03-10 23:26:10 +01:00
Milkey Mouse 55759f76e6 Address SSH batch mode in docs (fixes #2202) (#2270) 2017-03-09 21:43:23 +01:00
Marian Beermann cdb4df0885 --log-json: time property on most progress/log objects, remove is_prompt 2017-03-09 21:36:37 +01:00
Marian Beermann e07a289e34 frontends.rst: it's "null" in json 2017-03-09 21:36:37 +01:00
Marian Beermann e98b5b20df yes(): handle JSON output 2017-03-09 21:36:37 +01:00
Milkey Mouse 85a2d2fc08 Add warning about running build_usage on Python >3.4 (fixes #2123)
Python 3.6 added some new guaranteed hashing algorithms that will
show up as available in the docs even though the baseline for support
is Python 3.4; running build_usage with Python 3.4 will fix this.

# Conflicts:
#	docs/development.rst
2017-03-09 21:35:33 +01:00
Abdel-Rahman 63b5cbfc99 extract: warning RC for unextracted big extended attributes, followup (#2258)
* Set warning exit code when xattr is too big

* Warnings for more extended attributes errors (ENOTSUP, EACCES)

* Add tests for all xattr warnings
2017-03-08 17:13:42 +01:00
Florent Hemmi 1551be54be Docs: one link per distro in the installation page 2017-03-08 17:11:10 +01:00
enkore fc41c98a86 Redo key_creator, key_factory, centralise key knowledge (#2272)
* key: put key metadata (name, storage) into key classses

* keymanager: use key-declared storage types
2017-03-08 17:08:54 +01:00
TW 1f861ec78e Merge pull request #2269 from edgimar/master
docs:  improve --exclude-if-present and --keep-exclude-tags, fixes #2268
2017-03-08 16:58:16 +01:00
Mark Edgington e4391dec54 docs: improve --exclude-if-present and --keep-exclude-tags 2017-03-07 22:32:52 -05:00
TW ad0eab8b59 Merge pull request #2265 from ThomasWaldmann/test-3.6-dev
help python development by testing 3.6-dev
2017-03-07 21:02:59 +01:00
Thomas Waldmann 98e80ef2fb help python development by testing 3.6-dev 2017-03-07 05:07:51 +01:00