Commit Graph

1075 Commits

Author SHA1 Message Date
TW 70d97c4467 Merge pull request #180 from ThomasWaldmann/read-device
read special files as if they were regular files, update docs, closes #79
2015-09-06 21:38:31 +02:00
TW 3ab068b834 Merge pull request #181 from ThomasWaldmann/hash-collision
detect inconsistency / corruption / hash collision, closes #170
2015-09-06 21:35:53 +02:00
TW 18e428f33c Merge pull request #185 from ThomasWaldmann/fix-182
fix reaction to "no" answer at delete repo prompt, fixes #182
2015-09-06 21:33:24 +02:00
Thomas Waldmann f5069c4e81 fix reaction to "no" answer at delete repo prompt, fixes #182 2015-09-06 21:11:52 +02:00
Thomas Waldmann 817ce18bc6 fix repository arg default 2015-09-06 20:19:28 +02:00
Thomas Waldmann b3f5231bac BORG_REPO env var support
sets the default repository to use, e.g. like:

export BORG_REPO=/mnt/backup/repo
borg init
borg create ::archive
borg list
borg mount :: /mnt
fusermount -u /mnt
borg delete ::archive
2015-09-06 18:18:24 +02:00
Thomas Waldmann a912c02757 detect inconsistency / corruption / hash collision, closes #170
added a check that compares the size of the new chunk with the stored size of the
already existing chunk in storage that has the same id_hash value.
raise an exception if there is a size mismatch.

this could happen if:

- the stored size is somehow incorrect (corruption or software bug)
- we found a hash collision for the id_hash (for sha256, this is very unlikely)
2015-09-06 01:10:43 +02:00
Thomas Waldmann 0b1035746e read special files as if they were regular files, update docs, closes #79
do not use the files cache for such special files
2015-09-06 00:29:46 +02:00
Thomas Waldmann 54ccbc5ae2 chunks index resync: do all in one pass
if we do not have a cached archive index: fetch and build and merge it
if we have one: merge it
2015-08-30 15:15:15 +02:00
Thomas Waldmann 22dd925986 chunks index archive: remove all tar and compression related stuff and just use separate files in a directory
the compression was quite cpu intensive and didn't work that great anyway.
now the disk space usage is a bit higher, but it is much faster and less hard on the cpu.

disk space needs grow linearly with the amount and size of the archives, this
is a problem esp. if one has many and/or big archives (but this problem existed
before also because compression was not as effective as I believed).

the tar archive always needed a complete rebuild (and thus: decompression
and recompression) because deleting outdated archive indexes was not
possible in the tar file.

now we just have a directory chunks.archive.d and keep archive index files
there for all archives we already know.
if an archive does not exist any more in the repo, we just delete its index file.
if an archive is unknown still, we fetch the infos and build a new index file.

when merging, we avoid growing the hash table from zero, but just start
with the first archive's index as basis for merging.
2015-08-30 03:03:48 +02:00
Thomas Waldmann f7210c749f remove cpu intensive compression methods for the chunks.archive
also remove the comment about how good xz compresses - while that was true for smaller index files,
it seems to be less effective with bigger ones. maybe just an issue with compression dict size.
2015-08-29 23:42:28 +02:00
TW 17c4394896 Merge pull request #161 from RonnyPfannschmidt/setuptools-scm
replace versioneer with setuptools_scm
2015-08-29 16:46:41 +02:00
Thomas Waldmann ab545ae45d CHANGES: release 0.25.0 2015-08-29 14:10:31 +02:00
Thomas Waldmann 2332ee433e remove docs/misc/create_compression.txt
outdated - it just showed different levels of zlib compression,
but not we additionally have "lzma", "lz4" and "none" compression.
the "usage" and "internals" docs give some hints about them, too.
2015-08-29 13:35:58 +02:00
Thomas Waldmann 31e97d568b remove x bits from repository.py 2015-08-29 12:52:18 +02:00
Thomas Waldmann d2e5ce90f2 add FAQ entry about OS X commandline vs. HFS+ encoding issue, see #143.
a real fix for this is in development, but needs some time.
2015-08-29 05:41:41 +02:00
Thomas Waldmann 273c04329f update CHANGES 2015-08-29 05:02:05 +02:00
Thomas Waldmann d779057b79 fix issue with negative "all archives" size, fixes #165
This fixes a infrequent problem when (refcount * chunksize) overflowed a int32_t.
chunksize is always <= 8MiB and usually rather ~64KiB (with default chunker params).
Thus, this happened only for high refcounts and/or unusually big chunks.
2015-08-29 04:46:13 +02:00
Thomas Waldmann 1a949eab3e docs: internals: improve structure and formatting 2015-08-29 04:23:57 +02:00
Thomas Waldmann ddd0e2700b docs: improve faq and support section 2015-08-29 04:16:03 +02:00
Thomas Waldmann c823554b6b docs: usage: improved formatting, cosmetic changes 2015-08-29 04:00:22 +02:00
Thomas Waldmann e4dc482598 docs: quickstart: key modes, cosmetic changes 2015-08-29 03:39:53 +02:00
Thomas Waldmann 8a483511cb docs: theme: use a lighter green 2015-08-29 03:01:39 +02:00
Thomas Waldmann cebb61d8dd docs: add pypi, wheel, dist package based install docs 2015-08-29 02:55:09 +02:00
Thomas Waldmann 79d7f49a44 docs: split install into system-specific preparations and generic instructions 2015-08-29 01:17:19 +02:00
Thomas Waldmann ee58d4f074 docs: be more specific about Debian/Ubuntu release 2015-08-29 00:45:54 +02:00
TW acf5ddc68d Merge pull request #166 from ThomasWaldmann/fix-restore-xattrs
Fix restore xattrs
2015-08-29 00:26:38 +02:00
Thomas Waldmann 9ebc53ad77 restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
e.g.:
- setting any security.* key is expected to fail with EACCES if one is not root.
- issue #162 on our issue tracker: user was root, but due to some specific scenario
  involving docker and selinux, setting security.selinux key fails even when running as root

not sure if it is the best solution to silently ignore this, but some lines below this change
failure to do a chown is also silently ignored (happens e.g. when restoring a file not owned
by the current user as a non-root user).
2015-08-29 00:11:04 +02:00
Thomas Waldmann ea8f3bd7e7 restore_xattrs: minor cleanup / simplification
if we use {} as default for item.get(), we do not need the "if" as iteration over an empty dict won't do anything.
also fixes too deep indentation the original code had.
2015-08-28 23:22:26 +02:00
Ronny Pfannschmidt 8b6ca0d912 propperly handle borg._version using setuptools_scm 2015-08-22 15:54:40 +02:00
Ronny Pfannschmidt c7023ed12a replace versioneer with setuptools_scm 2015-08-22 14:58:06 +02:00
Thomas Waldmann 034e3477bd travis: use requirements file 2015-08-21 01:10:34 +02:00
Thomas Waldmann fdc18eb0a0 support docs: write some words about BountySource 2015-08-21 00:25:24 +02:00
Thomas Waldmann 2402a2269e update docs about fakeroot 2015-08-20 23:39:40 +02:00
Thomas Waldmann 3d974d28b3 update CHANGES 2015-08-20 23:27:24 +02:00
Thomas Waldmann 1e11e24fc4 document locking system 2015-08-20 22:37:38 +02:00
Alan Jenkins 59a44296e4 chunker - cast from size_t to off_t can now be removed
Sorry, this should really have been part of the previous commit -
it's why I noticed a problem.
2015-08-20 17:48:59 +01:00
Thomas Waldmann 0a2bd8dad5 lock roster: catch file not found in remove() method and ignore it 2015-08-20 18:40:24 +02:00
Alan Jenkins ce3e67cb96 chunker - fix 4GB files on 32-bit systems
From code inspection - effect not actually tested.
2015-08-20 17:23:50 +01:00
Alan Jenkins 7c6f3ece66 Initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0) 2015-08-20 17:23:41 +01:00
Thomas Waldmann d3d78f7ae3 call fadvise DONTNEED for the byterange we actually have read, fixes #158
avoid throwing away potential readahead data the OS might have read into the cache.
2015-08-20 05:33:51 +02:00
Per Guth 0926c2821f docs: installation: korora/fedora -> `lz4-devel` 2015-08-17 17:10:37 +02:00
Per Guth 8cf0ead693 docs: added `favicon.ico` 2015-08-17 12:58:49 +02:00
Thomas Waldmann 7f0fe83483 Merge branch 'master' of github.com:borgbackup/borg 2015-08-17 11:52:05 +02:00
Thomas Waldmann 93a89d97fa ChunkerParams: fix parameter order
the parser for the --chunker-params argument had a wrong parameter order.
fixed the order so it conforms to the help text and the docs.
also added some tests for it and a text for the ValueError exception.
2015-08-17 11:50:47 +02:00
Per Guth 1c7b5b23f4 minor change in copyright notice 2015-08-17 11:31:42 +02:00
Thomas Waldmann b180158876 generalize hashindex code for any key length
currently, we only use sha256 hashes as key, so key length is always 32.
but instead of hardcoding 32 everywhere, using key_length is just better
readable and also more flexible for the future.
2015-08-16 14:51:15 +02:00
Thomas Waldmann 608c0935e0 borg list --short, remove requirement for fakeroot, xfail a test
borg list --short just spills out the list of files / dirs - better for some tests
and also useful on the commandline for interactive use.

the tests previously needed fakeroot because in the test setup it always
made calls to mknod and chown, which require (fake)root.
now, the tests adapt to whether it detects (fake)root or not - to run the
the tests completely, you still need fakeroot, but it won't fail all the archiver
tests just due to failing test setup.

also, a test not working correctly due to fakeroot was found:
it should detect whether a read-only repo is usable, but it failed to do that
because with (fake)root, there is no "read only" (at least not via taking away
 the w permission bits).
2015-08-15 20:52:14 +02:00
Thomas Waldmann 738ed5d91b 2 small archiver testsuite fixes
environment context manager: if a env var was not present before, it should not be present afterwards

teardown: cd out of the tmpdir before deleting it
2015-08-15 17:07:09 +02:00
Thomas Waldmann e5b647fbd1 minor lrucache test fix 2015-08-15 16:15:10 +02:00