mirror
/
borg
mirror of https://github.com/borgbackup/borg.git
1
0
Fork 0
Code Issues Releases Wiki Activity
582 Commits 9 Branches 137 Tags 57 MiB
Commit Graph

582 Commits

Author SHA1 Message Date
Thomas Waldmann ee80db4eb2 add commandline argument --sparse 
default is to not create sparse files.

if you give --sparse, it will create a hole in the sparse output file when a all-zero chunk is extracted.
 2015-04-17 22:28:40 +02:00
Thomas Waldmann a2bf2aea22 simple sparse file support, made chunk buffer size flexible 
Implemented sparse file support to remove this blocker for people backing up lots of
huge sparse files (like VM images). Attic could not support this use case yet as it would
have restored all files to their fully expanded size, possibly running out of disk space if
the total expanded size would be bigger than the available space.

Please note that this is a very simple implementation of sparse file support - at backup time,
it does not do anything special (it just reads all these zero bytes, chunks, compresses and
encrypts them as usual). At restore time, it detects chunks that are completely filled with zeros
and does a seek on the output file rather than a normal data write, so it creates a hole in
a sparse file. The chunk size for these all-zero chunks is currently 10MiB, so it'll create holes
of multiples of that size (depends also a bit on fs block size, alignment, previously written data).

Special cases like sparse files starting and/or ending with a hole are supported.

Please note that it will currently always create sparse files at restore time if it detects all-zero
chunks.

Also improved:
I needed a constant for the max. chunk size, so I introduced CHUNK_MAX (see also
existing CHUNK_MIN) for the maximum chunk size (which is the same as the chunk
buffer size).

Attic still always uses 10MiB chunk buffer size now, but it could be changed now more easily.
 2015-04-15 16:29:18 +02:00
Jonas Borgström bbc8886bfe Updated CHANGES 
Closes #163.
 2015-04-13 23:29:29 +02:00
Jonas Borgström 27012a50e5 Merge remote-tracking branch 'adept/patch-1' 2015-04-13 23:28:28 +02:00
Jonas Borgström 70af748363 Added Thomas Waldmann to AUTHORS and updated CHANGES 2015-04-13 23:12:30 +02:00
Jonas Borgström 8227e4788a Merge remote-tracking branch 'tw/fix_pipe' 2015-04-13 23:08:05 +02:00
Jonas Borgström 20026e58aa Fix incorrect behaviour with two character directory names. 
Independetely found and fixed by both Thomas Waldemann and
Cam Hutchison. Closes #265 and #268.
 2015-04-13 22:44:14 +02:00
Jonas Borgström 7198929bae cache: Warn user before accessing relocated repositories 
This also closes #225
 2015-04-13 22:42:32 +02:00
Jonas Borgström 78f9ad1fab Require approval before accessing previously unknown unencrypted repositories 
Closes #271
 2015-04-06 23:07:10 +02:00
Dmitry Astapov 6be2734b7e Do not create items_buffer and chunker unless we are creating new archive 
Listing repositories with lots of archives on low-memory systems would cause attic to run out of memory due to items_buffer and chunker being created for each visited archive.

See https://github.com/jborg/attic/issues/163
 2015-03-18 12:42:28 +00:00
Thomas Waldmann cbbe0fcc9b serve: use os.write(stdout_fd, ...), fixes #233 
this way, serve() is more consistent with the other code, which always uses os.read/write (not sys.std*.buffer.read/write).

also: reduce code duplication a bit.
 2015-03-14 19:45:01 +01:00
Jonas Borgström 4ab4ecc7af hashindex: Fix issue with file sizes larger than 2GB. 
See: http://librelist.com/browser//attic/2015/1/6/does-attic-have-an-upper-data-size-limit/
 2015-01-06 22:28:04 +01:00
Jonas Borgström df6324b725 Include missing pyx files in dist file. 
Closes #168
 2015-01-04 22:25:11 +01:00
Jonas Borgström 77bd65aa03 Merge pull request #165 from brodul/docs 
Fix documentation 'snapshot' to 'checkpoint'
 2015-01-03 23:14:13 +01:00
Jonas Borgström b5a9085717 Fix Python 3.2 compatibility issue with noatime open(). 
Closes #164.
 2015-01-03 23:12:47 +01:00
Andraz Brodnik 6d64a1c76b Fix documentation 'snapshot' to 'checkpoint' 
In the code the string `checkpoint` is appended.
./attic/archive.py:137
 2014-12-25 13:23:30 +01:00
Jonas Borgström f342621dff Preparing release 2014-12-17 23:19:22 +01:00
Jonas Borgström add59e4ab3 Prepare for future payload format change 2014-12-17 21:55:33 +01:00
Jonas Borgström e9c27e8e6d Added missing changelog entry 2014-12-14 14:30:41 +01:00
Jonas Borgström 18641ae687 xattr: Fix issue with empty (0 bytes) xattr values. 
Closes #106
 2014-12-14 14:28:33 +01:00
Jonas Borgström 53f6738090 Merge pull request #111 from c4rlo/patch-1 
fix typo in comment
 2014-12-08 21:48:36 +01:00
Jonas Borgström 5d96434abd Merge pull request #113 from c4rlo/noatime 
open files with O_NOATIME if possible
 2014-12-08 21:48:04 +01:00
Jonas Borgström 7e15abd29c hashindex: Fix issue with indicies larger than 2GB 2014-11-22 14:41:24 +01:00
Jonas Borgström 5c699b30a7 Merge pull request #129 from kannes/master 
fixing two typos
 2014-11-20 21:29:29 +01:00
kannes aee2983c6e fixing two typos 2014-11-19 13:09:53 +01:00
Jonas Borgström ca177aba60 Merge pull request #126 from dfries/master 
limit create file permissions
 2014-11-12 20:01:43 +01:00
David Fries f22b62c1a4 limit create file permissions 
Be safe by default, create files so that other users can't read them,
at least until the original permissions are set.
 2014-10-31 20:02:12 -05:00
Jonas Borgström 678fdd1d42 Fix "Number of files" output for attic info. 
Closes #124
 2014-10-27 22:00:56 +01:00
Jonas Borgström 547fa0a590 Speed up initial repository access 
Use os.listdir() instead of os.walk() to avoid unnecessary
stat() calls.

Closes #119
 2014-10-10 21:01:55 +02:00
Jonas Borgström 7b9d129bcd Remove dead code 2014-10-09 20:43:58 +02:00
Jonas Borgström 5a69f58823 Introduce a separate exception for missing repository objects 2014-10-09 20:41:47 +02:00
Carlo Teubner f4804c07ca open files with O_NOATIME if possible 2014-09-12 22:22:08 +01:00
c4rlo ec73e45ae2 fix typo in comment 2014-09-07 13:54:18 +01:00
Jonas Borgström 21e03af56b Fix minor location parser bug. 
Make sure the entire input is parsed.
 2014-08-08 23:34:27 +02:00
Jonas Borgström 9f64e39d9f Reuse chunker buffer between files. 2014-08-03 15:04:41 +02:00
Jonas Borgström 193fb1fcd5 Added support for stripping leading path segments 
closes #95
 2014-08-02 22:15:21 +02:00
Jonas Borgström 8f4e0f7506 acl: Added workaround for old Linux systems 
Really old Linux systems do not have extended_file_no_follow()
 2014-08-01 15:50:18 +02:00
Jonas Borgström 0741aef880 Add MacPorts' path to the default openssl search path 
Closes #101
 2014-08-01 14:48:49 +02:00
Jonas Borgström 1b0e21b95a Fixed typo 2014-07-10 15:44:29 +02:00
Jonas Borgström 2f72b9f960 HashIndex: Switch to a non-mmap based implementation 
This eliminates unnecessary IO on low memory systems
 2014-07-10 15:32:29 +02:00
Jonas Borgström 4b2dae29d4 Fix incorrect detect of follow_symlinks support 
This only affects system with Python 3.3+ and a really old libc
 2014-07-01 21:28:07 +02:00
Jonas Borgström 7a950be929 Preparing release 2014-06-29 14:28:02 +02:00
Jonas Borgström 7ce3290fb2 docs: Added link to prebuilt Linux binaries. 2014-06-29 14:24:45 +02:00
Jonas Borgström 15065dbaa4 Fix sporadic "Resource temporarily unavailable" error 
Issue reported on the mailing list while backing up to a remote
repository on OS X over a slow uplink.
 2014-06-13 20:07:01 +02:00
Jonas Borgström 95162ce1f8 buzhash: One more test case 
Test with more than 31 bytes to make sure our BARREL_SHIFT macro
works correctly.
 2014-06-05 15:11:46 +02:00
Jonas Borgström df85e72d3c Fix file cache save bug 2014-06-03 23:10:52 +02:00
Jonas Borgström 243481f6da Fix test suite breakage 2014-06-03 22:00:34 +02:00
Jonas Borgström 80a2a4113f Revert "Reduce memory usage when backing up many small files" 
The memory usage should be reduced a fair bit by #90 so this
might not be needed anymore. Or at least not enabled by default
 2014-06-03 21:39:26 +02:00
Jonas Borgström 7f9fe03917 cache: Reduce file cache memory usage (#90) 
The memory usage is reduced at the expence of extra msgpack
packing/unpacking. Hopefully this is a good tradeoff.

Ideally the entire cache should be replaced with a radix tree
but that will have to wait a bit.
 2014-06-03 21:19:02 +02:00
Jonas Borgström 64cd6632a1 Avoid deadlock problems 
Explicitly use write locks, instead of read locks (that are later
upgraded) unless we know we will never modify the repository
 2014-05-31 15:39:51 +02:00