Thomas Waldmann
ee80db4eb2
add commandline argument --sparse
...
default is to not create sparse files.
if you give --sparse, it will create a hole in the sparse output file when a all-zero chunk is extracted.
2015-04-17 22:28:40 +02:00
Thomas Waldmann
a2bf2aea22
simple sparse file support, made chunk buffer size flexible
...
Implemented sparse file support to remove this blocker for people backing up lots of
huge sparse files (like VM images). Attic could not support this use case yet as it would
have restored all files to their fully expanded size, possibly running out of disk space if
the total expanded size would be bigger than the available space.
Please note that this is a very simple implementation of sparse file support - at backup time,
it does not do anything special (it just reads all these zero bytes, chunks, compresses and
encrypts them as usual). At restore time, it detects chunks that are completely filled with zeros
and does a seek on the output file rather than a normal data write, so it creates a hole in
a sparse file. The chunk size for these all-zero chunks is currently 10MiB, so it'll create holes
of multiples of that size (depends also a bit on fs block size, alignment, previously written data).
Special cases like sparse files starting and/or ending with a hole are supported.
Please note that it will currently always create sparse files at restore time if it detects all-zero
chunks.
Also improved:
I needed a constant for the max. chunk size, so I introduced CHUNK_MAX (see also
existing CHUNK_MIN) for the maximum chunk size (which is the same as the chunk
buffer size).
Attic still always uses 10MiB chunk buffer size now, but it could be changed now more easily.
2015-04-15 16:29:18 +02:00
Jonas Borgström
bbc8886bfe
Updated CHANGES
...
Closes #163 .
2015-04-13 23:29:29 +02:00
Jonas Borgström
27012a50e5
Merge remote-tracking branch 'adept/patch-1'
2015-04-13 23:28:28 +02:00
Jonas Borgström
70af748363
Added Thomas Waldmann to AUTHORS and updated CHANGES
2015-04-13 23:12:30 +02:00
Jonas Borgström
8227e4788a
Merge remote-tracking branch 'tw/fix_pipe'
2015-04-13 23:08:05 +02:00
Jonas Borgström
20026e58aa
Fix incorrect behaviour with two character directory names.
...
Independetely found and fixed by both Thomas Waldemann and
Cam Hutchison. Closes #265 and #268 .
2015-04-13 22:44:14 +02:00
Jonas Borgström
7198929bae
cache: Warn user before accessing relocated repositories
...
This also closes #225
2015-04-13 22:42:32 +02:00
Jonas Borgström
78f9ad1fab
Require approval before accessing previously unknown unencrypted repositories
...
Closes #271
2015-04-06 23:07:10 +02:00
Dmitry Astapov
6be2734b7e
Do not create items_buffer and chunker unless we are creating new archive
...
Listing repositories with lots of archives on low-memory systems would cause attic to run out of memory due to items_buffer and chunker being created for each visited archive.
See https://github.com/jborg/attic/issues/163
2015-03-18 12:42:28 +00:00
Thomas Waldmann
cbbe0fcc9b
serve: use os.write(stdout_fd, ...), fixes #233
...
this way, serve() is more consistent with the other code, which always uses os.read/write (not sys.std*.buffer.read/write).
also: reduce code duplication a bit.
2015-03-14 19:45:01 +01:00
Jonas Borgström
4ab4ecc7af
hashindex: Fix issue with file sizes larger than 2GB.
...
See: http://librelist.com/browser//attic/2015/1/6/does-attic-have-an-upper-data-size-limit/
2015-01-06 22:28:04 +01:00
Jonas Borgström
df6324b725
Include missing pyx files in dist file.
...
Closes #168
2015-01-04 22:25:11 +01:00
Jonas Borgström
77bd65aa03
Merge pull request #165 from brodul/docs
...
Fix documentation 'snapshot' to 'checkpoint'
2015-01-03 23:14:13 +01:00
Jonas Borgström
b5a9085717
Fix Python 3.2 compatibility issue with noatime open().
...
Closes #164 .
2015-01-03 23:12:47 +01:00
Andraz Brodnik
6d64a1c76b
Fix documentation 'snapshot' to 'checkpoint'
...
In the code the string `checkpoint` is appended.
./attic/archive.py:137
2014-12-25 13:23:30 +01:00
Jonas Borgström
f342621dff
Preparing release
2014-12-17 23:19:22 +01:00
Jonas Borgström
add59e4ab3
Prepare for future payload format change
2014-12-17 21:55:33 +01:00
Jonas Borgström
e9c27e8e6d
Added missing changelog entry
2014-12-14 14:30:41 +01:00
Jonas Borgström
18641ae687
xattr: Fix issue with empty (0 bytes) xattr values.
...
Closes #106
2014-12-14 14:28:33 +01:00
Jonas Borgström
53f6738090
Merge pull request #111 from c4rlo/patch-1
...
fix typo in comment
2014-12-08 21:48:36 +01:00
Jonas Borgström
5d96434abd
Merge pull request #113 from c4rlo/noatime
...
open files with O_NOATIME if possible
2014-12-08 21:48:04 +01:00
Jonas Borgström
7e15abd29c
hashindex: Fix issue with indicies larger than 2GB
2014-11-22 14:41:24 +01:00
Jonas Borgström
5c699b30a7
Merge pull request #129 from kannes/master
...
fixing two typos
2014-11-20 21:29:29 +01:00
kannes
aee2983c6e
fixing two typos
2014-11-19 13:09:53 +01:00
Jonas Borgström
ca177aba60
Merge pull request #126 from dfries/master
...
limit create file permissions
2014-11-12 20:01:43 +01:00
David Fries
f22b62c1a4
limit create file permissions
...
Be safe by default, create files so that other users can't read them,
at least until the original permissions are set.
2014-10-31 20:02:12 -05:00
Jonas Borgström
678fdd1d42
Fix "Number of files" output for attic info.
...
Closes #124
2014-10-27 22:00:56 +01:00
Jonas Borgström
547fa0a590
Speed up initial repository access
...
Use os.listdir() instead of os.walk() to avoid unnecessary
stat() calls.
Closes #119
2014-10-10 21:01:55 +02:00
Jonas Borgström
7b9d129bcd
Remove dead code
2014-10-09 20:43:58 +02:00
Jonas Borgström
5a69f58823
Introduce a separate exception for missing repository objects
2014-10-09 20:41:47 +02:00
Carlo Teubner
f4804c07ca
open files with O_NOATIME if possible
2014-09-12 22:22:08 +01:00
c4rlo
ec73e45ae2
fix typo in comment
2014-09-07 13:54:18 +01:00
Jonas Borgström
21e03af56b
Fix minor location parser bug.
...
Make sure the entire input is parsed.
2014-08-08 23:34:27 +02:00
Jonas Borgström
9f64e39d9f
Reuse chunker buffer between files.
2014-08-03 15:04:41 +02:00
Jonas Borgström
193fb1fcd5
Added support for stripping leading path segments
...
closes #95
2014-08-02 22:15:21 +02:00
Jonas Borgström
8f4e0f7506
acl: Added workaround for old Linux systems
...
Really old Linux systems do not have extended_file_no_follow()
2014-08-01 15:50:18 +02:00
Jonas Borgström
0741aef880
Add MacPorts' path to the default openssl search path
...
Closes #101
2014-08-01 14:48:49 +02:00
Jonas Borgström
1b0e21b95a
Fixed typo
2014-07-10 15:44:29 +02:00
Jonas Borgström
2f72b9f960
HashIndex: Switch to a non-mmap based implementation
...
This eliminates unnecessary IO on low memory systems
2014-07-10 15:32:29 +02:00
Jonas Borgström
4b2dae29d4
Fix incorrect detect of follow_symlinks support
...
This only affects system with Python 3.3+ and a really old libc
2014-07-01 21:28:07 +02:00
Jonas Borgström
7a950be929
Preparing release
2014-06-29 14:28:02 +02:00
Jonas Borgström
7ce3290fb2
docs: Added link to prebuilt Linux binaries.
2014-06-29 14:24:45 +02:00
Jonas Borgström
15065dbaa4
Fix sporadic "Resource temporarily unavailable" error
...
Issue reported on the mailing list while backing up to a remote
repository on OS X over a slow uplink.
2014-06-13 20:07:01 +02:00
Jonas Borgström
95162ce1f8
buzhash: One more test case
...
Test with more than 31 bytes to make sure our BARREL_SHIFT macro
works correctly.
2014-06-05 15:11:46 +02:00
Jonas Borgström
df85e72d3c
Fix file cache save bug
2014-06-03 23:10:52 +02:00
Jonas Borgström
243481f6da
Fix test suite breakage
2014-06-03 22:00:34 +02:00
Jonas Borgström
80a2a4113f
Revert "Reduce memory usage when backing up many small files"
...
The memory usage should be reduced a fair bit by #90 so this
might not be needed anymore. Or at least not enabled by default
2014-06-03 21:39:26 +02:00
Jonas Borgström
7f9fe03917
cache: Reduce file cache memory usage ( #90 )
...
The memory usage is reduced at the expence of extra msgpack
packing/unpacking. Hopefully this is a good tradeoff.
Ideally the entire cache should be replaced with a radix tree
but that will have to wait a bit.
2014-06-03 21:19:02 +02:00
Jonas Borgström
64cd6632a1
Avoid deadlock problems
...
Explicitly use write locks, instead of read locks (that are later
upgraded) unless we know we will never modify the repository
2014-05-31 15:39:51 +02:00
