mirror
/
borg
mirror of https://github.com/borgbackup/borg.git
1
0
Fork 0
Code Issues Releases Wiki Activity
3,801 Commits 9 Branches 137 Tags 57 MiB
Commit Graph

3801 Commits

Author SHA1 Message Date
Thomas Waldmann c8ec698d73 Location: accept //servername/share/path 2017-02-24 04:22:12 +01:00
Thomas Waldmann 4d81b186ec borg delete --force --force to delete severely corrupted archives, fixes #1975 2017-02-24 01:28:42 +01:00
Abogical e487b8404c Add testsuite to test handling of too big xattr 2017-02-23 23:42:56 +02:00
Abogical 4c9bc96fb7 Print a warning for too big extended attributes 2017-02-23 23:42:56 +02:00
Thomas Waldmann 7da0a9c982 borg extract: check file size consistency 2017-02-23 21:46:15 +01:00
Thomas Waldmann adc4da280d borg check: check file size consistency 2017-02-23 21:46:15 +01:00
Marian Beermann 4f1db82f6d info <archive>: use Archive.info() for both JSON and human display 2017-02-23 21:39:56 +01:00
Thomas Waldmann 50068c596d rename Item.file_size -> get_size 
file_size is misleading here because one thinks of on-disk file size,
but for compressed=True, there is no such on-disk file.
 2017-02-23 21:27:05 +01:00
Thomas Waldmann 97bb1b7d9a deduplicate / refactor item (c)size code 2017-02-23 21:27:05 +01:00
Thomas Waldmann 0021052dbd reduce code duplication 2017-02-23 21:24:37 +01:00
Thomas Waldmann ae6742fb34 fuse: use precomputed size from Item 2017-02-23 21:24:37 +01:00
Thomas Waldmann fe8e14cb2c fuse: get rid of chunk accounting 
the chunk accounting code tried to reflect repo space usage via the st_blocks of the files.
so, a specific chunk that was shared between multiple files [inodes] was only accounted for one specific file.
thus, the overall "du" of everything in the fuse mounted repo was maybe correctly reflecting the repo space usage,
but the decision which file has the chunk (the space) was kind of arbitrary and not really useful.

otoh, a simple fuse getattr() was rather expensive due to this as it needed to iterate over the chunks list
to compute the st_blocks value. also it needed quite some memory for the accounting.

thus, st_blocks is now just ceil(size / blocksize).

also: fixed bug that st_blocks was a floating point value previously.

also: preparing for further optimization of size computation (see next cs)
 2017-02-23 21:24:37 +01:00
Thomas Waldmann a52b54dc3c archived file items: add size metadata 
if an item has a chunk list, pre-compute the total size and store it into "size" metadata entry.

this speeds up access to item size (e.g. for regular files) and could also be used to verify the validity of the chunks list.

note about hardlinks: size is only stored for hardlink masters (only they have an own chunk list)
 2017-02-23 21:24:37 +01:00
Marian Beermann f3c7e7cd36 RemoteRepository: account rx/tx bytes 2017-02-23 16:50:52 +01:00
Marian Beermann 8cdf192511 list: add "name" key for consistency with info cmd 2017-02-23 15:23:27 +01:00
Marian Beermann 56563a4392 move JSON generation and utilities to helpers 2017-02-23 15:23:27 +01:00
enkore 4862efe718 Merge pull request #2194 from jdchristensen/master 
In api decorator, pass wait argument to RemoteRepository.call; fixes …
 2017-02-23 14:50:33 +01:00
Marian Beermann 1f8c0929bf list: --json for archive contents listing 2017-02-23 14:28:30 +01:00
Marian Beermann 25781f53d4 list: --json for archive listing 2017-02-23 14:28:15 +01:00
Marian Beermann 6180f5055c info: --json for archives 2017-02-23 14:28:15 +01:00
Marian Beermann 2ab5d0f213 use custom JSON encoder for repr'ing Borg objects consistently 2017-02-23 14:27:52 +01:00
Marian Beermann 7cbade2f8c create: add --json option 2017-02-23 12:00:21 +01:00
Marian Beermann cc26bdf810 info: add --json option 2017-02-23 11:55:14 +01:00
enkore e3a8ccdf3b Merge pull request #2197 from enkore/f/info-limits 
info: show utilization of maximum archive size
 2017-02-23 09:18:04 +01:00
Marian Beermann b0e4f13fba set MAX_DATA_SIZE = 20971479 bytes in solid stone 2017-02-23 00:34:40 +01:00
Marian Beermann 69f7810658 info: show utilization of maximum archive size 
See #1452

This is 100 % accurate.

Also increases maximum data size by ~41 bytes. Not 100 % side-effect free;
if you manage to exactly land in that area then older Borg would not read
it. OTOH it gives us a nice round number there.
 2017-02-22 23:47:21 +01:00
Dan Christensen 20a5282a4c In api decorator, pass wait argument to RemoteRepository.call; fixes #2185 2017-02-22 16:26:08 -05:00
Marian Beermann 19d50cff76 docs: internals: move toctree to after the introduction text 
this only changes the location on the page, nothing about how the TOC
is arranged.
 2017-02-22 15:56:27 +01:00
Marian Beermann 1652525858 docs: clarify metadata kind, manifest ops 2017-02-22 15:29:33 +01:00
Marian Beermann ecdf6ba25f docs: key enc: correct / clarify some stuff, link to internals/security 2017-02-22 00:21:14 +01:00
Marian Beermann c03a7ad844 docs: datas: enc: 1.1.x mas different MACs 2017-02-22 00:21:14 +01:00
Marian Beermann 6b21d63086 docs: datas: enc: correct factual error -- no nonce involved there. 2017-02-22 00:21:14 +01:00
Marian Beermann e5bbba573a docs: make internals.rst an index page 
Subsections:

- Security
- Data structures and file formats
 2017-02-22 00:21:14 +01:00
Marian Beermann 63f17087c8 docs: edited internals section a bit 2017-02-22 00:13:46 +01:00
TW 268d74bb43 Merge pull request #2181 from ThomasWaldmann/fix-2180 
archive check: detect and fix missing replacement chunks, fixes #2180
 2017-02-21 21:57:23 +01:00
enkore cd1df96ef5 Merge pull request #2171 from ThomasWaldmann/limit-unpacker 
borg rpc: use limited msgpack.Unpacker
 2017-02-21 21:53:09 +01:00
TW 21884763ac Merge pull request #2183 from ThomasWaldmann/doc-env 
document BORG_HOSTNAME_IS_UNIQUE, fixes #2087
 2017-02-21 21:38:31 +01:00
enkore 6dd5f0fbec Merge pull request #2187 from ThomasWaldmann/fix-hashindex-contains-bigendian 
fix ChunkIndex.__contains__ assertion  for big-endian archs
 2017-02-20 09:30:05 +01:00
Thomas Waldmann 8d7dfe739f fix ChunkIndex.__contains__ assertion for big-endian archs 
also: add some missing assertion messages

severity:

- no issue on little-endian platforms (== most, including x86/x64)
- harmless even on big-endian as long as refcount is below 0xfffbffff,
  which is very likely always the case in practice anyway.
 2017-02-20 07:38:55 +01:00
Thomas Waldmann 4e08bbcf0b document BORG_HOSTNAME_IS_UNIQUE, fixes #2087 2017-02-19 05:12:00 +01:00
Thomas Waldmann b82f648875 archive check: detect and fix missing all-zero replacement chunks, fixes #2180 2017-02-19 03:05:41 +01:00
enkore 63d4cf4c82 Merge pull request #2172 from Abogical/master 
Add dsize and dcsize keys, fixes #2164
 2017-02-18 18:24:52 +01:00
Abogical 38e4817b48 Correct calculation of unique chunks 2017-02-18 12:37:28 +02:00
Abogical cd3cbee962 Refactor unique chunks summing 2017-02-18 12:36:08 +02:00
Thomas Waldmann 6a25b6bdfa update docs about limited msgpack Unpacker for RPC code 2017-02-18 07:15:53 +01:00
Abogical 31f3ddf503 Join the hall of fame 2017-02-17 19:12:01 +02:00
Abogical 6ed0746934 Count non-unique chunks deduplicated sizes 2017-02-17 17:33:04 +02:00
Abogical 59571115a1 Add tests for dsize and dcsize 2017-02-17 15:26:14 +02:00
Abogical 5aa74abedf Add dsize and dcsize keys 
These keys shows the amount of deduplicated size and compressed
size of each file in the archive.
 2017-02-17 14:28:39 +02:00
Thomas Waldmann b05893e723 borg rpc: use limited msgpack.Unpacker, fixes #2139 
we do not trust the remote, so we are careful unpacking its responses.

the remote could return manipulated msgpack data that announces e.g.
a huge array or map or string. the local would then need to allocate huge
amounts of RAM in expectation of that data (no matter whether really
that much is coming or not).

by using limits in the Unpacker, a ValueError will be raised if unexpected
amounts of data shall get unpacked. memory DoS will be avoided.
 2017-02-17 05:44:48 +01:00