822aefc7b0
re-add dsize placeholder
2022-06-12 17:15:13 +02:00
49adb77157
calc_stats: deduplicated size now, was deduplicated csize
...
also: remove pre12_meta cache
2022-06-12 17:15:13 +02:00
19dfbe5c5c
compute the deduplicated size before compression
...
so we do not need csize for it.
2022-06-12 17:15:13 +02:00
1fd571a4d0
fix comments
2022-06-12 17:15:13 +02:00
2c1f7951c4
remove csize from ChunkIndexEntry
2022-06-12 17:15:13 +02:00
b82a39c3b3
remove csize from stats_against()
2022-06-12 15:48:33 +02:00
0211948cac
remove csize from summarize return tuple
2022-06-12 15:48:33 +02:00
b726aa5665
remove csize support from get_size
2022-06-12 15:48:33 +02:00
ace5957524
remove csize from item.chunks elements
2022-06-12 15:48:33 +02:00
b9f9623a6d
prepare to remove csize (set it to 0 for now)
2022-06-12 15:48:33 +02:00
2c9be35886
Merge pull request #6704 from ThomasWaldmann/msgpack-str-bytes-cleanup
...
borg2: cleanup msgpack related str/bytes mess
2022-06-09 18:18:13 +02:00
d4ee968b07
use borg 2.0 to refer to this, not 1.3
...
also, some type conversions are now done in update_internal once,
not in the decode methods of the classes in item.pyx.
2022-06-09 18:13:40 +02:00
08228fbd32
Item: remove unused hardlink_masters param
2022-06-09 17:57:28 +02:00
421d4bdfb0
docs: fix bytes -> str in data-structures docs
2022-06-09 17:57:28 +02:00
58009f6773
Key: fix once, remove decode=...
2022-06-09 17:57:28 +02:00
ed22f721f3
EncryptedKey: fix once, remove decode=...
2022-06-09 17:57:28 +02:00
f2b085787b
Item: disallow None value for .user/group/chunks/chunks_healthy
...
If we do not know the value, just do not have that key/value pair in the item.
2022-06-09 17:57:28 +02:00
64cc16a9f4
Item: fix xattr processing
...
Item.xattrs is now always a StableDict mapping bytes keys -> bytes values.
The special casing of empty values (b'') getting replaced by None was removed.
2022-06-09 17:57:28 +02:00
9d684120a2
Item: assert type also in property getter
...
also: fixed Item.xattrs to be StableDict (not just a dict, as the
msgpack unpacker gives us)
2022-06-09 17:57:28 +02:00
7b138cc710
Item: convert timestamps once, get rid of bigint code
2022-06-09 17:57:28 +02:00
8e58525fc6
Item: remove some decode= params
...
update_internal() makes sure they have the desired type already.
2022-06-09 17:57:28 +02:00
655c1b9cc2
update docstrings / comments
2022-06-09 17:57:28 +02:00
33444be926
more str vs bytes fixing
2022-06-09 17:57:28 +02:00
8e87f1111b
cleanup msgpack related str/bytes mess, fixes #968
...
see ticket and borg.helpers.msgpack docstring.
this changeset implements the full migration to
msgpack 2.0 spec (use_bin_type=True, raw=False).
still needed compat to the past is done via want_bytes decoder in borg.item.
2022-06-09 17:57:28 +02:00
f8dbe5b542
cleanup msgpack related str/bytes mess, see #968
...
see ticket and borg.helpers.msgpack docstring.
2022-06-09 17:57:28 +02:00
86fe8bdd57
Merge pull request #6703 from ThomasWaldmann/r2r-transfer
...
borg2: "borg transfer" cmd (and also getting rid of legacy)
2022-06-09 17:56:03 +02:00
c5540c2dd9
upgrade compressed chunk: fix treatment of ObfuscateSize chunks
...
the inner payload of ObfuscateSize chunks are compressed chunks and need
the same zlib fix and level patching as non-obfuscated compressed chunks.
2022-06-09 17:49:16 +02:00
72c68c49d0
obfuscation: fix byte order for size, fixes #6701
2022-06-09 17:49:16 +02:00
6584a92c81
compression: use the 2 bytes for type and level, fixes #6698
...
adapt borg transfer, transferred chunks are set to compression level "unknown".
2022-06-09 17:49:16 +02:00
32a3601e4a
compute hlid from inode / device
2022-06-09 17:49:16 +02:00
8798b0340a
use whitelist approach to make sure item._dict is clean
2022-06-09 17:49:16 +02:00
d3dfa3be30
use version 2 for new archives
...
but still be able to read v1 archives
for borg transfer.
2022-06-09 17:49:16 +02:00
e5f1a4fb4d
recreate: cachedir_masters not needed any more
...
now all hardlinked regular file items have chunks.
2022-05-18 14:20:01 +02:00
6bfdb3f630
refactor hardlink_master processing globally
...
borg now has the chunks list in every item with content.
due to the symmetric way how borg now deals with hardlinks using
item.hlid, processing gets much simpler.
but some places where borg deals with other "sources" of hardlinks
still need to do some hardlink management:
borg uses the HardLinkManager there now (which is not much more
than a dict, but keeps documentation at one place and avoids some
code duplication we had before).
item.hlid is computed via hardlink_id function.
support hardlinked symlinks, fixes #2379
as we use item.hlid now to group hardlinks together,
there is no conflict with the item.source usage for
symlink targets any more.
2nd+ hardlinks now add to the files count as did the 1st one.
for borg, now all hardlinks are created equal.
so any hardlink item with chunks now adds to the "file" count.
ItemFormatter: support {hlid} instead of {source} for hardlinks
2022-05-18 14:20:01 +02:00
7903dad183
transfer: convert timestamps int/bigint -> msgpack.Timestamp, see #2323
...
Timestamp scales to 64 or 96bit serialization formats, that should be enough for everybody.
We use this in archived items and also in the files cache.
2022-05-18 14:20:01 +02:00
e4a97ea8cc
transfer: all hardlinks have chunks, maybe chunks_healty, hlid
...
Item.hlid: same id, same hardlink (xxh64 digest)
Item.hardlink_master: not used for new archives any more
Item.source: not used for hardlink slaves any more
2022-05-18 14:20:01 +02:00
01f72d15b4
transfer: remove the zlib type bytes hack
...
hack: see the docstring of ZLIB_legacy class.
New clean ZLIB class that works as every other compressor.
ZLIB ID 0x0500, ZLIB_legacy ID 0x.8..
2022-05-18 14:20:01 +02:00
ba1dbe6111
transfer: make sure items with chunks have precomputed size
2022-05-18 14:20:01 +02:00
98b7dc0bf5
transfer: clean item of attic 0.13 'acl' bug remnants
...
also: remove attic bug support code from borg check.
borg transfer removes the acl key. we do not run borg check on old repos.
2022-05-18 14:20:00 +02:00
116f67036f
transfer: copy archives from another repo
...
this is somehow similar to borg recreate,
but with different focus and way simpler:
not changing compression algo
not changing chunking
not excluding files inside an archive by path match
only dealing with complete archives
but:
different src and dst repo
only reading each chunk once
keeping the compressed payload (no decompression/recompression effort)
--dry-run can be used before and afterwards to check
2022-05-18 14:20:00 +02:00
1b4b84dfd8
Remove scp syntax for locations ( #6697 )
...
remove scp syntax support from cli and from Location parser, add note about converting scp-style URLs to ssh-style, fixes #6691
2022-05-15 21:55:19 +02:00
c79fd61b5c
Fix typos ( #6688 )
...
fix typos
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
2022-05-15 21:23:48 +02:00
ab4bef0372
Merge pull request #6694 from ThomasWaldmann/versions-mount-1archive-error-master
...
mount -o versions: give clear error msg instead of crashing
2022-05-15 19:52:32 +02:00
2bed467e92
mount -o versions: give clear error msg instead of crashing
...
it does not make sense to request versions view if you only
look at 1 archive, but the code shall not crash in that case
as it did, but give a clear error msg.
2022-05-15 00:37:45 +02:00
594d83aad5
docs: authentication primitives: improved security and performance infos (master) ( #6667 )
...
docs: authentication primitives: improved security and performance infos
2022-05-14 22:57:22 +02:00
ac4666d7f4
documented workaround for getting help for sub-sub-commands ( #6346 )
...
docs: how to get help for sub-sub-commands
2022-05-13 18:29:43 +02:00
c85bcfd3ad
documented secret key usage against fingerprinting ( #6345 )
...
docs: secret key usage against fingerprinting
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
2022-05-10 21:18:40 +02:00
0e3ff0ab70
Merge pull request #6682 from ThomasWaldmann/fix-archive-progress-json-master
...
show_progress: add finished=true/false to archive_progress json
2022-05-08 19:45:39 +02:00
09a84fbd24
Merge pull request #6685 from ThomasWaldmann/fix-create-filter-master
...
fix create_filter_process exception handler, fixes #6681
2022-05-08 19:15:35 +02:00
18eb696a17
fix create_filter_process exception handler, fixes #6681
...
if cmd was falsy (e.g. None), there is no proc.
then, if "yield stream" raises an exception, the exception handler crashed at "proc.kill()".
2022-05-08 18:46:13 +02:00
Thomas Waldmann