Commit Graph

7014 Commits

Author SHA1 Message Date
Michael Deyaso 6c042d7192 Show ctime and mtime on borg diff. Fixes #7248
---

- Added assert_line_exists helper in BaseTestCase
- JSON strings in diff output are now sorted alphabetically
- Modified diff test cases to confirm to new output format
- Added a test case to test ctime and mtime inclusion
- Mode, ctime & mtime are now only displayed on diff if --content-only flag is used
2023-03-07 10:54:29 +03:00
Thomas Waldmann 843c12cbea
set .hardlink_master for hardlinkable items, fixes #7175
before, we only set this for regular files, but we better also set this
for block/char devices and fifos also, so we have it for all item types
borg 1.x considers "hardlinkable".

this is important for sequential processing of items in an archive:
if we encounter a hardlinkable item and .source is not set (that
would make it a hardlink slave), it could be a hardlink master or not:

- hardlink_master == False  # this item is not part of a hl group
- hardlink_master == True  # this item is a hardlink master

This will also be important when items are converted to borg2.
2023-03-05 19:20:22 +01:00
TW d44cf3482e
Merge pull request #7402 from ThomasWaldmann/unicode-filenames-docs-1.2
update FAQ about locale/unicode issues, fixes #6999
2023-03-05 17:48:55 +01:00
TW d5e4f3f8cf
Merge pull request #7404 from ThomasWaldmann/mount-docs-1.2
docs: improve mount options rendering, fixes #7359
2023-03-05 17:48:01 +01:00
TW ca1fea37a6
Merge pull request #7410 from snsmac/backport/autocompressortests
"auto" compressor tests: don't assume a specific size (backport to 1.2-maint)
2023-03-05 17:22:45 +01:00
TW 831c9d0c40
Merge pull request #7408 from snsmac/backport/hashindex
hashindex bugfix and refactoring (backport to 1.2-maint)
2023-03-05 15:29:50 +01:00
Thomas Waldmann 8cf313900a "auto" compressor tests: do not assume zlib is better than lz4, fixes #7363
while that might be true for many cases, we can not assume it is
always true, as proven by the failing test on S390.
2023-03-05 11:05:47 +01:00
Thomas Waldmann a7ce1db529 add num_entries assertion 2023-03-05 11:01:44 +01:00
Thomas Waldmann f1d68fe4b3 hashindex: simplify size_idx function
Thanks to @jdchristensen for the code.
2023-03-05 11:01:44 +01:00
Thomas Waldmann d57fafadad Simplify full HT scan assertion 2023-03-05 11:01:44 +01:00
Thomas Waldmann ec32413b5e hashindex: always have at least 1 empty bucket
avoid rounding / integer conversion issues bringing this down to 0.
2023-03-05 11:01:44 +01:00
Thomas Waldmann 241eaec413 implement ht idx wrap around less strangely, add comment 2023-03-05 11:01:44 +01:00
Thomas Waldmann 51189e1383 if HT is full with entries and tombstones: give up/fail early 2023-03-05 11:01:44 +01:00
Thomas Waldmann 32519617b6 more comments for hashindex_lookup 2023-03-05 11:01:44 +01:00
Thomas Waldmann 3a44894aca bugfix: do not resize hashindex with wrong num_empty
otherwise we would lose the decrement operation on num_empty.
2023-03-05 11:01:44 +01:00
Thomas Waldmann ecf5ad43ed hashindex: simplify assert 2023-03-05 11:01:44 +01:00
Thomas Waldmann b2117bfd4f _hashindex: easier to understand code, dubious loops removed, asserts
hashindex_index returns the perfect hashtable index, but does not
check what's in the bucket there, so we had these loops afterwards
to search for an empty or deleted bucket.

problem: if the HT were completely filled with no empty and no deleted
buckets, that loop would never end. due to our HT resizing, it can
never happen, but still not pretty.

when using hashindex_lookup (as also used some lines above), the code
is easier to understand, because (after we resized the HT), we freshly
create the same situation as after the first call of that function:
- return value < 0, because we (still) can not find the key
- start_idx will point to an empty bucket

Thus, we do not need the problematic loops we had there.

Modified the checks to make sure we really have an empty or deleted
bucket before overwriting it with data.

Added some additional asserts to make sure the code behaves.
2023-03-05 11:01:44 +01:00
TW cef5b446ac
Merge pull request #7411 from ThomasWaldmann/debug-id-hash-command-1.2
debug id-hash: implement file content id-hash computation, see #7406
2023-03-04 22:33:20 +01:00
TW 42fb72c022
Merge pull request #7409 from snsmac/backport/hashtablestress
Add hashtable stress test (backport to 1.2-maint)
2023-03-04 22:14:47 +01:00
Thomas Waldmann eb8ccd22d4
debug id-hash: implement file content id-hash computation, see #7406 2023-03-04 21:47:14 +01:00
snsmac c339d2e9e2 Update hashtable stress test to work with borg < 2 where NSIndex has 2-tuples as values 2023-03-04 21:41:31 +01:00
TW 086a4237ca
Merge pull request #7407 from ThomasWaldmann/fix-debug-cmds-1.2
fix borg debug get-obj/put-obj (1.2-maint)
2023-03-04 21:21:09 +01:00
Thomas Waldmann 356ba3cc79
debug put-obj: id must be given on commandline, fixes #7290
this is an incompatible change:

before:
borg debug put-obj :: path1 path2 ...
(and borg computed all IDs automatically) (*)

after:
borg debug put-obj :: id path
(id must be given)

(*) the code just using sha256(data) was outdated and incorrect anyway.

also: fix error handling in get-obj.
2023-03-04 20:25:45 +01:00
Thomas Waldmann 8fe980c94e "auto" compressor tests: don't assume a specific size, fixes #7363
The tests assumed a specific compressed results size, which
is bad, because it might vary depending on the zlib implementation.

Now the "auto" compressor tests just check if it is the same size
as when unconditionally using the zlib compressor.
2023-03-04 20:19:08 +01:00
Thomas Waldmann b7d70f2064 add comment about how to provoke more collisions 2023-03-04 20:09:58 +01:00
Thomas Waldmann b33fea7718 add hashtable stress tests
Using NSIndex (repo index) HashIndex, but they all are very similar.
2023-03-04 20:09:50 +01:00
Thomas Waldmann d739f29657
docs: improve mount options rendering, fixes #7359 2023-03-04 17:08:15 +01:00
Thomas Waldmann e8573193e4
update FAQ about locale/unicode issues, fixes #6999 2023-03-04 16:32:48 +01:00
TW 6e8c58c125
Merge pull request #7395 from helmutg/reproducible-docs-1.2
Make timestamps in manual pages reproducible
2023-03-01 09:20:20 +01:00
Vagrant Cascadian a5c9231e88 Make timestamps in manual pages reproducible
Refer to https://reproducible-builds.org/docs/source-date-epoch/ for
documentation on SOURCE_DATE_EPOCH.

Bug-Debian: https://bugs.debian.org/1029807
Signed-off-by: Helmut Grohne <helmut@subdivi.de>

(cherry picked from commit 98352cf667)
2023-02-28 13:38:45 +01:00
TW a7a80ffe90
Merge pull request #7382 from ThomasWaldmann/rtd-config-1.2
add .readthedocs.yaml
2023-02-25 18:51:51 +01:00
Thomas Waldmann fd6e76e6a4
add .readthedocs.yaml
same as in master branch
2023-02-25 18:42:49 +01:00
TW 836bd0339d
Merge pull request #7361 from voegelas/1.2-maint-fedora
Installation: Update Fedora in distribution list
2023-02-17 18:58:50 +01:00
Andreas Vögele 007b73ac67 Installation: Update Fedora in distribution list
Fixes #7357
2023-02-17 14:45:17 +01:00
TW 1dce595d8f
Merge pull request #7338 from ThomasWaldmann/timestamp-comparisons-1.2
relaxed timestamp comparisons (1.2)
2023-02-11 00:19:08 +01:00
TW e41466dbb3
Merge pull request #7337 from ThomasWaldmann/fix-recreate-rechunkify-1.2
recreate without --chunker-params shall not rechunk (1.2)
2023-02-10 18:37:11 +01:00
Thomas Waldmann 6bf818a2a1
tests: use same_ts_ns for all timestamp comparisons 2023-02-10 18:36:10 +01:00
Thomas Waldmann b3da30e6d4
same_ts_ns: add a timestamp comparison function
also: HAVE_UTIMES means 1us resolution (*)

(*) at least as far as the utimes() call is concerned.

Whether there are other limitations has to be seen.
2023-02-10 17:58:46 +01:00
Thomas Waldmann cd23d810d3
recreate: --chunker-params must default to None, fixes #7337
before the fix, it triggered rechunking if an archive
was created with non-default chunker params.

but it only should rechunk if borg recreate is invoked with explicitly giving --chunker-params=....
2023-02-10 14:12:08 +01:00
Thomas Waldmann dbe1c98cb3
tests: recreate without --chunker-params shall not rechunk 2023-02-10 14:01:04 +01:00
TW 5ccfaa7a45
Merge pull request #7299 from ThomasWaldmann/test-dir-mtime-1.2
add test for extracted directory mtime
2023-01-28 00:59:59 +01:00
Thomas Waldmann b472768b57
more directory timestamp tests 2023-01-24 15:58:30 +01:00
Thomas Waldmann 2291ab755d
add test for extracted directory mtime 2023-01-24 15:22:01 +01:00
TW 48713ebff6
Merge pull request #7265 from ThomasWaldmann/fix-locking-1.2
Fix locking (1.2-maint)
2023-01-19 18:15:28 +01:00
TW dbba9cbc53
Merge pull request #7268 from ThomasWaldmann/recreate-with-target-1.2
recreate: when --target is given, do not detect "nothing to do"
2023-01-16 17:22:50 +01:00
Thomas Waldmann c43140686b
recreate: when --target is given, do not detect "nothing to do"
use case:

borg recreate -a src --target dst can be used to make a copy
of an archive inside the same repository, see #7254.
2023-01-14 19:28:57 +01:00
Thomas Waldmann 16eb38e20a
fix host, pid, tid order
using "differenthost" (== not the current hostname) makes
the process_alive check always return True (to play safe,
because in can not check for processes on other hosts).
2023-01-13 22:36:50 +01:00
Thomas Waldmann 91104674e7
use os.replace not os.rename 2023-01-13 22:32:11 +01:00
Thomas Waldmann 804867d3ad
bugfix: thread id must be parsed as hex from lock file name 2023-01-13 22:30:28 +01:00
TW 1dcaa74788
Merge pull request #7252 from puetzk/test_size_on_disk_accurate-flush-1.2
[1.2-maint] Fix test_size_on_disk_accurate for large st_blksize, fixes #7250
2023-01-12 20:52:48 +01:00