7990 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Thomas Waldmann	79d125212d	readthedocs: don't create a pdf	2023-02-25 02:27:08 +01:00
TW	a96cba4ebd	Merge pull request #7380 from ThomasWaldmann/rtd-config-master ... add .readthedocs.yaml	2023-02-25 02:17:03 +01:00
Thomas Waldmann	a40390b1ac	add .readthedocs.yaml ... otherwise rtd uses: - python 3.7, which is too old for borg. - shallow git checkout, which might make setuptools_scm fail to create a correct version	2023-02-25 02:16:21 +01:00
Soumik Dutta	cad138aa23	Add files changed while reading to Statistics class #7354 (#7378) ... add files changed while reading to Statistics class, fixes #7354 Signed-off-by: Soumik Dutta <shalearkane@gmail.com>	2023-02-25 01:47:39 +01:00
TW	b208785fff	Merge pull request #7351 from ThomasWaldmann/retry-failed-files ... create: implement retries for individual fs files	2023-02-23 01:46:36 +01:00
Thomas Waldmann	7e6afc93e9	create: implement retries for individual fs files ... Errors handled for backup src files: - BackupOSError (converted from OSError), e.g. I/O Error - BackupError (stats race, file changed while we backed it up) Error Handling: - retry the same file after some sleep time - sleep time starts from 1ms, increases exponentially up to 10s - 10 tries If retrying does not help: - BackupOSError: skip the file, log it with "E" status - BackupError: last try will back it up, log it with "C" status Works for: - borg create's normal (builtin) fs recursion - borg create --paths-from-command - borg create --paths-from-stdin Notes: - update stats.files_stats late (so we don't get wrong stats in case of e.g. IOErrors while reading the file). - _process_any: no changes to the big block, just indented for adding the retry loop and the try/except. - test_create_erroneous_file succeeds because we retry the file.	2023-02-23 01:19:19 +01:00
TW	6da5b7d1ba	Merge pull request #7367 from ThomasWaldmann/new-crypto-assert-id ... assert_id: better be paranoid, fixes #7362	2023-02-23 01:14:25 +01:00
TW	3f2aac8f00	Merge pull request #7368 from vhadzhiev/vhadzhiev_contribution ... fixed Statistics.__add__(), fixes #7355	2023-02-20 18:07:21 +01:00
vhadzhiev	a08a3eb173	fixed Statistics.__add__(), fixes #7355	2023-02-20 11:25:28 +02:00
Thomas Waldmann	74a19ee2a0	verify_data: always decompress and call assert_id(), see #7362	2023-02-19 21:25:24 +01:00
Thomas Waldmann	fea630027c	assert_id: better be paranoid, fixes #7362 ... This makes borg2's new crypto do the same check as borg1's old crypto and makes sure that no chunks created by an evil borg client would go unnoticed.	2023-02-19 21:14:20 +01:00
TW	93a4bd61f8	Merge pull request #7360 from voegelas/fedora-package-url ... Installation: Update Fedora in distribution list	2023-02-17 14:32:34 +01:00
Andreas Vögele	4b18980768	Installation: Update Fedora in distribution list ... Fixes #7357	2023-02-16 18:27:48 +01:00
TW	3849ebe312	Merge pull request #7349 from ThomasWaldmann/avoid-orphan-content-chunks3 ... avoid orphan content chunks (master)	2023-02-14 00:01:50 +01:00
Thomas Waldmann	71f8dd3a17	FilesystemObjectProcessors.process_pipe: also add same exception handler there	2023-02-13 20:46:48 +01:00
Thomas Waldmann	c9dbe323e3	TarfileObjectProcessors.process_file: also add same exception handler there	2023-02-13 20:46:46 +01:00
Thomas Waldmann	d0c61bbbf1	FilesystemObjectProcessors.process_file: clean up orphaned chunks in case of exceptions ... Note: no changes inside the indented block, just added the try and the except block.	2023-02-13 20:46:45 +01:00
Thomas Waldmann	f1981715e4	2nd+ hardlinks: add to item.chunks after incref'ing ... we do book-keeping in item.chunks: in case something goes wrong and we need to clean up, we will have a list with chunks to decref in item.chunks. also: - make variable naming more consistent - cosmetic changes	2023-02-13 20:46:21 +01:00
Thomas Waldmann	5cb3a17796	Revert "avoid orphan content chunks on BackupOSError, fixes #6709" ... This reverts commit ffe32316a5.	2023-02-13 18:24:28 +01:00
TW	6cfe77ebaf	Merge pull request #7350 from ThomasWaldmann/testing-chunker ... implement "fail" chunker for testing purposes	2023-02-13 18:17:56 +01:00
Thomas Waldmann	11fd6afb0f	use "fail" chunker to test erroneous input file skipping ... if a file can't be read (like here: there is a simulated I/O error in the 2nd chunk of file2), it should be logged with "E" status, skipped and backup shall proceed with next file(s). also, check that the repo has no orphan chunks (exception handling code needs to deal with 1st chunk of file2 which already has been written / incref'd in the repo).	2023-02-13 17:19:49 +01:00
Thomas Waldmann	7f973a5b34	implement "fail" chunker for testing purposes ... --chunker-params=fail,4096,rrrEErrrr means: - cut chunks of 4096b fixed size (last chunk in a file can be less) - read chunks 0, 1 and 2 successfully - error at chunk 3 and 4 (simulated OSError(errno.EIO)) - read successfully again for the next 4 chunks Chunks are counted inside the chunker instance, starting from 0, always increasing while the same instance is used. Read chunks as well as failed chunks count up by 1.	2023-02-13 17:15:45 +01:00
TW	52040bc043	Merge pull request #7347 from jdchristensen/hashindex_compact ... Make hashindex_compact simpler and probably faster	2023-02-12 17:01:18 +01:00
Dan Christensen	3daf7e186d	testsuite/hashindex.py: add comments about validity of hashtables	2023-02-12 09:39:07 -05:00
TW	ee7fee75fc	Merge pull request #7345 from ThomasWaldmann/update-changes ... update CHANGES	2023-02-12 14:56:18 +01:00
Dan Christensen	c705c8573d	testsuite/hashindex.py: add a few more tests	2023-02-12 08:28:14 -05:00
Dan Christensen	8eff0b2f02	_hashindex.c: fix comment	2023-02-12 08:24:27 -05:00
Dan Christensen	2f9f22e2a8	hashindex.py: make black happy	2023-02-11 20:33:07 -05:00
Dan Christensen	36aa395e49	_hashindex.c: set min_empty and num_empty even when permit_compact=True	2023-02-11 20:11:09 -05:00
Dan Christensen	60bea46eb7	_hashindex.c: rewrite hashindex_compact	2023-02-11 20:10:02 -05:00
Dan Christensen	6f5efb3f74	testsuite/hashindex.py: create and use compare_compact to avoid repetition	2023-02-11 20:06:38 -05:00
Dan Christensen	d3fd93f82b	testsuite/hashindex.py: use compare_indexes in one more spot	2023-02-11 19:53:28 -05:00
Dan Christensen	cc78ef8939	testsuite/hashindex.py: add two more tests	2023-02-11 19:44:56 -05:00
Dan Christensen	6aa53a5627	hashindex_pytest.py: add test_hashindex_compact_stress	2023-02-11 19:13:20 -05:00
Dan Christensen	51b29ab426	hashindex.py: make .compact tests independent of order	2023-02-11 17:37:53 -05:00
Dan Christensen	133ffebc8d	hashindex_pytest.py: factor out verify_hash_table	2023-02-11 17:37:20 -05:00
Thomas Waldmann	b42d2bf6dd	update CHANGES	2023-02-11 18:56:30 +01:00
TW	6db21fa330	Merge pull request #7344 from RayyanAnsari/parseformat-rm-hack ... parseformat: remove Windows-specific hack	2023-02-11 18:23:47 +01:00
TW	098fca7861	Merge pull request #7343 from ThomasWaldmann/timestamp-comparisons-master ... relaxed timestamp comparisons (master)	2023-02-11 18:15:02 +01:00
Rayyan Ansari	03a27654e0	parseformat: remove Windows-specific hack ... This fixes the rest of the tests.	2023-02-11 16:32:19 +00:00
Thomas Waldmann	aee17a87e2	tests: use same_ts_ns for all timestamp comparisons	2023-02-11 17:08:56 +01:00
TW	d90ebc8bef	Merge pull request #7340 from ThomasWaldmann/fix-recreate-rechunkify-master ... recreate: --chunker-params must default to None, fixes #7337	2023-02-11 14:11:03 +01:00
TW	ce04521a9a	Merge pull request #7333 from ThomasWaldmann/excluded-included-status-chars ... better included/excluded status chars, docs, fixes #7321	2023-02-11 13:48:43 +01:00
TW	f39d1bcd7b	Merge pull request #7334 from ThomasWaldmann/list-progress-json-is-nice ... allow mixing --progress and --list in log-json mode	2023-02-11 13:42:59 +01:00
Thomas Waldmann	2ef09eaa77	same_ts_ns: add a timestamp comparison function ... also: HAVE_UTIMES means 1us resolution (*) (*) at least as far as the utimes() call is concerned. Whether there are other limitations has to be seen.	2023-02-11 01:19:01 +01:00
TW	b233dbb4db	Merge pull request #7339 from ThomasWaldmann/test-dir-mtime-master ... add test for extracted directory mtime (master)	2023-02-11 01:13:32 +01:00
Thomas Waldmann	ce5f1589ea	recreate: --chunker-params must default to None, fixes #7336 ... also add a test: recreate without --chunker-params shall not rechunk before the fix, it triggered rechunking if an archive was created with non-default chunker params. but it only should rechunk if borg recreate is invoked with explicitly giving --chunker-params=....	2023-02-11 01:08:01 +01:00
Thomas Waldmann	fdc48fdd1b	more directory timestamp tests	2023-02-11 00:42:30 +01:00
Thomas Waldmann	a42cd161de	add test for extracted directory mtime	2023-02-11 00:29:57 +01:00
Thomas Waldmann	ff564de1cc	allow mixing --progress and --list in log-json mode	2023-02-10 01:45:59 +01:00