1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2024-12-27 02:08:54 +00:00
Commit graph

4075 commits

Author SHA1 Message Date
Marian Beermann
0e31f78dd6 cache sync: avoid "l" and such as a variable name, note vanilla changes 2017-06-10 10:17:28 +02:00
Marian Beermann
5eb43b8464 cache sync: give overview of the source's structure 2017-06-10 10:17:28 +02:00
Marian Beermann
310a71e4f0 cache sync: use ro_buffer to accept bytes, memoryview, ... 2017-06-10 10:17:28 +02:00
Marian Beermann
5af66dbb12 cache sync: add more refcount tests 2017-06-03 15:02:27 +02:00
Marian Beermann
4faaa7d1fa RepositoryCache: abort on data corruption 2017-06-03 12:27:35 +02:00
Marian Beermann
b544af2af1 RepositoryCache: checksum decrypted cache 2017-06-03 12:23:23 +02:00
Marian Beermann
5b3667b617 cache sync: macros in all-caps 2017-06-02 19:31:56 +02:00
Marian Beermann
795cdfc9ab cache sync: move stat initialization to main unpack 2017-06-02 19:30:53 +02:00
Marian Beermann
cb98cb838d fuse: fix read(2) caching data in metadata cache
The OS page cache is responsible for handling this and is much more
empowered to do a good job at that than Borg.
2017-06-02 17:43:15 +02:00
Marian Beermann
67b97f2223 cache sync: cleanup progress handling, unused parameters 2017-06-02 17:43:15 +02:00
Marian Beermann
7f04e00ba2 testsuite: add TestRepositoryCache 2017-06-02 17:43:15 +02:00
Marian Beermann
835b0e5ee0 cache sync/remote: compressed, decrypted cache 2017-06-02 17:43:15 +02:00
Marian Beermann
c786a5941e CacheSynchronizer: redo as quasi FSM on top of unpack.h
This is a (relatively) simple state machine running in the
data callbacks invoked by the msgpack unpacking stack machine
(the same machine is used in msgpack-c and msgpack-python,
changes are minor and cosmetic, e.g. removal of msgpack_unpack_object,
removal of the C++ template thus porting to C and so on).

Compared to the previous solution this has multiple advantages
- msgpack-c dependency is removed
- this approach is faster and requires fewer and smaller
  memory allocations

Testability of the two solutions does not differ in my
professional opinion(tm).

Two other changes were rolled up; _hashindex.c can be compiled
without Python.h again (handy for fuzzing and testing);
a "small" bug in the cache sync was fixed which allocated too
large archive indices, leading to excessive archive.chunks.d
disk usage (that actually gave me an idea).
2017-06-02 17:43:15 +02:00
Marian Beermann
bf895950ac RepositoryCache: limit cache size
Unbounded cache size is inacceptable.

I don't see why a full-fledged repository needs to be used here, either,
since this cache requires none of the consistency or durability guarantees
made by it (and bought with a performance impact).

A notable issue is that posix_fadvise is slow (for some reason) on tmpfs,
which could eat 30-35 % of the total CPU time of a cache sync.
2017-06-02 17:43:15 +02:00
Marian Beermann
167875b753 cache sync: fix n^2 behaviour in lookup_name 2017-06-02 17:43:14 +02:00
Marian Beermann
9f8b967a6f cache sync: initialize master index to known capacity 2017-06-02 17:43:14 +02:00
Marian Beermann
740898d83b CacheSynchronizer 2017-06-02 17:43:14 +02:00
enkore
47894072ad Merge pull request #2580 from enkore/issue/1101.integration.xxh
1101.integration.xxh
2017-06-02 16:23:54 +02:00
enkore
d3373bebf1 Merge pull request #2588 from enkore/docs/quota
docs: quotas: clarify compatbility; only relevant to serve side
2017-06-02 16:22:20 +02:00
Marian Beermann
d51f2bbbae docs: quotas: local repo disclaimer
...
2017-06-02 14:50:51 +02:00
Marian Beermann
089224975b docs: quotas: clarify compatbility; only relevant to serve side
also cf. "Enforcing the quota":

  The quota is enforcible only if *all* :ref:`borg_serve` versions
  accessible to clients support quotas
2017-06-02 13:12:43 +02:00
Marian Beermann
6c91a750d1 algorithms: rename crc32 to checksums 2017-06-01 21:26:42 +02:00
Marian Beermann
0221e31058 file_integrity: use xxh64 2017-06-01 21:26:42 +02:00
enkore
23d591c1c3 Merge pull request #2565 from enkore/issue/2517
Implement storage quotas
2017-06-01 14:39:17 +02:00
Marian Beermann
bcf4b4492b testsuite: add test for parse_storage_quota 2017-05-31 20:28:17 +02:00
enkore
1b66d8d109 Merge pull request #2576 from enkore/pr/2573+split
BORG_PASSCOMMAND: use same cmd-string splitting as BORG_RSH
2017-05-31 19:41:42 +02:00
Marian Beermann
4e6a771ee7 BORG_PASSCOMMAND: use same cmd-string splitting as BORG_RSH 2017-05-31 19:41:17 +02:00
enkore
349a4ade7c Merge pull request #2568 from enkore/issue/1101.integration.cache
1101.integration.cache
2017-05-31 19:34:43 +02:00
TuXicc
578b76af3a Added BORG_PASSCOMMAND environment variable (#2573) 2017-05-31 19:25:21 +02:00
Marian Beermann
f8b48dc8d7 remote: propagate Error.traceback correctly 2017-05-31 18:48:48 +02:00
Marian Beermann
4edf77788d Implement storage quotas 2017-05-31 18:36:03 +02:00
Marian Beermann
9032aa062b testsuite: simplify ArchiverCorruptionTestCase 2017-05-31 18:08:20 +02:00
Marian Beermann
0a5d9b6f7c cache sync: close archive chunks file before renaming 2017-05-31 18:06:28 +02:00
enkore
02689174ab Merge pull request #2574 from edgimar/patch-1
Update changes.rst
2017-05-28 09:12:28 +02:00
edgimar
b57fb1a5e5 Update changes.rst
fix denglish
2017-05-28 00:52:56 -04:00
enkore
7548f525c8 Merge pull request #2569 from enkore/f/setup-cclean
setup.py clean to remove compiled files
2017-05-26 00:47:11 +02:00
enkore
7d7b2f3407 Merge pull request #2566 from enkore/issue/2560
Restrict key file names
2017-05-26 00:43:42 +02:00
Marian Beermann
0a295dd753 setup.py clean to remove compiled files 2017-05-25 17:58:57 +02:00
Marian Beermann
d35d388d9c cache integrity: handle interference from old versions 2017-05-25 17:44:01 +02:00
Marian Beermann
50ac9d914d testsuite: add ArchiverCorruptionTestCase 2017-05-25 17:44:01 +02:00
Marian Beermann
83bca02a4e file_integrity: hash_part: mix length into state 2017-05-25 17:44:01 +02:00
Marian Beermann
d463dd89aa hashindex: read/write: use hash_part for HashHeader 2017-05-25 17:44:01 +02:00
Marian Beermann
f59affe585 cache: fix possible printf issue with archive names in sync 2017-05-25 17:44:01 +02:00
Marian Beermann
addd7addfe cache: chunks.archive.d: autofix corruption 2017-05-25 17:44:01 +02:00
Marian Beermann
1dfe693003 cache: integrity checking in archive.chunks.d 2017-05-25 16:28:46 +02:00
Marian Beermann
2b518b7188 cache: add integrity checking of chunks and files caches 2017-05-25 16:28:46 +02:00
Marian Beermann
06cf15cc6d hashindex: read/write: accept file-like objects for path 2017-05-25 14:04:41 +02:00
Marian Beermann
39051ac5f1 file_integrity: split in IntegrityCheckedFile + Detached variant 2017-05-25 14:04:41 +02:00
Marian Beermann
38ed9a20af key file names: limit to 100 characters (not bytes) 2017-05-25 12:36:45 +02:00
Marian Beermann
4b8a04b5e7 key file names: remove colons from host name 2017-05-25 12:32:42 +02:00