From 76c5f1a258717e14ccc6cd5bf9d73e989154cdc9 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 26 Nov 2016 04:34:01 +0100 Subject: [PATCH 1/2] add more details about resource usage --- docs/internals.rst | 21 ++++++++++++- docs/usage.rst | 78 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 81 insertions(+), 18 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index 4df70e94f..138761b2d 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -364,7 +364,7 @@ varies between 33% and 300%. Indexes / Caches memory usage ----------------------------- -Here is the estimated memory usage of |project_name|: +Here is the estimated memory usage of |project_name| - it's complicated: chunk_count ~= total_file_size / 2 ^ HASH_MASK_BITS @@ -377,6 +377,14 @@ Here is the estimated memory usage of |project_name|: mem_usage ~= repo_index_usage + chunks_cache_usage + files_cache_usage = chunk_count * 164 + total_file_count * 240 +Due to the hashtables, the best/usual/worst cases for memory allocation can +be estimated like that: + + mem_allocation = mem_usage / load_factor # l_f = 0.25 .. 0.75 + + mem_allocation_peak = mem_allocation * (1 + growth_factor) # g_f = 1.1 .. 2 + + All units are Bytes. It is assuming every chunk is referenced exactly once (if you have a lot of @@ -388,6 +396,17 @@ more chunks than estimated above, because 1 file is at least 1 chunk). If a remote repository is used the repo index will be allocated on the remote side. +The chunks cache, files cache and the repo index are all implemented as hash +tables. A hash table must have a significant amount of unused entries to be +fast - the so-called load factor gives the used/unused elements ratio. + +When a hash table gets full (load factor getting too high), it needs to be +grown (allocate new, bigger hash table, copy all elements over to it, free old +hash table) - this will lead to short-time peaks in memory usage each time this +happens. Usually does not happen for all hashtables at the same time, though. +For small hash tables, we start with a growth factor of 2, which comes down to +~1.1x for big hash tables. + E.g. backing up a total count of 1 Mi (IEC binary prefix i.e. 2^20) files with a total size of 1TiB. a) with ``create --chunker-params 10,23,16,4095`` (custom, like borg < 1.0 or attic): diff --git a/docs/usage.rst b/docs/usage.rst index 61b469b15..b00ad8c34 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -206,36 +206,79 @@ Resource Usage |project_name| might use a lot of resources depending on the size of the data set it is dealing with. -CPU: +If one uses |project_name| in a client/server way (with a ssh: repository), +the resource usage occurs in part on the client and in another part on the +server. + +If one uses |project_name| as a single process (with a filesystem repo), +all the resource usage occurs in that one process, so just add up client + +server to get the approximate resource usage. + +CPU client: + borg create: does chunking, hashing, compression, crypto (high CPU usage) + chunks cache sync: quite heavy on CPU, doing lots of hashtable operations. + borg extract: crypto, decompression (medium to high CPU usage) + borg check: similar to extract, but depends on options given. + borg prune / borg delete archive: low to medium CPU usage + borg delete repo: done on the server It won't go beyond 100% of 1 core as the code is currently single-threaded. Especially higher zlib and lzma compression levels use significant amounts - of CPU cycles. + of CPU cycles. Crypto might be cheap on the CPU (if hardware accelerated) or + expensive (if not). -Memory (RAM): +CPU server: + It usually doesn't need much CPU, it just deals with the key/value store + (repository) and uses the repository index for that. + + borg check: the repository check computes the checksums of all chunks + (medium CPU usage) + borg delete repo: low CPU usage + +CPU (only for client/server operation): + When using borg in a client/server way with a ssh:-type repo, the ssh + processes used for the transport layer will need some CPU on the client and + on the server due to the crypto they are doing - esp. if you are pumping + big amounts of data. + +Memory (RAM) client: The chunks index and the files index are read into memory for performance - reasons. + reasons. Might need big amounts of memory (see below). Compression, esp. lzma compression with high levels might need substantial amounts of memory. -Temporary files: - Reading data and metadata from a FUSE mounted repository will consume about - the same space as the deduplicated chunks used to represent them in the - repository. +Memory (RAM) server: + The server process will load the repository index into memory. Might need + considerable amounts of memory, but less than on the client (see below). -Cache files: - Contains the chunks index and files index (plus a compressed collection of - single-archive chunk indexes). - -Chunks index: +Chunks index (client only): Proportional to the amount of data chunks in your repo. Lots of chunks in your repo imply a big chunks index. It is possible to tweak the chunker params (see create options). -Files index: - Proportional to the amount of files in your last backup. Can be switched +Files index (client only): + Proportional to the amount of files in your last backups. Can be switched off (see create options), but next backup will be much slower if you do. -Network: +Repository index (server only): + Proportional to the amount of data chunks in your repo. Lots of chunks + in your repo imply a big repository index. + It is possible to tweak the chunker params (see create options) to + influence the amount of chunks being created. + +Temporary files (client): + Reading data and metadata from a FUSE mounted repository will consume about + the same space as the deduplicated chunks used to represent them in the + repository. + +Temporary files (server): + Not much. + +Cache files (client only): + Contains the chunks index and files index (plus a collection of single- + archive chunk indexes which might need huge amounts of disk space, + depending on archive count and size - see FAQ about how to reduce). + +Network (only for client/server operation): If your repository is remote, all deduplicated (and optionally compressed/ encrypted) data of course has to go over the connection (ssh: repo url). If you use a locally mounted network filesystem, additionally some copy @@ -243,7 +286,8 @@ Network: you backup multiple sources to one target repository, additional traffic happens for cache resynchronization. -In case you are interested in more details, please read the internals documentation. +In case you are interested in more details (like formulas), please read the +internals documentation. Units From 30d1e21e5370444b910bb585af4fb31f248ac78b Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sat, 26 Nov 2016 20:49:39 +0100 Subject: [PATCH 2/2] fixup: fixes, clarify --- docs/usage.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/usage.rst b/docs/usage.rst index b00ad8c34..21d9c9ccd 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -257,7 +257,8 @@ Chunks index (client only): Files index (client only): Proportional to the amount of files in your last backups. Can be switched - off (see create options), but next backup will be much slower if you do. + off (see create options), but next backup might be much slower if you do. + The speed benefit of using the files cache is proportional to file size. Repository index (server only): Proportional to the amount of data chunks in your repo. Lots of chunks @@ -266,12 +267,12 @@ Repository index (server only): influence the amount of chunks being created. Temporary files (client): - Reading data and metadata from a FUSE mounted repository will consume about - the same space as the deduplicated chunks used to represent them in the - repository. + Reading data and metadata from a FUSE mounted repository will consume up to + the size of all deduplicated, small chunks in the repository. Big chunks + won't be locally cached. Temporary files (server): - Not much. + None. Cache files (client only): Contains the chunks index and files index (plus a collection of single- @@ -286,8 +287,8 @@ Network (only for client/server operation): you backup multiple sources to one target repository, additional traffic happens for cache resynchronization. -In case you are interested in more details (like formulas), please read the -internals documentation. +In case you are interested in more details (like formulas), please see +:ref:`internals`. Units