Commit Graph

10 Commits

Author SHA1 Message Date
Alexander Neumann 66efa425bf Reuse buffer in worker functions 2019-04-13 13:38:39 +02:00
Alexander Neumann d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
2019-04-13 13:38:39 +02:00
Alexander Neumann 663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Matthew Dawson 3789e55e20
repostiory/index: Remove logging from Lookup function.
The logging in these functions double the time they take to execute.
However, it is only really useful on failures, which are better
reported by the calling functions.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              897           395           -55.96%
BenchmarkMasterIndexLookupMultipleIndex-6            2001          1090          -45.53%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       492           215           -56.30%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     1649          912           -44.69%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              1              -88.89%
BenchmarkMasterIndexLookupMultipleIndex-6            19             1              -94.74%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       6              0              -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     16             0              -100.00%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           96            -40.00%
BenchmarkMasterIndexLookupMultipleIndex-6            240           96            -60.00%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       48            0             -100.00%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     128           0             -100.00%
2018-01-23 22:28:38 -05:00
Matthew Dawson df2c03a6a4
repository/master_index: Optimize Index.Lookup()
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
2018-01-23 22:25:56 -05:00
Matthew Dawson 539599d1f1 repository/index: Optimize index.Has()
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
2018-01-08 21:46:17 +01:00
Alexander Neumann 1eaad6cebb index: Add TreePacks() 2017-09-24 21:54:53 +02:00
Alexander Neumann 23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann 6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann 83d1a46526 Moves files 2017-07-23 14:19:13 +02:00
Renamed from src/restic/repository/index.go (Browse further)