mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
1,195 Commits 7 Branches 49 Tags 76 MiB
Commit Graph

50 Commits

Author SHA1 Message Date
Philipp Serr e7bf936d2b Increase number of chunks and test repetitions 2015-12-28 18:33:28 +01:00
Philipp Serr 3d7f72311a Provoke unreferenced packs using fewer goroutines 
TestParallelSaveWithDuplication has been reworked to provoke
unreferenced packs using fewer goroutines than before and create
only one bytes.Reader per blob. This reduces memory usage
significantly.

The following actions have been taken to keep the chance of provoking
unreferenced packs due to #358 high:
 * Interweaved processing of subsequent chunks
 * Delaying each goroutine by a few pseudo-randomly chosen nanoseconds
   (depending on the platform this will most probably only make the os
   yield execution to another thread): together with the interweaved
   processing of subsequent chunks, this ensures a minimalistic delay
   between processing of (some) duplicated chunks
 * Repeating the test 5 times with different seeds

On my test machine, the modified test provoked unreferenced packs 60
times in a row.
 2015-12-28 18:33:26 +01:00
Philipp Serr 6a548336ec Add a test concurrently saving duplicated chunks 
This commit adds an integration test, that calls Archiver.Save from
many goroutines processing several duplicated chunks concurrently.
The test asserts, that after all chunks have been saved, there are no
unreferenced packs in the repository.

The test has been checked to give the expected results:
 1) Running the test with maxParallel=1 (all chunks are processed
    sequentially) has been verified not to produce any unreferenced
    packs. Consequently the test passes.
 2) Running the test with unbounded parallelism (maxParallel=
    math.MaxInt32) has been verified to produce unreferenced packs
    all the time (at least 25 test runs). Consequently the test fails
    due to #358.

references: #358
 2015-12-28 18:33:22 +01:00
Alexander Neumann 86fcd170f6 Add and use MasterIndex 2015-10-12 22:34:12 +02:00
Florian Weingarten 36a62cf77d Extract chunker 2015-07-08 16:58:23 -04:00
Alexander Neumann 26e4d2e019 tests: Remove more flags 2015-06-28 13:52:22 +02:00
Alexander Neumann 189a33730a tests: Standardize use of SetupRepo/Teardown 2015-06-28 13:52:22 +02:00
Alexander Neumann d9b5832034 Rename variables 2015-05-09 13:47:21 +02:00
Alexander Neumann f79e530e18 chunker: Further cleanup 2015-05-05 00:56:07 +02:00
Alexander Neumann ea22b2dfb1 chunker: move buffer pool to chunker package 2015-05-05 00:51:47 +02:00
Alexander Neumann bdbb3ab329 Remove pools.go 2015-05-05 00:14:07 +02:00
Alexander Neumann 64a290c8db chunker: Add buffer to New() 2015-05-04 23:25:07 +02:00
Florian Weingarten 282c4379db Remove chunker.Reset() and chunker pool 2015-05-04 16:49:49 -04:00
Alexander Neumann c57c4f0b8f Merge branch 'add_config' 2015-05-04 22:13:59 +02:00
Alexander Neumann ab3da5c8af archiver: Remove trees and packs from dedup test 2015-05-04 22:11:53 +02:00
Alexander Neumann 1213d87b1a server: Only save crypto.Key 
At the moment, the server doesn't need the full server.Key (master and
user key), just the master key.
 2015-05-03 18:56:20 +02:00
Alexander Neumann 1cedff0e9e tests: Simplify key handling 2015-05-03 16:36:52 +02:00
Alexander Neumann d753a5ffa3 archiver: correct benchmark 2015-05-01 22:58:50 +02:00
Florian Weingarten 0d9360a815 more cleanup 2015-04-29 22:33:34 -04:00
Alexander Neumann ec108fb708 Remove backend.Tree 2015-04-29 23:27:36 +02:00
Alexander Neumann 60a0fe8349 Major restructure, bundle blobs 
This commit also breaks the repository format.
 2015-04-29 23:27:32 +02:00
Alexander Neumann d19b23d4f1 Move Server and Key to new sub-package 2015-04-26 14:46:15 +02:00
Alexander Neumann 387082f983 Fix test BenchmarkChunkEncrypt 2015-04-25 00:54:15 +02:00
Alexander Neumann 3a2525809c Reorganize crypto code 
Move all crypto functions to package "crypto", move random generators
for tests into helper package.
 2015-04-12 09:36:14 +02:00
Alexander Neumann 4f4f3c421a Use package "restic/test" 2015-04-09 21:15:48 +02:00
Alexander Neumann b2307cafa2 Seed chunker with random per-repository polynomial 2015-04-06 00:22:19 +02:00
Alexander Neumann f157f775da Add checks for Server.Load(), use Blob for load 2015-03-28 15:07:08 +01:00
Alexander Neumann 5e69788eac Refactor backends 2015-03-28 14:36:50 +01:00
Alexander Neumann 91578fb2a3 Fix counting for tests for Preload 2015-03-10 15:58:23 +01:00
Alexander Neumann 6d5f1c0197 Add test for incremental backup 2015-03-10 14:17:54 +01:00
Alexander Neumann 14298fe232 Cache blobs for each snapshot 2015-03-09 23:50:32 +01:00
Alexander Neumann f167366be5 Add walk for trees, restructure 2015-03-02 19:58:06 +01:00
Alexander Neumann c8be54564f Add local cache 2015-03-02 10:04:01 +01:00
Alexander Neumann a59b0ec1f6 Add progress report for loading blobs 2015-02-21 15:32:48 +01:00
Alexander Neumann 58cded6b75 Refactor progress reporting 2015-02-21 14:25:06 +01:00
Alexander Neumann f214dce87c Add tests for Archiver.Preload() and a few more 2015-02-18 22:48:59 +01:00
Alexander Neumann 2462f93533 Disable Tests when benchdir is unset 2015-02-17 23:40:37 +01:00
Alexander Neumann 215fb982c6 Do not run archiver tests by default 2015-02-17 23:23:12 +01:00
Alexander Neumann 4bb7f2f2ed Add Preloader for blobs 2015-02-17 22:39:44 +01:00
Alexander Neumann 43ae0b69ab Fix benchmarks for ChunkEncrypt 2015-02-17 20:02:43 +01:00
Alexander Neumann 0a164ad5e0 Add test and benchmark for Scanner 2015-02-16 23:13:23 +01:00
Alexander Neumann f8f8107d55 wip 2015-02-15 15:17:24 +01:00
Alexander Neumann 93abaf204a Fix get_random for count != 4k, add randomReader() 2015-02-11 17:38:51 +01:00
Alexander Neumann 869ba83c6d Add Benchmarks for Scanner and Archiver 2015-02-10 21:59:18 +01:00
Alexander Neumann 9fcd23bd38 Add parallel benchmark for chunk+encrypt 2015-02-09 23:39:16 +01:00
Alexander Neumann bda33e612c Implement streaming chunker using io.Reader 2015-02-09 00:28:54 +01:00
Alexander Neumann e2fea0d088 Rename khepri -> restic 2014-12-05 21:45:49 +01:00
Alexander Neumann 719e121c74 Use large, dynamic buffer for encrypting maps 2014-11-23 22:58:41 +01:00
Alexander Neumann f95788ed90 Limit memory usage, add several sync.Pool 2014-11-23 21:26:20 +01:00
Alexander Neumann 1ac4f92299 Add benchmark for Chunk+Encrypt 2014-11-17 23:52:31 +01:00