mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
1,266 Commits 7 Branches 48 Tags 92 MiB
Commit Graph

185 Commits

Author SHA1 Message Date
Alexander Neumann d79c85af62 Fix s3 tests 2016-01-17 18:48:05 +01:00
Alexander Neumann 407819e5a9 s3: properly integrate minio-go lib 2016-01-17 18:48:05 +01:00
Alexander Neumann 1922a4272c s3: fix usage 
Ignore error response for existing bucket, add more debug code.
 2016-01-17 18:48:05 +01:00
Alexander Neumann 2b0b44c5ce s3: implement open with config 2016-01-17 18:48:05 +01:00
Alexander Neumann f7c9091970 sftp: implement open with config 2016-01-17 18:48:05 +01:00
Alexander Neumann 7b1e8fdd06 local: correct comment 2016-01-17 18:48:05 +01:00
Alexander Neumann de933a1d48 Rename URI -> Config/Location 2016-01-17 18:48:05 +01:00
Alexander Neumann 566a15285a Add repository location parsing code 2016-01-17 18:48:05 +01:00
Alexander Neumann edfb31f4fe s3: Run integration test with minio server 2016-01-17 18:47:24 +01:00
Alexander Neumann 5736742c3e s3: Open() creates bucket if it does not exist 2016-01-17 18:46:08 +01:00
Alexander Neumann 248f991ad4 s3: don't remove the bucket on Delete() 2016-01-17 18:46:08 +01:00
Alexander Neumann 55f10eb1c1 Fix s3 test with local minio server instance 2016-01-17 18:46:08 +01:00
Alexander Neumann d0ca118387 Fix usage of the `done` chan 2016-01-17 18:46:08 +01:00
Chris Howey 69a9adc4c3 Use local instance of minio server. 
Need to figure out how to have tests automatically start and kill
server.
 2016-01-17 18:46:08 +01:00
Chris Howey e2445f4c97 GetPartialObject does not work. 2016-01-17 18:46:08 +01:00
Chris Howey ed2a4ba1d5 Fix s3 backend test 2016-01-17 18:46:08 +01:00
Chris Howey 6d1552af51 Switch s3 library to allow for s3 compatible backends. Fixes #315 2016-01-17 18:46:08 +01:00
Alexander Neumann 8d229bfd21 Make ReadCloser public 2015-12-02 22:37:58 +01:00
Alexander Neumann 4f6bc754b8 MemBackend: Add Delete() and more debug 2015-11-29 14:53:02 +01:00
Alexander Neumann 26697a0223 Fix MemoryBackend GetReader() method 2015-11-29 14:52:19 +01:00
Alexander Neumann 480054bc3a MemoryBackend: handle config correctly, add tests for that 2015-11-29 14:52:19 +01:00
Alexander Neumann 538e5878a1 add debug logging to MemoryBackend 2015-11-29 14:52:19 +01:00
Alexander Neumann 9cb4e14327 add MemBackend and MockBackend 2015-11-29 14:52:19 +01:00
Alexander Neumann da71da23d9 Add MockBackend 2015-11-29 14:52:19 +01:00
Alexander Neumann 0d5731383f Remove HashAppendWriter 2015-11-29 14:29:59 +01:00
Alexander Neumann 4fd7676e92 HashingWriter: Add documentation 2015-11-29 14:29:59 +01:00
Alexander Neumann acba82c8f7 Merge pull request #252 from restic/repack-blobs 
WIP: Repack blobs
 2015-11-09 20:57:57 +01:00
Jan Bader af960b9b40 Simplify Implementation of Join 2015-11-03 18:48:51 +01:00
Jan Bader d09e6d5b0f Fix missing Join calls 2015-11-03 18:47:01 +01:00
Alexander Neumann 30cf002574 Sort IDSet.List() 2015-11-02 18:52:13 +01:00
Alexander Neumann 181963ba08 Fix IDSet.String() 2015-11-02 17:36:05 +01:00
Jan Bader 81ec7337e0 Always use forward slashes in SFTP (Fixes #334) 
Add custom Join func that always uses forward slashes in SFTP
 2015-11-02 14:53:42 +01:00
Alexander Neumann 5d617edbbf local/sftp backend: Do not seek if offset is 0 2015-10-25 17:51:26 +01:00
Alexander Neumann d9f9b77d68 Add Index.Packs() and IDSet.Equals() 2015-10-25 15:28:01 +01:00
Alexander Neumann 5c46dc41de Add methods to IDSet 2015-09-05 18:49:28 +02:00
Alexander Neumann 2cb0fbf589 backend: Add String() to IDs 2015-09-05 18:41:58 +02:00
Alexander Neumann 3063ad1d05 Split id.go into several files 2015-09-05 18:41:56 +02:00
Alexander Neumann 0a457eafed Correctly test for config file 2015-08-26 22:06:52 +02:00
Alexander Neumann b211f834fa Remove tests for directories 
For testing whether a repository already exists it is sufficient to
test if the config file (and therefore the master key) exists.

Closes #279
 2015-08-26 21:51:40 +02:00
Chris Howey ccb2f00b8a typo 2015-08-26 07:54:39 -05:00
Chris Howey 3bf447b422 Update tests for new s3 lib 2015-08-26 07:44:00 -05:00
Chris Howey 10cd672a92 Use new version of s3 library, Fixes #276 2015-08-26 06:25:05 -05:00
Alexander Neumann 2644722198 Reset read-only flag before removing 2015-08-19 22:02:47 +02:00
Alexander Neumann 9367c57bcc Merge pull request #267 from bchapuis/fix-254 
test the GetReader() function
 2015-08-18 22:48:33 +02:00
Chapuis Bertil bc5e0fa68f test the reader obtained with GetReader() 2015-08-18 17:57:06 +02:00
Chapuis Bertil 513be4b8ab test the GetReader() function 2015-08-18 15:15:17 +02:00
Klaus Post 0e7d0d8dba Create ignoreSigIntProcAttr() 
Retrieve the SysProcAttr from a separate function. Completely eliminates syscall from main file.
 2015-08-16 12:51:01 +02:00
Klaus Post 520b1b65b0 Create setNewFileMode function. 
Create separate files with setNewFileMode to avoid
runtime checks.
 2015-08-16 12:39:38 +02:00
Klaus Post 2dcb527828 Only use Setsid on Unix. 
Setsid is not a part of syscall.SysProcAttr on Windows, so we only set that on
systems that have it.
 2015-08-14 15:32:07 +02:00
Klaus Post 7c84d810d3 Make sure we can delete files on Windows. 
Files must be closed on Windows before they can be deleted.
Therefore we keep track of all open files, and closes them before
we delete them.

Also we don't set finished blobs to read-only on Windows, since
that prevents us from deleting them.
 2015-08-14 15:30:36 +02:00
Alexander Neumann 5cdcc99eba Use array instead of hash for backend.ID 
Since backend.ID is always a slice of constant length, use an array
instead of a slice. Mostly, arrays behave as slices, except that an
array cannot be nil, so use `*backend.ID` insteaf of `backend.ID` in
places where the absence of an ID is possible (e.g. for the Subtree of a
Node, which may not present when the node is a file node).

This change allows to directly use backend.ID as the the key for a map,
so that arbitrary data structures (e.g. a Set implemented as a
map[backend.ID]struct{}) can easily be formed.
 2015-07-25 18:01:57 +02:00
Alexander Neumann 1c14be8993 backend: Remove IDSet 
This can easily be replaced by a map[{32}byte]struct{}, and this is much
faster.
 2015-07-11 16:12:50 +02:00
Alexander Neumann 281eea9c05 sftp: Use own process group for ssh subprocess 2015-07-05 11:07:10 +02:00
Emil Hessman 52b6cfbbfa backend: fix doc typos 2015-06-28 09:44:06 +02:00
Alexander Neumann 6c2d6bfd11 Merge pull request #184 from howeyc/backend-s3 
Add Amazon S3 backend.
 2015-06-23 20:14:47 +02:00
Alexander Neumann 9853fbcf48 Remove more flags from tests 2015-06-21 17:40:21 +02:00
Alexander Neumann 030f08a410 Remove flags from tests 2015-06-21 17:40:21 +02:00
Alexander Neumann 12677b4f8a Use flag instead of build tag to run integration tests 2015-06-21 17:40:21 +02:00
Alexander Neumann 9b7db4df24 travis: Add new integration tests 2015-06-21 17:40:21 +02:00
Chris Howey cf0531a2b1 Pass buffer instead of byte slice 
s3 library creates internal buffer pass in byte slice
 2015-06-14 09:05:20 -05:00
Chris Howey 12596c6841 Fix test to use new type name on backend 2015-06-14 08:43:47 -05:00
Chris Howey 57ee1db8aa Rename backend type to S3Backend 2015-06-14 08:28:50 -05:00
Chris Howey e963faf846 Place all restic data under a key prefix. 
Deleting a restic repository now only deletes restic keys, leaving the
rest of the bucket in tact.
 2015-06-14 08:28:50 -05:00
Chris Howey d30a95e8ad Return bucket readers instead of buffering a key on read. 2015-06-14 08:28:50 -05:00
Chris Howey aaae7f33d3 Have number of connections limited by channel 
Removes previous limit of 1 connection
 2015-06-14 08:28:50 -05:00
Chris Howey 2350419f59 Modify test suite to use s3 test server. 
Fix s3 backend to return error on attempting to write to already
existing key.
 2015-06-14 08:28:50 -05:00
Chris Howey 4aa75994b0 Add ability to open bucket using S3 object for testing purposes. 2015-06-14 08:28:50 -05:00
Chris Howey d1554d1133 Add backend test for s3 2015-06-14 08:28:50 -05:00
Chris Howey bfe221e71c Handle long list responses from s3. 
Also change naming of backend struct.
 2015-06-14 08:28:50 -05:00
Chris Howey 47186806cf Add Amazon S3 backend. 2015-06-14 08:28:50 -05:00
Alexander Neumann 6e38a8a033 Move FindSnapshot, make Repository.List() return IDs 2015-05-17 20:58:22 +02:00
Alexander Neumann 4e5094b7c9 Fix loading config on sftp backend 2015-05-14 18:08:45 +02:00
Alexander Neumann 9ac6a1d232 Add minimal documentation for backend/{local,sftp} 2015-05-09 12:54:45 +02:00
Alexander Neumann e4e502959f Clarify documentation on backend 2015-05-09 12:53:04 +02:00
Alexander Neumann c57c4f0b8f Merge branch 'add_config' 2015-05-04 22:13:59 +02:00
Alexander Neumann 5399358272 Fix spelling errors in comments 2015-05-04 20:39:45 +02:00
Alexander Neumann 250080db9c sftp: Annotate a few errors 2015-05-03 21:06:20 +02:00
Alexander Neumann 2fb1783885 backend/server: remove id and version from backend 2015-05-03 18:56:16 +02:00
Alexander Neumann 6fe38b0cd8 fix comments to have the standard form 2015-05-02 16:22:43 +02:00
Alexander Neumann 1f94ac724f backend: rename LimitReader to LimitReadCloser 2015-05-02 16:13:56 +02:00
Alexander Neumann 98dc811536 Add nil-receiver awareness to all Reader/Writer 2015-05-01 17:32:42 +02:00
Alexander Neumann ec108fb708 Remove backend.Tree 2015-04-29 23:27:36 +02:00
Alexander Neumann 60a0fe8349 Major restructure, bundle blobs 
This commit also breaks the repository format.
 2015-04-29 23:27:32 +02:00
Alexander Neumann 87ce38b13a Add type 'Index' to backend 2015-04-29 23:27:08 +02:00
Alexander Neumann edfd86624c Fix "cleanup" flag 2015-04-26 15:40:25 +02:00
Florian Weingarten ebae478631 Prefer mixedCaps over underscore 2015-04-24 19:39:32 -04:00
Alexander Neumann 4f4f3c421a Use package "restic/test" 2015-04-09 21:15:48 +02:00
Alexander Neumann 24b14e21cc Remove pools for nodes and IDs 
This removes the allocation pools for nodes and IDs. I feel they aren't
really needed. Benchmarks:

    benchmark                         old ns/op      new ns/op      delta
    BenchmarkChunkEncrypt             197890867      198616293      +0.37%
    BenchmarkChunkEncryptParallel     196127004      198819818      +1.37%
    BenchmarkArchiveDirectory         1098848419     1087237723     -1.06%
    BenchmarkPreload                  30464455       29910239       -1.82%
    BenchmarkLoadTree                 3265092        3088543        -5.41%
    BenchmarkEncryptWriter            37213511       37134683       -0.21%
    BenchmarkEncrypt                  36037879       36166546       +0.36%
    BenchmarkDecryptReader            38165659       38556734       +1.02%
    BenchmarkEncryptDecryptReader     77027044       77194987       +0.22%
    BenchmarkDecrypt                  36017602       35937888       -0.22%
    BenchmarkSaveJSON                 47906          50270          +4.93%
    BenchmarkSaveFrom                 49775973       50520969       +1.50%
    BenchmarkLoadJSONID               105290245      107281849      +1.89%
    BenchmarkChunkerWithSHA256        151501430      148264078      -2.14%
    BenchmarkChunkerWithMD5           93606346       94036392       +0.46%
    BenchmarkChunker                  74285431       75933882       +2.22%
    BenchmarkPipelineWalker           387689         346467         -10.63%

    benchmark                         old MB/s     new MB/s     speedup
    BenchmarkChunkEncrypt             52.99        52.79        1.00x
    BenchmarkChunkEncryptParallel     53.46        52.74        0.99x
    BenchmarkEncryptWriter            225.42       225.90       1.00x
    BenchmarkEncrypt                  232.77       231.94       1.00x
    BenchmarkDecryptReader            219.79       217.57       0.99x
    BenchmarkEncryptDecryptReader     108.90       108.67       1.00x
    BenchmarkDecrypt                  232.90       233.42       1.00x
    BenchmarkSaveFrom                 84.26        83.02        0.99x
    BenchmarkChunkerWithSHA256        69.21        70.72        1.02x
    BenchmarkChunkerWithMD5           112.02       111.51       1.00x
    BenchmarkChunker                  141.15       138.09       0.98x

    benchmark                         old allocs     new allocs     delta
    BenchmarkChunkEncrypt             110            110            +0.00%
    BenchmarkChunkEncryptParallel     100            100            +0.00%
    BenchmarkArchiveDirectory         475591         476635         +0.22%
    BenchmarkPreload                  28059          24182          -13.82%
    BenchmarkLoadTree                 3124           2889           -7.52%
    BenchmarkEncryptWriter            19             19             +0.00%
    BenchmarkEncrypt                  13             13             +0.00%
    BenchmarkDecryptReader            16             15             -6.25%
    BenchmarkEncryptDecryptReader     39             39             +0.00%
    BenchmarkDecrypt                  11             11             +0.00%
    BenchmarkSaveJSON                 74             74             +0.00%
    BenchmarkSaveFrom                 109            112            +2.75%
    BenchmarkLoadJSONID               103630         97849          -5.58%
    BenchmarkChunkerWithSHA256        13             13             +0.00%
    BenchmarkChunkerWithMD5           12             12             +0.00%
    BenchmarkChunker                  6              6              +0.00%
    BenchmarkPipelineWalker           212            165            -22.17%

    benchmark                         old bytes     new bytes     delta
    BenchmarkChunkEncrypt             64697         64697         +0.00%
    BenchmarkChunkEncryptParallel     64681         64681         +0.00%
    BenchmarkArchiveDirectory         193385504     193790864     +0.21%
    BenchmarkPreload                  4064701       3942000       -3.02%
    BenchmarkLoadTree                 344954        325396        -5.67%
    BenchmarkEncryptWriter            12793         12793         +0.00%
    BenchmarkEncrypt                  1950          1950          +0.00%
    BenchmarkDecryptReader            3120          2774          -11.09%
    BenchmarkEncryptDecryptReader     1528036       1528036       +0.00%
    BenchmarkDecrypt                  1919          1919          +0.00%
    BenchmarkSaveJSON                 5524          5524          +0.00%
    BenchmarkSaveFrom                 31353         40804         +30.14%
    BenchmarkLoadJSONID               12872020      16010968      +24.39%
    BenchmarkChunkerWithSHA256        26821         26821         +0.00%
    BenchmarkChunkerWithMD5           13554         13554         +0.00%
    BenchmarkChunker                  13458         13458         +0.00%
    BenchmarkPipelineWalker           58584         55560         -5.16%
 2015-03-28 16:51:37 +01:00
Alexander Neumann 702b6cfc1e Make ID.Str() handle nil ids correctly 2015-03-28 16:46:21 +01:00
Alexander Neumann 2abda75404 Refactor mockBackend for test 2015-03-28 16:42:35 +01:00
Alexander Neumann 223108c6ae Remove unused vars and const 2015-03-28 15:28:08 +01:00
Alexander Neumann 5e69788eac Refactor backends 2015-03-28 14:36:50 +01:00
Alexander Neumann 12c8522b33 Fix file mode for data blobs 2015-03-14 20:19:52 +01:00
Alexander Neumann 23ac4f84dd Set file-mode in repository read-only (Closes #48) 2015-03-14 14:30:26 +01:00
Alexander Neumann f69a39cff5 Add ID to repository 
This allows identifying a repository regardless if it's accessed over
SFTP or locally. Introduced for having a per-repository cache.
 2015-03-14 12:06:27 +01:00
Alexander Neumann b6f25aa690 Remove Each(), add basic stats 2015-02-17 23:11:56 +01:00
Alexander Neumann c631b5c738 Add Len() functions for IDSet and Map 2015-02-17 22:39:16 +01:00
Alexander Neumann 06ed5c12b8 Remove Compress() and Uncompress() from backend 2015-02-16 00:30:51 +01:00
Alexander Neumann fb2cd7b485 HashingWriter: Track size 2015-02-16 00:24:43 +01:00
Alexander Neumann b816192d1c backend: Throw error when Close() is called twice 2015-02-16 00:24:02 +01:00