mirror
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Releases Wiki Activity
1,279 Commits 7 Branches 49 Tags 92 MiB
Commit Graph

45 Commits

Author SHA1 Message Date
Alexander Neumann 4f1f03cdb9 Move testing for known blobs to Archiver 
This removes the list of in-flight blobs from the master index and
instead keeps a list of "known" blobs in the Archiver. "known" here
means: either already processed, or included in an index. This property
is tested atomically, when the blob is not in the list of "known" blobs,
it is added to the list and the caller is responsible to make this
happen (i.e. save the blob).
 2016-02-01 23:50:56 +01:00
Alexander Neumann ac2fe4e04f Remove BlobWriter 2016-01-24 18:53:39 +01:00
Alexander Neumann cfdd3a853d Remove usage of CreateEncryptedBlob() 2016-01-24 18:52:11 +01:00
Alexander Neumann 01e40e62bf repo: Use Save() instead of Create() 2016-01-24 18:50:41 +01:00
Alexander Neumann 782a1bf7b0 repository: remove GetDecryptReader() 2016-01-24 00:12:17 +01:00
Alexander Neumann 3191778d33 repository: Use Load() instead of GetReader() 2016-01-24 00:12:09 +01:00
Alexander Neumann 8b7bf8691d backend: Remove Get() 
This is the first commit that removes the (redundant) Get() method of
the backend interface. Get(x, y) is equivalent to GetReader(x, y, 0, 0).
 2016-01-23 13:13:05 +01:00
Alexander Neumann f53008d916 Allow saving duplicate blobs in the repacker 
This adds code to the master index to allow saving duplicate blobs
within the repacker. In this mode, only the list of currently in flight
blobs is consulted, and not the index. This correct because while
repacking, a unique list of blobs is saved again to the index.
 2016-01-17 21:14:55 +01:00
Alexander Neumann 8209bb309b split out decryptReader and packerManager 2015-11-29 14:29:59 +01:00
Alexander Neumann 5b601f00b1 Add error checking 2015-11-29 14:25:57 +01:00
Alexander Neumann 567de35df4 Save new packs in index atomically 
This commit fixes a situation reported by a user where two indexes
contained information about the same pack without overlap, e.g.:

Index 3e6a32 contained:

    {
      "id": "c02e3b",
      "blobs": [
        {
          "id": "8114b1",
          "type": "data",
          "offset": 0,
          "length": 530107
        }
      ]
    }

And index 62da5f contained:

    {
      "id": "c02e3b",
      "blobs": [
        {
          "id": "e344f8",
          "type": "data",
          "offset": 1975848,
          "length": 3426468
        },
        {
          "id": "939ed9",
          "type": "data",
          "offset": 530107,
          "length": 1445741
        }
      ]
    }

This commit adds all blobs in a pack in one atomic operation so that
intermediate such as these do not happen.
 2015-11-20 22:56:56 +01:00
Alexander Neumann 1fc0d78913 Refactor Index.Store() to take a PackedBlob 2015-11-02 19:07:03 +01:00
Alexander Neumann f3f84b1544 Add ID handling for index 2015-11-02 18:52:13 +01:00
Alexander Neumann 60a34087c9 Move LoadIndexWithDecoder to index.go 2015-11-02 18:52:13 +01:00
Alexander Neumann fccde030d5 Refactor Index.Lookup() to return struct PackedBlob 2015-10-31 14:47:42 +01:00
Alexander Neumann 461d54e43c Refactor repository.SaveIndex() 2015-10-25 17:05:54 +01:00
Alexander Neumann f7ff5b766c Mark written indexes as finalized 2015-10-25 15:35:18 +01:00
Alexander Neumann af0d6f58b9 Remove unneeded pointer to pack id 2015-10-25 14:35:08 +01:00
Alexander Neumann 1dd731fdb8 Handle concurrent access to the inFlight list 2015-10-14 20:50:54 +02:00
Alexander Neumann 6fa4be5af2 Regularly save intermediate indexes 2015-10-12 23:59:17 +02:00
Alexander Neumann 86fcd170f6 Add and use MasterIndex 2015-10-12 22:34:12 +02:00
Alexander Neumann eb73182fcf Rework index decode and handling old format 2015-10-12 20:53:07 +02:00
Alexander Neumann 356bb62243 Add CreateEncryptedBlob and GetDecryptReader 2015-10-12 20:53:07 +02:00
Alexander Neumann 96f2165067 Allow loading index with old format 2015-10-12 20:53:07 +02:00
Alexander Neumann 7944e8e323 Update index format 2015-10-12 20:53:07 +02:00
Alexander Neumann cc7acba02b Return the original backend ID on duplicate entries 2015-10-11 20:45:50 +02:00
Philipp Serr 7b11660f4f Prevent concurrent processing of same blob 
... by first adding a preliminary index entry and making this fail if
an index entry for the same blob already exists.

A preliminary index entry is characterized by not yet being associated
with a pack. Until now, these entries where added to the index just
like final index entries using index.Store, which silently overwrites
existing index entries.

This commit adds a new method index.StoreInProgress which refuses to
overwrite existing index entries and allows for creating preliminary
index entries only. The existing method index.Store has not been
changed and continues to silently overwrite existing index entries.
This distinction is important, as otherwise, it would be impossible to
update a preliminary index entry after the blob has been written to a
pack.

Resolves: restic#292
 2015-09-27 16:56:49 +02:00
Alexander Neumann b85927576b Address code review comments 2015-07-26 18:01:20 +02:00
Alexander Neumann bd746a0425 fuse: refactor and add tests for fuse.file 2015-07-26 16:43:42 +02:00
Alexander Neumann 55ddd5317d Reduce memory usage for fuse mount 
This changes `repository.LoadBlob()` so that a destination buffer must
be provided, which enables the fuse code to use a buffer from a
`sync.Pool`. In addition, release the buffers when the file is closed.
At the moment, the max memory usage is defined by the max file size that
is read in one go (e.g. with `cat`). It could be further optimized by
implementing a LRU caching scheme.
 2015-07-26 14:25:59 +02:00
Alexander Neumann 5cdcc99eba Use array instead of hash for backend.ID 
Since backend.ID is always a slice of constant length, use an array
instead of a slice. Mostly, arrays behave as slices, except that an
array cannot be nil, so use `*backend.ID` insteaf of `backend.ID` in
places where the absence of an ID is possible (e.g. for the Subtree of a
Node, which may not present when the node is a file node).

This change allows to directly use backend.ID as the the key for a map,
so that arbitrary data structures (e.g. a Set implemented as a
map[backend.ID]struct{}) can easily be formed.
 2015-07-25 18:01:57 +02:00
Florian Weingarten 36a62cf77d Extract chunker 2015-07-08 16:58:23 -04:00
Alexander Neumann 5c1fe5784a repository: Load index in parallel 2015-07-05 11:07:10 +02:00
Alexander Neumann 58600e3bb8 repository: Add comments, silence golint 2015-07-05 11:07:10 +02:00
Alexander Neumann 9a02148daf repository: Rename Save() -> SaveAndEncrypt() 2015-07-05 11:07:10 +02:00
Alexander Neumann 03983f0907 repository: Rename Load() -> LoadAndDecrypt() 2015-07-05 11:07:10 +02:00
Alexander Neumann c553a57e0d repository: Refactor Config 2015-07-05 11:07:10 +02:00
Alexander Neumann 867f6c8e24 repository: Rename receiver: s -> r 
No functional changes.
 2015-07-05 11:07:10 +02:00
Alexander Neumann 0ad3d71f01 repository: Add more debug to Create() 2015-06-28 13:52:22 +02:00
Alexander Neumann f773feeb04 Remove Repository.Close() and .Location() 2015-05-17 20:58:22 +02:00
Alexander Neumann af381c2ab0 Remove Repository.Remove() 2015-05-17 20:58:22 +02:00
Alexander Neumann eb6dfcf58c Remove Repository.Test() 2015-05-17 20:58:22 +02:00
Alexander Neumann 6e38a8a033 Move FindSnapshot, make Repository.List() return IDs 2015-05-17 20:58:22 +02:00
Alexander Neumann 95536e8a21 Rename 'Repo' -> 'Repository' 2015-05-09 23:59:58 +02:00
Alexander Neumann 232c472836 Move package 'repo' to package 'repository' 2015-05-09 23:52:03 +02:00
Renamed from repo/repository.go (Browse further)