1
0
Fork 0
mirror of https://github.com/restic/restic.git synced 2024-12-25 01:06:39 +00:00
No description
Find a file
Michael Pratt 9fa4f5eb6b gs: disable resumable uploads
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
2017-10-17 21:12:04 -07:00
.github Improve issue template 2017-10-07 12:16:46 +02:00
cmd/restic dump: Allow absolute paths 2017-10-16 20:22:01 +02:00
doc Add 'dump' to manual 2017-10-16 20:24:47 +02:00
docker install fuse and ca-certificates 2017-06-23 10:38:19 +02:00
internal gs: disable resumable uploads 2017-10-17 21:12:04 -07:00
scripts Add release script 2017-09-26 22:14:58 +02:00
vendor Add dependency cenkalti/backoff 2017-10-14 15:56:25 +02:00
.gitignore Update other files 2017-07-23 14:40:05 +02:00
.hound.yml
.travis.yml travis: update go versions 2017-09-17 19:02:22 +02:00
appveyor.yml Update Go versions for CI 2017-09-02 09:29:02 +02:00
build.go Upgrade min Go version to 1.8 2017-08-06 21:47:04 +02:00
CHANGELOG.md gs: disable resumable uploads 2017-10-17 21:12:04 -07:00
CONTRIBUTING.md Remove integration test for manpage 2017-09-26 13:16:55 +02:00
Gopkg.lock Add dependency cenkalti/backoff 2017-10-14 15:56:25 +02:00
Gopkg.toml Update vendored dependencies 2017-10-01 10:13:39 +02:00
LICENSE
Makefile test 2017-09-09 16:33:51 +02:00
README.rst Fix links to manual 2017-10-13 20:01:19 +02:00
run_integration_tests.go Remove integration test for manpage 2017-09-26 13:16:55 +02:00
VERSION Add VERSION file for 0.7.3 2017-09-20 20:50:07 +02:00

|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks|

Introduction
------------

restic is a backup program that is fast, efficient and secure.

For detailed usage and installation instructions check out the `documentation <https://restic.readthedocs.io/en/latest>`__.

You can ask questions in our `Discourse forum <https://forum.restic.net>`__.

Quick start
-----------

Once you've `installed
<https://restic.readthedocs.io/en/latest/020_installation.html>`__ restic, start
off with creating a repository for your backups:

.. code-block:: console

    $ restic init --repo /tmp/backup
    enter password for new backend:
    enter password again:
    created restic backend 085b3c76b9 at /tmp/backup
    Please note that knowledge of your password is required to access the repository.
    Losing your password means that your data is irrecoverably lost.

and add some data:

.. code-block:: console

    $ restic -r /tmp/backup backup ~/work
    enter password for repository:
    scan [/home/user/work]
    scanned 764 directories, 1816 files in 0:00
    [0:29] 100.00%  54.732 MiB/s  1.582 GiB / 1.582 GiB  2580 / 2580 items  0 errors  ETA 0:00
    duration: 0:29, 54.47MiB/s
    snapshot 40dc1520 saved

Next you can either use ``restic restore`` to restore files or use ``restic
mount`` to mount the repository via fuse and browse the files from previous
snapshots.

For more options check out the `online documentation <https://restic.readthedocs.io/en/latest/>`__.

Backends
--------

Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:

- `Local directory <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local>`__
- `sftp server (via SSH) <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp>`__
- `HTTP REST server <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server>`__ (`protocol <doc/rest_backend.rst>`__ `rest-server <https://github.com/restic/rest-server>`__)
- `AWS S3 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3>`__ (either from Amazon or using the `Minio <https://minio.io>`__ server)
- `OpenStack Swift <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift>`__
- `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
- `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
- `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__

Design Principles
-----------------

Restic is a program that does backups right and was designed with the
following principles in mind:

-  **Easy:** Doing backups should be a frictionless process, otherwise
   you might be tempted to skip it. Restic should be easy to configure
   and use, so that, in the event of a data loss, you can just restore
   it. Likewise, restoring data should not be complicated.

-  **Fast**: Backing up your data with restic should only be limited by
   your network or hard disk bandwidth so that you can backup your files
   every day. Nobody does backups if it takes too much time. Restoring
   backups should only transfer data that is needed for the files that
   are to be restored, so that this process is also fast.

-  **Verifiable**: Much more important than backup is restore, so restic
   enables you to easily verify that all data can be restored.

-  **Secure**: Restic uses cryptography to guarantee confidentiality and
   integrity of your data. The location the backup data is stored is
   assumed not to be a trusted environment (e.g. a shared space where
   others like system administrators are able to access your backups).
   Restic is built to secure your data against such attackers.

-  **Efficient**: With the growth of data, additional snapshots should
   only take the storage of the actual increment. Even more, duplicate
   data should be de-duplicated before it is actually written to the
   storage back end to save precious backup space.

Reproducible Builds
-------------------

The binaries released with each restic version starting at 0.6.1 are
`reproducible <https://reproducible-builds.org/>`__, which means that you can
easily reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
`builder repository <https://github.com/restic/builder>`__.

News
----

You can follow the restic project on Twitter `@resticbackup <https://twitter.com/resticbackup>`__ or by subscribing to
the `development blog <https://restic.github.io/blog/>`__.

License
-------

Restic is licensed under "BSD 2-Clause License". You can find the
complete text in ``LICENSE``.

.. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
   :target: https://restic.readthedocs.io/en/latest/?badge=latest
.. |Build Status| image:: https://travis-ci.org/restic/restic.svg?branch=master
   :target: https://travis-ci.org/restic/restic
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
   :target: https://ci.appveyor.com/project/fd0/restic/branch/master
.. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
   :target: https://goreportcard.com/report/github.com/restic/restic
.. |Say Thanks| image:: https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg
   :target: https://saythanks.io/to/restic