From 780a7d816a222bd91168c2a5fd98eb5f5e814321 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 11:08:19 -0400 Subject: [PATCH 1/7] remove mention of "borg" in changelog title we know what we are looking at already... also demote the attic changelog so it doesn't show in the main table of contents. --- docs/changes.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/changes.rst b/docs/changes.rst index 5f347dfea..17b2ce450 100644 --- a/docs/changes.rst +++ b/docs/changes.rst @@ -1,5 +1,5 @@ -Borg Changelog -============== +Changelog +========= Version 0.27.0 -------------- @@ -345,20 +345,20 @@ Other changes: Attic Changelog -=============== +--------------- Here you can see the full list of changes between each Attic release until Borg forked from Attic: Version 0.17 ------------- +~~~~~~~~~~~~ (bugfix release, released on X) - Fix hashindex ARM memory alignment issue (#309) - Improve hashindex error messages (#298) Version 0.16 ------------- +~~~~~~~~~~~~ (bugfix release, released on May 16, 2015) - Fix typo preventing the security confirmation prompt from working (#303) @@ -368,7 +368,7 @@ Version 0.16 - Fix parsing of iso 8601 timestamps with zero microseconds (#282) Version 0.15 ------------- +~~~~~~~~~~~~ (bugfix release, released on Apr 15, 2015) - xattr: Be less strict about unknown/unsupported platforms (#239) @@ -382,7 +382,7 @@ Version 0.15 - Include missing pyx files in dist files (#168) Version 0.14 ------------- +~~~~~~~~~~~~ (feature release, released on Dec 17, 2014) @@ -396,7 +396,7 @@ Version 0.14 - Fix issue with empty xattr values (#106) Version 0.13 ------------- +~~~~~~~~~~~~ (feature release, released on Jun 29, 2014) @@ -412,7 +412,7 @@ Version 0.13 - Fix Python 3.2 specific lockf issue (EDEADLK) Version 0.12 ------------- +~~~~~~~~~~~~ (feature release, released on April 7, 2014) @@ -429,7 +429,7 @@ Version 0.12 - Switch to SI units (Power of 1000 instead 1024) when printing file sizes Version 0.11 ------------- +~~~~~~~~~~~~ (feature release, released on March 7, 2014) @@ -444,7 +444,7 @@ Version 0.11 - Ignore xattr errors during "extract" if not supported by the filesystem. (#46) Version 0.10 ------------- +~~~~~~~~~~~~ (bugfix release, released on Jan 30, 2014) @@ -454,7 +454,7 @@ Version 0.10 - Make source code endianness agnostic (#1) Version 0.9 ------------ +~~~~~~~~~~~ (feature release, released on Jan 23, 2014) @@ -467,14 +467,14 @@ Version 0.9 - Improved libcrypto path detection (#23). Version 0.8.1 -------------- +~~~~~~~~~~~~~ (bugfix release, released on Oct 4, 2013) - Fix segmentation fault issue. Version 0.8 ------------ +~~~~~~~~~~~ (feature release, released on Oct 3, 2013) @@ -487,7 +487,7 @@ Version 0.8 Version 0.7 ------------ +~~~~~~~~~~~ (feature release, released on Aug 5, 2013) @@ -498,7 +498,7 @@ Version 0.7 Version 0.6.1 -------------- +~~~~~~~~~~~~~ (bugfix release, released on July 19, 2013) @@ -506,6 +506,6 @@ Version 0.6.1 Version 0.6 ------------ +~~~~~~~~~~~ First public release on July 9, 2013 From 500c2a8a204f9007e06b1c4bda8b82bc0adc530a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 11:09:28 -0400 Subject: [PATCH 2/7] fix warnings in docs generation --- README.rst | 3 +-- docs/development.rst | 4 ++-- docs/faq.rst | 3 ++- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.rst b/README.rst index 3a2088e2f..2181d7227 100644 --- a/README.rst +++ b/README.rst @@ -102,8 +102,7 @@ Now doing another backup, just to show off the great deduplication:: This archive: 57.16 MB 46.78 MB 151.67 kB <--- ! All archives: 114.02 MB 93.46 MB 44.81 MB -For a graphical frontend refer to our complementary project -`BorgWeb `_. +For a graphical frontend refer to our complementary project `BorgWeb`_. Links ===== diff --git a/docs/development.rst b/docs/development.rst index cce5cb5d3..6ae364dc1 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -136,11 +136,11 @@ Checklist: python setup.py register sdist upload --identity="Thomas Waldmann" --sign - close release milestone on Github -- announce on:: +- announce on: - `mailing list `_ - Twitter (follow @ThomasJWaldmann for these tweets) - - `IRC channel `_ (change ``/topic`` + - `IRC channel `_ (change ``/topic``) - create a Github release, include: * standalone binaries (see above for how to create them) diff --git a/docs/faq.rst b/docs/faq.rst index 3316d96f8..b91666af9 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -5,7 +5,7 @@ Frequently asked questions ========================== Can I backup VM disk images? - Yes, the :ref:`deduplication ` technique used by + Yes, the `deduplication`_ technique used by |project_name| makes sure only the modified parts of the file are stored. Also, we have optional simple sparse file support for extract. @@ -52,6 +52,7 @@ Which file types, attributes, etc. are *not* preserved? Why is my backup bigger than with attic? Why doesn't |project_name| do compression by default? + * attic was rather unflexible when it comes to compression, it always compressed using zlib level 6 (no way to switch compression off or adjust the level or algorithm) From b122fca58088c44906888674e11fa0eabd142d1a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 11:12:14 -0400 Subject: [PATCH 3/7] use titles instead of definitions in FAQ this way the titles show up in the table of contents and we can link to individual entries --- docs/faq.rst | 183 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 107 insertions(+), 76 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index b91666af9..4009def13 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -5,24 +5,30 @@ Frequently asked questions ========================== Can I backup VM disk images? - Yes, the `deduplication`_ technique used by - |project_name| makes sure only the modified parts of the file are stored. - Also, we have optional simple sparse file support for extract. +---------------------------- + +Yes, the `deduplication`_ technique used by +|project_name| makes sure only the modified parts of the file are stored. +Also, we have optional simple sparse file support for extract. Can I backup from multiple servers into a single repository? - Yes, but in order for the deduplication used by |project_name| to work, it - needs to keep a local cache containing checksums of all file - chunks already stored in the repository. This cache is stored in - ``~/.cache/borg/``. If |project_name| detects that a repository has been - modified since the local cache was updated it will need to rebuild - the cache. This rebuild can be quite time consuming. +------------------------------------------------------------ - So, yes it's possible. But it will be most efficient if a single - repository is only modified from one place. Also keep in mind that - |project_name| will keep an exclusive lock on the repository while creating - or deleting archives, which may make *simultaneous* backups fail. +Yes, but in order for the deduplication used by |project_name| to work, it +needs to keep a local cache containing checksums of all file +chunks already stored in the repository. This cache is stored in +``~/.cache/borg/``. If |project_name| detects that a repository has been +modified since the local cache was updated it will need to rebuild +the cache. This rebuild can be quite time consuming. + +So, yes it's possible. But it will be most efficient if a single +repository is only modified from one place. Also keep in mind that +|project_name| will keep an exclusive lock on the repository while creating +or deleting archives, which may make *simultaneous* backups fail. Which file types, attributes, etc. are preserved? +------------------------------------------------- + * Directories * Regular files * Hardlinks (considering all files in the same archive) @@ -40,6 +46,8 @@ Which file types, attributes, etc. are preserved? * BSD flags on OS X and FreeBSD Which file types, attributes, etc. are *not* preserved? +------------------------------------------------------- + * UNIX domain sockets (because it does not make sense - they are meaningless without the running process that created them and the process needs to recreate them in any case). So, don't panic if your backup @@ -50,8 +58,8 @@ Which file types, attributes, etc. are *not* preserved? Archive extraction has optional support to extract all-zero chunks as holes in a sparse file. -Why is my backup bigger than with attic? -Why doesn't |project_name| do compression by default? +Why is my backup bigger than with attic? Why doesn't |project_name| do compression by default? +---------------------------------------------------------------------------------------------- * attic was rather unflexible when it comes to compression, it always compressed using zlib level 6 (no way to switch compression off or @@ -63,90 +71,113 @@ Why doesn't |project_name| do compression by default? level you want to use. This is why compression defaults to none. How can I specify the encryption passphrase programmatically? - The encryption passphrase can be specified programmatically using the - `BORG_PASSPHRASE` environment variable. This is convenient when setting up - automated encrypted backups. Another option is to use - key file based encryption with a blank passphrase. See - :ref:`encrypted_repos` for more details. +------------------------------------------------------------- + +The encryption passphrase can be specified programmatically using the +`BORG_PASSPHRASE` environment variable. This is convenient when setting up +automated encrypted backups. Another option is to use +key file based encryption with a blank passphrase. See +:ref:`encrypted_repos` for more details. When backing up to remote encrypted repos, is encryption done locally? - Yes, file and directory metadata and data is locally encrypted, before - leaving the local machine. We do not mean the transport layer encryption - by that, but the data/metadata itself. Transport layer encryption (e.g. - when ssh is used as a transport) applies additionally. +---------------------------------------------------------------------- + +Yes, file and directory metadata and data is locally encrypted, before +leaving the local machine. We do not mean the transport layer encryption +by that, but the data/metadata itself. Transport layer encryption (e.g. +when ssh is used as a transport) applies additionally. When backing up to remote servers, do I have to trust the remote server? - Yes and No. - No, as far as data confidentiality is concerned - if you use encryption, - all your files/dirs data and metadata are stored in their encrypted form - into the repository. - Yes, as an attacker with access to the remote server could delete (or - otherwise make unavailable) all your backups. +------------------------------------------------------------------------ + +Yes and No. + +No, as far as data confidentiality is concerned - if you use encryption, +all your files/dirs data and metadata are stored in their encrypted form +into the repository. + +Yes, as an attacker with access to the remote server could delete (or +otherwise make unavailable) all your backups. If a backup stops mid-way, does the already-backed-up data stay there? - Yes, |project_name| supports resuming backups. - During a backup a special checkpoint archive named ``.checkpoint`` - is saved every checkpoint interval (the default value for this is 5 - minutes) containing all the data backed-up until that point. This means - that at most worth of data needs to be retransmitted - if a backup needs to be restarted. - Once your backup has finished successfully, you can delete all ``*.checkpoint`` - archives. +---------------------------------------------------------------------- + +Yes, |project_name| supports resuming backups. + +During a backup a special checkpoint archive named ``.checkpoint`` +is saved every checkpoint interval (the default value for this is 5 +minutes) containing all the data backed-up until that point. This means +that at most worth of data needs to be retransmitted +if a backup needs to be restarted. + +Once your backup has finished successfully, you can delete all ``*.checkpoint`` +archives. If it crashes with a UnicodeError, what can I do? - Check if your encoding is set correctly. For most POSIX-like systems, try:: +------------------------------------------------- - export LANG=en_US.UTF-8 # or similar, important is correct charset +Check if your encoding is set correctly. For most POSIX-like systems, try:: + + export LANG=en_US.UTF-8 # or similar, important is correct charset I can't extract non-ascii filenames by giving them on the commandline!? - This might be due to different ways to represent some characters in unicode - or due to other non-ascii encoding issues. - If you run into that, try this: +----------------------------------------------------------------------- - - avoid the non-ascii characters on the commandline by e.g. extracting - the parent directory (or even everything) - - mount the repo using FUSE and use some file manager +This might be due to different ways to represent some characters in unicode +or due to other non-ascii encoding issues. + +If you run into that, try this: + +- avoid the non-ascii characters on the commandline by e.g. extracting + the parent directory (or even everything) +- mount the repo using FUSE and use some file manager Can |project_name| add redundancy to the backup data to deal with hardware malfunction? - No, it can't. While that at first sounds like a good idea to defend against - some defect HDD sectors or SSD flash blocks, dealing with this in a - reliable way needs a lot of low-level storage layout information and - control which we do not have (and also can't get, even if we wanted). +--------------------------------------------------------------------------------------- - So, if you need that, consider RAID or a filesystem that offers redundant - storage or just make backups to different locations / different hardware. +No, it can't. While that at first sounds like a good idea to defend against +some defect HDD sectors or SSD flash blocks, dealing with this in a +reliable way needs a lot of low-level storage layout information and +control which we do not have (and also can't get, even if we wanted). - See also `ticket 225 `_. +So, if you need that, consider RAID or a filesystem that offers redundant +storage or just make backups to different locations / different hardware. + +See also `ticket 225 `_. Can |project_name| verify data integrity of a backup archive? - Yes, if you want to detect accidental data damage (like bit rot), use the - ``check`` operation. It will notice corruption using CRCs and hashes. - If you want to be able to detect malicious tampering also, use a encrypted - repo. It will then be able to check using CRCs and HMACs. +------------------------------------------------------------- + +Yes, if you want to detect accidental data damage (like bit rot), use the +``check`` operation. It will notice corruption using CRCs and hashes. +If you want to be able to detect malicious tampering also, use a encrypted +repo. It will then be able to check using CRCs and HMACs. Why was Borg forked from Attic? - Borg was created in May 2015 in response to the difficulty of getting new - code or larger changes incorporated into Attic and establishing a bigger - developer community / more open development. +------------------------------- - More details can be found in `ticket 217 - `_ that led to the fork. +Borg was created in May 2015 in response to the difficulty of getting new +code or larger changes incorporated into Attic and establishing a bigger +developer community / more open development. - Borg intends to be: +More details can be found in `ticket 217 +`_ that led to the fork. - * simple: +Borg intends to be: - * as simple as possible, but no simpler - * do the right thing by default, but offer options - * open: +* simple: - * welcome feature requests - * accept pull requests of good quality and coding style - * give feedback on PRs that can't be accepted "as is" - * discuss openly, don't work in the dark - * changing: + * as simple as possible, but no simpler + * do the right thing by default, but offer options +* open: - * Borg is not compatible with Attic - * do not break compatibility accidentally, without a good reason - or without warning. allow compatibility breaking for other cases. - * if major version number changes, it may have incompatible changes + * welcome feature requests + * accept pull requests of good quality and coding style + * give feedback on PRs that can't be accepted "as is" + * discuss openly, don't work in the dark +* changing: + + * Borg is not compatible with Attic + * do not break compatibility accidentally, without a good reason + or without warning. allow compatibility breaking for other cases. + * if major version number changes, it may have incompatible changes From 3d906ab7318ae8996a00ad7841718ecf023f62d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 11:17:10 -0400 Subject: [PATCH 4/7] word-wrap a bullet list as paragraphs instead it looks nicer that way --- docs/faq.rst | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index 4009def13..91306a0bc 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -61,14 +61,16 @@ Which file types, attributes, etc. are *not* preserved? Why is my backup bigger than with attic? Why doesn't |project_name| do compression by default? ---------------------------------------------------------------------------------------------- - * attic was rather unflexible when it comes to compression, it always - compressed using zlib level 6 (no way to switch compression off or - adjust the level or algorithm) - * |project_name| offers a lot of different compression algorithms and - levels. Which of them is the best for you pretty much depends on your use - case, your data, your hardware - so you need to do an informed decision - about whether you want to use compression, which algorithm and which - level you want to use. This is why compression defaults to none. +Attic was rather unflexible when it comes to compression, it always +compressed using zlib level 6 (no way to switch compression off or +adjust the level or algorithm). + +|project_name| offers a lot of different compression algorithms and +levels. Which of them is the best for you pretty much depends on your +use case, your data, your hardware - so you need to do an informed +decision about whether you want to use compression, which algorithm +and which level you want to use. This is why compression defaults to +none. How can I specify the encryption passphrase programmatically? ------------------------------------------------------------- From 5cd5fa72f64fdff08c0ab2350c56f1b83cf89905 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 11:29:22 -0400 Subject: [PATCH 5/7] warn users about the environment on multi-user systems --- docs/faq.rst | 5 +++++ docs/quickstart.rst | 7 ++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/faq.rst b/docs/faq.rst index 91306a0bc..c1745a130 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -81,6 +81,11 @@ automated encrypted backups. Another option is to use key file based encryption with a blank passphrase. See :ref:`encrypted_repos` for more details. +.. caution:: When passing the passphrase through the environment, the + passphrase can be read by any user on the same system, so + the use of this technique is strongly discouraged on + multi-user systems. + When backing up to remote encrypted repos, is encryption done locally? ---------------------------------------------------------------------- diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 9ad86d5e2..ff3aa80ea 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -150,7 +150,12 @@ by providing the correct passphrase. For automated backups the passphrase can be specified using the `BORG_PASSPHRASE` environment variable. -**The repository data is totally inaccessible without the key:** +.. caution:: When passing the passphrase through the environment, the + passphrase can be read by any user on the same system, so + the use of this technique is strongly discouraged on + multi-user systems. + +.. important:: The repository data is totally inaccessible without the key:** Make a backup copy of the key file (``keyfile`` mode) or repo config file (``repokey`` mode) and keep it at a safe place, so you still have the key in case it gets corrupted or lost. From c7c02ef725b6a5f19c22afdadad47559076e2810 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 12:15:39 -0400 Subject: [PATCH 6/7] fix path to favicon --- docs/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 8836c6902..db9cd2de1 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -122,7 +122,7 @@ html_logo = '_static/favicon.ico' # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. -html_favicon = 'favicon.ico' +html_favicon = '_static/favicon.ico' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, From de9e9d14b792649cffe70c94309175a6ad6d4935 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Mon, 19 Oct 2015 16:25:24 -0400 Subject: [PATCH 7/7] soften environment security warning to a note, and cross-ref to avoid dupe --- docs/faq.rst | 13 +++++++++---- docs/quickstart.rst | 7 +++---- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/faq.rst b/docs/faq.rst index c1745a130..d98d26258 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -81,10 +81,15 @@ automated encrypted backups. Another option is to use key file based encryption with a blank passphrase. See :ref:`encrypted_repos` for more details. -.. caution:: When passing the passphrase through the environment, the - passphrase can be read by any user on the same system, so - the use of this technique is strongly discouraged on - multi-user systems. +.. _password_env: +.. note:: Be careful how you set the environment; using the ``env`` + command, a ``system()`` call or using inline shell scripts + might expose the credentials in the process list directly + and they will be readable to all users on a system. Using + ``export`` in a shell script file should be safe, however, as + the environment of a process is `accessible only to that + user + `_. When backing up to remote encrypted repos, is encryption done locally? ---------------------------------------------------------------------- diff --git a/docs/quickstart.rst b/docs/quickstart.rst index ff3aa80ea..19ac429b4 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -150,10 +150,9 @@ by providing the correct passphrase. For automated backups the passphrase can be specified using the `BORG_PASSPHRASE` environment variable. -.. caution:: When passing the passphrase through the environment, the - passphrase can be read by any user on the same system, so - the use of this technique is strongly discouraged on - multi-user systems. +.. note:: Be careful about how you set that environment, see + :ref:`this note about password environments ` + for more information. .. important:: The repository data is totally inaccessible without the key:** Make a backup copy of the key file (``keyfile`` mode) or repo config