Merge pull request #3462 from ThomasWaldmann/update-changes-1.1

update CHANGES (1.1-maint)

docs/changes.rst

@@ -6,6 +6,46 @@ Important notes
 
 This section provides information about security and corruption issues.
 
+.. _broken_validator:
+
+Pre-1.1.4 potential data corruption issue
+-----------------------------------------
+
+A data corruption bug was discovered in borg check --repair, see issue #3444.
+
+This is a 1.1.x regression, releases < 1.1 (e.g. 1.0.x) are not affected.
+
+To avoid data loss, you must not run borg check --repair using an unfixed version
+of borg 1.1.x. The first official release that has the fix is 1.1.4.
+
+Package maintainers may have applied the fix to updated packages of 1.1.x (x<4)
+though, see the package maintainer's package changelog to make sure.
+
+If you never had missing item metadata chunks, the bug has not affected you
+even if you did run borg check --repair with an unfixed version.
+
+When borg check --repair tried to repair corrupt archives that miss item metadata
+chunks, the resync to valid metadata in still present item metadata chunks
+malfunctioned. This was due to a broken validator that considered all (even valid)
+item metadata as invalid. As they were considered invalid, borg discarded them.
+Practically, that means the affected files, directories or other fs objects were
+discarded from the archive.
+
+Due to the malfunction, the process was extremely slow, but if you let it
+complete, borg would have created a "repaired" archive that has lost a lot of items.
+If you interrupted borg check --repair because it was so strangely slow (killing
+borg somehow, e.g. Ctrl-C) the transaction was rolled back and no corruption occurred.
+
+The log message indicating the precondition for the bug triggering looks like:
+
+    item metadata chunk missing [chunk: 001056_bdee87d...a3e50d]
+
+If you never had that in your borg check --repair runs, you're not affected.
+
+But if you're unsure or you actually have seen that, better check your archives.
+By just using "borg list repo::archive" you can see if all expected filesystem
+items are listed.
+
 .. _tam_vuln:
 
 Pre-1.0.9 manifest spoofing vulnerability (CVE-2016-10099)
@@ -131,8 +171,8 @@ The best check that everything is ok is to run a dry-run extraction::
 Changelog
 =========
 
-Version 1.1.3 (2017-11-27)
+Version 1.1.4 (not released yet)
---------------------------
+--------------------------------
 
 Compatibility notes:
 
@@ -150,6 +190,51 @@ Compatibility notes:
     You can avoid the one-time slowdown by using the pre-1.1.0rc4-compatible
     mode (but that is less safe for detecting changed files than the default).
     See the --files-cache docs for details.
+- borg 1.1.4 changes:
+
+  - zstd compression is new in borg 1.1.4, older borg can't handle it.
+  - new minimum requirements for the compression libraries - if the required
+    versions (header and lib) can't be found at build time, bundled code will
+    be used:
+
+    - added: libzstd >= 1.3.0 (bundled: 1.3.2)
+    - updated: liblz4 >= 1.7.0 / r129 (bundled: 1.8.0)
+
+Fixes:
+
+- data corruption fix: fix for borg check --repair malfunction, #3444.
+  See the more detailled notes close to the top of this document.
+- also delete security dir when deleting a repo, #3427
+- fix building the "borg prune" man page, #3398
+
+New features:
+
+- added zstd compression. try it!
+- added placeholder for fqdn in reverse notation
+
+Other changes:
+
+- list help topics when invalid topic is requested
+- add auto-generated docs for borg config
+- don't generate HTML docs page for borgfs, #3404
+- refactored build of the compress and crypto.low_level extensions, #3415:
+
+  - move some lib/build related code to setup_{zstd,lz4,b2}.py
+  - bundle lz4 1.8.0 (requirement: >= 1.7.0 / r129)
+  - bundle zstd 1.3.2 (requirement: >= 1.3.0)
+  - blake2 was already bundled
+  - rename BORG_LZ4_PREFIX env var to BORG_LIBLZ4_PREFIX for better consistency:
+    we also have BORG_LIBB2_PREFIX and BORG_LIBZSTD_PREFIX now.
+  - add prefer_system_lib* = True settings to setup.py - by default the build
+    will prefer a shared library over the bundled code, if library and headers
+    can be found and meet the minimum requirements.
+- fix lz4 deprecation warning, requires lz4 >= 1.7.0 (r129)
+- add parens for C preprocessor macro argument usages (did not cause malfunction)
+- exclude broken pytest 3.3.0 release
+
+
+Version 1.1.3 (2017-11-27)
+--------------------------
 
 Fixes: