Merge pull request #5017 from MichaelEischer/rewrite-data-loss

rewrite: Document handling of "cannot encode tree" errors

doc/045_working_with_repos.rst

@@ -305,6 +305,13 @@ In order to preview the changes which ``rewrite`` would make, you can use the
 modifying the repository. Instead restic will only print the actions it would
 perform.
 
+.. note:: The ``rewrite`` command verifies that it does not modify snapshots in
+    unexpected ways and fails with an ``cannot encode tree at "[...]" without loosing information``
+    error otherwise. This can occur when rewriting a snapshot created by a newer
+    version of restic or some third-party implementation.
+
+    To convert a snapshot into the format expected by the ``rewrite`` command
+    use ``restic repair snapshots <snapshotID>``.
 
 Modifying metadata of snapshots
 ===============================

doc/design.rst

@@ -126,8 +126,8 @@ the option ``-o local.layout=default``, valid values are ``default`` and
 ``s3legacy``. The option for the sftp backend is named ``sftp.layout``, for the
 s3 backend ``s3.layout``.
 
-S3 Legacy Layout
-----------------
+S3 Legacy Layout (deprecated)
+-----------------------------
 
 Unfortunately during development the Amazon S3 backend uses slightly different
 paths (directory names use singular instead of plural for ``key``,
@@ -152,8 +152,7 @@ the ``data`` directory. The S3 Legacy repository layout looks like this:
     /snapshot
      └── 22a5af1bdc6e616f8a29579458c49627e01b32210d09adb288d1ecda7c5711ec
 
-The S3 backend understands and accepts both forms, new backends are
-always created with the default layout for compatibility reasons.
+Restic 0.17 is the last version that supports the legacy layout.
 
 Pack Format
 ===========
@@ -234,7 +233,9 @@ Individual files for the index, locks or snapshots are encrypted
 and authenticated like Data and Tree Blobs, so the outer structure is
 ``IV || Ciphertext || MAC`` again. In repository format version 1 the
 plaintext always consists of a JSON document which must either be an
-object or an array.
+object or an array. The JSON encoder must deterministically encode the
+document and should match the behavior of the Go standard library implementation
+in ``encoding/json``.
 
 Repository format version 2 adds support for compression. The plaintext
 now starts with a header to indicate the encoding version to distinguish
@@ -473,6 +474,10 @@ A snapshot references a tree by the SHA-256 hash of the JSON string
 representation of its contents. Trees and data are saved in pack files
 in a subdirectory of the directory ``data``.
 
+The JSON encoder must deterministically encode the document and should
+match the behavior of the Go standard library implementation in ``encoding/json``.
+This ensures that trees can be properly deduplicated.
+
 The command ``restic cat blob`` can be used to inspect the tree
 referenced above (piping the output of the command to ``jq .`` so that
 the JSON is indented):
@@ -507,12 +512,11 @@ this metadata is generated:
 - The name is quoted using `strconv.Quote <https://pkg.go.dev/strconv#Quote>`__
   before being saved. This handles non-unicode names, but also changes the
   representation of names containing ``"`` or ``\``.
-
 - The filemode saved is the mode defined by `fs.FileMode <https://pkg.go.dev/io/fs#FileMode>`__
   masked by ``os.ModePerm | os.ModeType | os.ModeSetuid | os.ModeSetgid | os.ModeSticky``
-
-When the entry references a directory, the field ``subtree`` contains the plain text
-ID of another tree object.
+- When the entry references a directory, the field ``subtree`` contains the plain text
+  ID of another tree object.
+- Check the implementation for a full struct definition.
 
 When the command ``restic cat blob`` is used, the plaintext ID is needed
 to print a tree. The tree referenced above can be dumped as follows: