merge upstream changes

docs/development.rst

@@ -146,12 +146,9 @@ follow their `guide about avoiding ruining git blame`_:
 Continuous Integration
 ----------------------
 
-All pull requests go through `GitHub Actions`_, which runs the tests on Linux
-and Mac OS X as well as the flake8 style checker. Windows builds run on AppVeyor_,
-while additional Unix-like platforms are tested on Golem_.
+All pull requests go through `GitHub Actions`_, which runs the tests on misc.
+Python versions and on misc. platforms as well as some additional checks.
 
-.. _AppVeyor: https://ci.appveyor.com/project/borgbackup/borg/
-.. _Golem: https://golem.enkore.de/view/Borg/
 .. _GitHub Actions: https://github.com/borgbackup/borg/actions
 
 Output and Logging
@@ -352,6 +349,8 @@ Checklist:
     scripts/upload-pypi X.Y.Z test
     scripts/upload-pypi X.Y.Z
 
+  Note: the signature is not uploaded to PyPi any more, but we upload it to
+  github releases.
 - Put binaries into dist/borg-OSNAME and sign them:
 
   ::
@@ -370,9 +369,10 @@ Checklist:
 
 - Create a GitHub release, include:
 
+  * pypi dist package and signature
   * Standalone binaries (see above for how to create them).
 
-    + For OS X, document the OS X Fuse version in the README of the binaries.
-      OS X FUSE uses a kernel extension that needs to be compatible with the
+    + For macOS, document the macFUSE version in the README of the binaries.
+      macFUSE uses a kernel extension that needs to be compatible with the
       code contained in the binary.
   * A link to ``CHANGES.rst``.

docs/installation.rst

@@ -264,7 +264,7 @@ the installed ``openssl`` formula, point pkg-config to the correct path::
 
     PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig" pip install borgbackup[llfuse]
 
-For OS X Catalina and later, be aware that you must authorize full disk access.
+Be aware that for all recent macOS releases you must authorize full disk access.
 It is no longer sufficient to run borg backups as root. If you have not yet
 granted full disk access, and you run Borg backup from cron, you will see
 messages such as::

docs/internals/data-structures.rst

@@ -384,8 +384,7 @@ The *tam* key is part of the :ref:`tertiary authentication mechanism <tam_descri
 the manifest, since an ID check is not possible.
 
 *config* is a general-purpose location for additional metadata. All versions
-of Borg preserve its contents (it may have been a better place for *item_keys*,
-which is not preserved by unaware Borg versions, releases predating 1.0.4).
+of Borg preserve its contents.
 
 Feature flags
 +++++++++++++
@@ -793,7 +792,7 @@ For small hash tables, we start with a growth factor of 2, which comes down to
 
 E.g. backing up a total count of 1 Mi (IEC binary prefix i.e. 2^20) files with a total size of 1TiB.
 
-a) with ``create --chunker-params buzhash,10,23,16,4095`` (custom, like borg < 1.0):
+a) with ``create --chunker-params buzhash,10,23,16,4095`` (custom):
 
   mem_usage  =  2.8GiB
 

docs/usage/create.rst

@@ -36,8 +36,7 @@ Examples
     $ fusermount -u sshfs-mount
 
     # Make a big effort in fine granular deduplication (big chunk management
-    # overhead, needs a lot of RAM and disk space, see formula in internals
-    # docs - same parameters as borg < 1.0):
+    # overhead, needs a lot of RAM and disk space, see formula in internals docs):
     $ borg create --chunker-params buzhash,10,23,16,4095 small /smallstuff
 
     # Backup a raw device (must not be active/in use/mounted at that time)

docs/usage/general/file-metadata.rst.inc

@@ -29,7 +29,7 @@ On some platforms additional features are supported:
 +=========================+==========+===========+============+
 | Linux                   | Yes      | Yes       | Yes [1]_   |
 +-------------------------+----------+-----------+------------+
-| Mac OS X                | Yes      | Yes       | Yes (all)  |
+| macOS                   | Yes      | Yes       | Yes (all)  |
 +-------------------------+----------+-----------+------------+
 | FreeBSD                 | Yes      | Yes       | Yes (all)  |
 +-------------------------+----------+-----------+------------+

docs/usage/general/file-systems.rst.inc

@@ -21,7 +21,7 @@ and readable after one of the failures mentioned above occurred, run
 - At least three directory levels with short names
 - Typically, file sizes up to a few hundred MB.
   Large repositories may require large files (>2 GB).
-- Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
+- Up to 1000 files per directory.
 - rename(2) / MoveFile(Ex) should work as specified, i.e. on the same file system
   it should be a move (not a copy) operation, and in case of a directory
   it should fail if the destination exists and is not an empty directory,

docs/usage/notes.rst

@@ -175,9 +175,7 @@ Separate compaction
 ~~~~~~~~~~~~~~~~~~~
 
 Borg does not auto-compact the segment files in the repository at commit time
-(at the end of each repository-writing command) any more.
-
-This is new since borg 1.2.0 and requires borg >= 1.2.0 on client and server.
+(at the end of each repository-writing command) any more (since borg 1.2.0).
 
 This causes a similar behaviour of the repository as if it was in append-only
 mode (see below) most of the time (until ``borg compact`` is invoked or an
@@ -236,7 +234,7 @@ in ``.ssh/authorized_keys``:
     command="borg serve --append-only ..." ssh-rsa <key used for not-always-trustable backup clients>
     command="borg serve ..." ssh-rsa <key used for backup management>
 
-Running ``borg init`` via a ``borg serve --append-only`` server will *not* create
+Running ``borg rcreate`` via a ``borg serve --append-only`` server will *not* create
 an append-only repository. Running ``borg rcreate --append-only`` creates an append-only
 repository regardless of server settings.
 
@@ -276,7 +274,7 @@ with file 6::
 
 That's all to do in the repository.
 
-If you want to access this rollbacked repository from a client that already has
+If you want to access this rolled back repository from a client that already has
 a cache for this repository, the cache will reflect a newer repository state
 than what you actually have in the repository now, after the rollback.
 

setup.cfg

@@ -159,7 +159,7 @@ per_file_ignores =
     src/borg/testsuite/__init__.py:E501,F401
     src/borg/testsuite/archive.py:E128,W504
     src/borg/testsuite/archiver/__init__.py:E128,E501,E722,F401,F405,F811
-    src/borg/testsuite/archiver/debug_cmds.py:E501
+    src/borg/testsuite/archiver/debug_cmds.py:E501,F405
     src/borg/testsuite/archiver/disk_full.py:F401,F405,F811
     src/borg/testsuite/archiver/extract_cmd.py:F405
     src/borg/testsuite/archiver/mount_cmds.py:E501,E722

setup_docs.py

@@ -6,7 +6,7 @@ import re
 import sys
 import textwrap
 from collections import OrderedDict
-from datetime import datetime
+from datetime import datetime, timezone
 import time
 
 from setuptools import Command
@@ -470,10 +470,8 @@ class build_man(Command):
         self.write_heading(write, description, double_sided=True)
         # man page metadata
         write(":Author: The Borg Collective")
-        write(
-            ":Date:",
-            datetime.utcfromtimestamp(int(os.environ.get("SOURCE_DATE_EPOCH", time.time()))).date().isoformat(),
-        )
+        source_date_epoch = int(os.environ.get("SOURCE_DATE_EPOCH", time.time()))
+        write(":Date:", datetime.fromtimestamp(source_date_epoch, timezone.utc).date().isoformat())
         write(":Manual section: 1")
         write(":Manual group: borg backup tool")
         write()

src/borg/helpers/fs.py

@@ -1,7 +1,7 @@
 import errno
 import hashlib
 import os
-import os.path
+import posixpath
 import re
 import stat
 import subprocess
@@ -231,7 +231,7 @@ def make_path_safe(path):
     path = path.lstrip("/")
     if path.startswith("../") or "/../" in path or path.endswith("/..") or path == "..":
         raise ValueError(f"unexpected '..' element in path {path!r}")
-    path = os.path.normpath(path)
+    path = posixpath.normpath(path)
     return path
 
 
@@ -245,6 +245,11 @@ def remove_dotdot_prefixes(path):
 
     `path` is expected to be normalized already (e.g. via `os.path.normpath()`).
     """
+    if is_win32:
+        if len(path) > 1 and path[1] == ":":
+            path = path.replace(":", "", 1)
+        path = path.replace("\\", "/")
+
     path = path.lstrip("/")
     path = _dotdot_re.sub("", path)
     if path in ["", ".."]:

src/borg/item.pyx

@@ -562,7 +562,7 @@ cdef class ManifestItem(PropDict):
     archives = PropDictProperty(dict, 'dict of str -> dict')  # name -> dict
     timestamp = PropDictProperty(str)
     config = PropDictProperty(dict)
-    item_keys = PropDictProperty(tuple, 'tuple of str')
+    item_keys = PropDictProperty(tuple, 'tuple of str')  # legacy. new location is inside config.
 
     def update_internal(self, d):
         # legacy support for migration (data from old msgpacks comes in as bytes always, but sometimes we want str)
@@ -650,7 +650,7 @@ class ItemDiff:
         self._can_compare_chunk_ids = can_compare_chunk_ids
         self._chunk_1 = chunk_1
         self._chunk_2 = chunk_2
-        
+
         self._changes = {}
 
         if self._item1.is_link() or self._item2.is_link():

src/borg/manifest.py

@@ -264,7 +264,9 @@ class Manifest:
         manifest.timestamp = m.get("timestamp")
         manifest.config = m.config
         # valid item keys are whatever is known in the repo or every key we know
-        manifest.item_keys = ITEM_KEYS | frozenset(m.get("item_keys", []))
+        manifest.item_keys = ITEM_KEYS
+        manifest.item_keys |= frozenset(m.config.get("item_keys", []))  # new location of item_keys since borg2
+        manifest.item_keys |= frozenset(m.get("item_keys", []))  # legacy: borg 1.x: item_keys not in config yet
 
         if manifest.tam_verified:
             manifest_required = manifest.config.get("tam_required", False)
@@ -321,12 +323,12 @@ class Manifest:
         assert len(self.archives) <= MAX_ARCHIVES
         assert all(len(name) <= 255 for name in self.archives)
         assert len(self.item_keys) <= 100
+        self.config["item_keys"] = tuple(sorted(self.item_keys))
         manifest = ManifestItem(
-            version=1,
+            version=2,
             archives=StableDict(self.archives.get_raw_dict()),
             timestamp=self.timestamp,
             config=StableDict(self.config),
-            item_keys=tuple(sorted(self.item_keys)),
         )
         self.tam_verified = True
         data = self.key.pack_and_authenticate_metadata(manifest.as_dict())

src/borg/testsuite/archiver/create_cmd.py

@@ -117,7 +117,6 @@ def test_basic_functionality(archivers, request):
     assert filter(info_output) == filter(info_output2)
 
 
-@pytest.mark.skipif(is_win32, reason="still broken on windows")
 def test_archived_paths(archivers, request):
     archiver = request.getfixturevalue(archivers)
     repo_location = archiver.repository_location

src/borg/testsuite/archiver/debug_cmds.py

@@ -154,9 +154,10 @@ def test_debug_dump_manifest(archivers, request):
         result = json.load(f)
     assert "archives" in result
     assert "config" in result
-    assert "item_keys" in result
     assert "timestamp" in result
     assert "version" in result
+    assert "item_keys" in result["config"]
+    assert frozenset(result["config"]["item_keys"]) == ITEM_KEYS
 
 
 def test_debug_dump_archive(archivers, request):

src/borg/testsuite/helpers.py

@@ -1202,6 +1202,11 @@ def test_swidth_slice_mixed_characters():
     assert swidth_slice(string, 6) == "나윤a"
 
 
+def utcfromtimestamp(timestamp):
+    """Returns a naive datetime instance representing the timestamp in the UTC timezone"""
+    return datetime.fromtimestamp(timestamp, timezone.utc).replace(tzinfo=None)
+
+
 def test_safe_timestamps():
     if SUPPORT_32BIT_PLATFORMS:
         # ns fit into int64
@@ -1213,9 +1218,9 @@ def test_safe_timestamps():
         # datetime won't fall over its y10k problem
         beyond_y10k = 2**100
         with pytest.raises(OverflowError):
-            datetime.utcfromtimestamp(beyond_y10k)
-        assert datetime.utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2038, 1, 1)
-        assert datetime.utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2038, 1, 1)
+            utcfromtimestamp(beyond_y10k)
+        assert utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2038, 1, 1)
+        assert utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2038, 1, 1)
     else:
         # ns fit into int64
         assert safe_ns(2**64) <= 2**63 - 1
@@ -1226,9 +1231,9 @@ def test_safe_timestamps():
         # datetime won't fall over its y10k problem
         beyond_y10k = 2**100
         with pytest.raises(OverflowError):
-            datetime.utcfromtimestamp(beyond_y10k)
-        assert datetime.utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2262, 1, 1)
-        assert datetime.utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2262, 1, 1)
+            utcfromtimestamp(beyond_y10k)
+        assert utcfromtimestamp(safe_s(beyond_y10k)) > datetime(2262, 1, 1)
+        assert utcfromtimestamp(safe_ns(beyond_y10k) / 1000000000) > datetime(2262, 1, 1)
 
 
 class TestPopenWithErrorHandling:

src/borg/testsuite/platform.py

@@ -161,7 +161,7 @@ class PlatformLinuxTestCase(BaseTestCase):
         self.assert_equal(acl_use_local_uid_gid(b"group:root:rw-:0"), b"group:0:rw-")
 
 
-@unittest.skipUnless(sys.platform.startswith("darwin"), "OS X only test")
+@unittest.skipUnless(sys.platform.startswith("darwin"), "macOS only test")
 @unittest.skipIf(fakeroot_detected(), "not compatible with fakeroot")
 class PlatformDarwinTestCase(BaseTestCase):
     def setUp(self):

src/borg/xattr.py

@@ -1,4 +1,4 @@
-"""A basic extended attributes (xattr) implementation for Linux, FreeBSD and MacOS X."""
+"""A basic extended attributes (xattr) implementation for Linux, FreeBSD and macOS."""
 
 import errno
 import os