Commit Graph

1766 Commits

Author SHA1 Message Date
Thomas Waldmann cc3b5c062c remove algorithms package, move checksums module to borg package 2022-03-17 00:24:49 +01:00
Björn Ketelaars 3d86df6649 `borg benchmark cpu` fails with libressl
`borg benchmark cpu` fails on OpenBSD with the error below, which is
caused by LibreSSL currently not supporting AES256_OCB and
CHACHA20_POLY1305.

Work around this by checking if borg is used with LibreSSL. Tested on
OpenBSD.

```
Chunkers =======================================================
buzhash,19,23,21,4095    1GB        14.294s
fixed,1048576            1GB        0.244s
Non-cryptographic checksums / hashes ===========================
crc32 (libdeflate, used) 1GB        0.724s
crc32 (zlib)             1GB        1.953s
xxh64                    1GB        0.361s
Cryptographic hashes / MACs ====================================
hmac-sha256              1GB        7.039s
blake2b-256              1GB        9.845s
Encryption =====================================================
aes-256-ctr-hmac-sha256  1GB        18.312s
aes-256-ctr-blake2b      1GB        21.213s
Local Exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/borg/archiver.py", line 5241, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.9/site-packages/borg/archiver.py", line 5172, in run
    return set_ec(func(args))
  File "/usr/local/lib/python3.9/site-packages/borg/archiver.py", line 607, in do_benchmark_cpu
    print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s")
  File "/usr/local/lib/python3.9/timeit.py", line 233, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/usr/local/lib/python3.9/timeit.py", line 177, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
  File "/usr/local/lib/python3.9/site-packages/borg/archiver.py", line 602, in <lambda>
    ("aes-256-ocb", lambda: AES256_OCB(
  File "src/borg/crypto/low_level.pyx", line 636, in borg.crypto.low_level.AES256_OCB.__init__
  File "src/borg/crypto/low_level.pyx", line 633, in borg.crypto.low_level.AES256_OCB.requirements_check
ValueError: AES OCB is not implemented by LibreSSL (yet?).

Platform: OpenBSD gateway.lan 7.1 GENERIC.MP#418 amd64
Borg: 1.2.1.dev98+gebaf0c32  Python: CPython 3.9.10 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
PID: 38614  CWD: /storage/8899fc1454db04de.a/home/code/git/ports/sysutils/borg
sys.argv: ['/usr/local/bin/borg', 'benchmark', 'cpu']
SSH_ORIGINAL_COMMAND: None
```
2022-03-14 18:56:01 +01:00
Thomas Waldmann 79eee27936 borg benchmark cpu: display benchmarks of cpu bound stuff 2022-03-13 22:49:52 +01:00
Thomas Waldmann 98a675f62a use crc32 from libdeflate, remove bundled crc32 code
we use zlib.crc32 (macOS) or libdeflate_crc32 (Linux and all others) now.
2022-03-13 21:39:34 +01:00
Thomas Waldmann 0b3b78e139 dedup code: assert_type 2022-03-11 23:05:32 +01:00
Thomas Waldmann b3e7e90c29 improve storage type handling, dedup code 2022-03-11 23:05:32 +01:00
Thomas Waldmann a63614e35b move key type/storage constants to borg.constants 2022-03-11 23:05:32 +01:00
Thomas Waldmann 2e536bcbe2 borg key change-location 2022-03-11 23:05:32 +01:00
Thomas Waldmann 766d976f46 move passphrase related stuff to borg.helpers.passphrase 2022-03-11 21:05:42 +01:00
Thomas Waldmann 8011fade91 move key loading/saving code to FlexiKey 2022-03-11 21:05:42 +01:00
Thomas Waldmann ad405892ca rename KeyfileKeyBase -> FlexiKeyBase
it is used as direct base class for both:
- RepoKey
- KeyfileKey
2022-03-11 21:05:42 +01:00
Thomas Waldmann d42e6f2c41 key: uses key_cls.TYPES_ACCEPTABLE to dispatch
for now only a quite simple change, replacing the hardcoded PassphraseKey -> RepoKey dispatching.
2022-03-11 21:05:42 +01:00
TW 153b6c3c70
Merge pull request #6429 from ThomasWaldmann/fix-savefile-mode-master
respect umask for files / dirs (master)
2022-03-09 19:00:58 +01:00
Thomas Waldmann dd0ae1e48d ensure_dir: respect umask for created directory modes, fixes #6400
we tried to be very private / secure here, but that created the issue
that a less secure umask (like e.g. 0o007) just did not work.

to make the umask work, we must start from 0o777 mode and let the
umask do its work, like e.g. 0o777 & ~0o007 --> 0o770.

with borg's default umask of 0o077, it usually ends up being 0o700,
so only permissions for the user (not group, not others).
2022-03-09 17:16:58 +01:00
Thomas Waldmann 0f9d88bf2e SaveFile: respect umask for final file mode, fixes #6400 2022-03-09 17:16:31 +01:00
James Buren 596e33b68d properly suppress -Wimplicit-fallthrough
It turns out that explaining the comment actually breaks its
intended usage.
2022-03-09 05:00:56 -06:00
Thomas Waldmann f0bf5915c5 fix "useless trailing comma" cython warnings 2022-03-08 21:15:24 +01:00
Andrey Bienkowski f75a6152ab ‘PyUnicode_AsUnicode’ is deprecated
Fix compilation warnings:
```
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9 -c src/borg/platform/posix.c -o build/temp.linux-x86_64-3.9/src/borg/platform/posix.o
src/borg/platform/posix.c: In function ‘__pyx_pf_4borg_8platform_5posix_2swidth’:
src/borg/platform/posix.c:1572:3: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
 1572 |   __pyx_t_2 = __Pyx_PyUnicode_AsUnicode(__pyx_v_s); if (unlikely((!__pyx_t_2) && PyErr_Occurred())) __PYX_ERR(0, 19, __pyx_L1_error)
      |   ^~~~~~~~~
In file included from /opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/unicodeobject.h:1026,
                 from /opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/Python.h:93,
                 from src/borg/platform/posix.c:19:
/opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
  580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
      |                                             ^~~~~~~~~~~~~~~~~~~
```

Ref: https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUnicode
2022-03-07 10:36:22 +03:00
Thomas Waldmann 37f237d3e0 remove PassphraseKey code and borg key migrate-to-repokey command
"passphrase" encryption mode repos can not be created since borg 1.0.
back then, users were advised to switch existing repos of that type
to repokey mode using the "borg key migrate-to-repokey" command.
that command is still available in borg 1.0, 1.1 and 1.2, but not
any more in borg >= 1.3.

while we still might see the PassphraseKey.TYPE byte in old repos,
it is handled by the RepoKey code since borg 1.0.
2022-03-06 15:26:18 +01:00
TW e4e243be0b
Merge pull request #6408 from ThomasWaldmann/hmac-digest-master
hmac and blake2b minor optimizations and cleanups
2022-03-06 01:32:30 +01:00
Thomas Waldmann 862eefb96d fixup: fix SELFTEST_COUNT 2022-03-05 22:26:04 +01:00
Thomas Waldmann 83deedb13a kill filter process in case of borg exceptions, fixes #6401
in the finally-block, we wait for the filter process to die. but it only dies
voluntarily if all data was processed by the filter and it terminates due to EOF.

otoh, if borg has thrown an early exception, e.g. "archive already exists",
we need to kill the filter process to bring it to an early end. in that
case, we also do not need to check the filter rc, because we know we killed it.
2022-03-05 16:25:10 -05:00
Thomas Waldmann 2bcee08b88 import-tar: fix mtime type bug
looks like with a .tar file created by the tar tool,
tarinfo.mtime is a float [s]. So, after converting to
nanoseconds, we need to cast to int because that's what
Item.mtime wants.

also added a safe_ns() there to clip values to the safe range.
2022-03-05 16:24:59 -05:00
Thomas Waldmann 41536f9f79 fixup: remove now unused import 2022-03-05 21:55:23 +01:00
Thomas Waldmann 07159342ba remove our blake2b_128/256 tests
as we're just calling python stdlib there,
we can assume that they have tests for it.
2022-03-05 21:47:40 +01:00
Thomas Waldmann 2566671d85 remove our hmac_sha256 tests
as we're just calling python stdlib there,
we can assume that they have tests for it.
2022-03-05 21:37:31 +01:00
Thomas Waldmann c63bd36a6c hmac.digest: some more minor optimizations
also some cosmetic changes:
- import hmac module
- use hmac.compare_digest
2022-03-05 21:29:42 +01:00
Thomas Waldmann 8317698f9b hmac_sha256: replace own cython wrapper code by hmac.digest python stdlib (since py38)
i measured performance of both: pretty much the same.
2022-03-05 21:22:24 +01:00
TW 34261aac0e
Merge pull request #6378 from braewoods/warnings
suppress -Wimplicit-fallthrough warning
2022-03-05 14:20:24 +01:00
TW 47a9e4686b
Merge pull request #6377 from ThomasWaldmann/fix-6374-master
create: skip with warning if opening the parent dir of recursion root fails, fixes #6374
2022-03-05 14:18:57 +01:00
James Buren 4a55d9de3c suppress -Wimplicit-fallthrough warning
These instances of implicit switch case fallthrough appear to be
intentional. Add comments that the compiler understands to suppress
the false positive warning.
2022-02-27 14:29:53 -06:00
Thomas Waldmann fade3c4799 manual formatting fixes (cosmetic / pep8) 2022-02-27 20:12:04 +01:00
Thomas Waldmann cbeef56454 pyupgrade --py38-plus ./**/*.py 2022-02-27 20:11:56 +01:00
Thomas Waldmann 64e7ccd3fc require python >= 3.9, fixes #6315 2022-02-27 18:46:30 +01:00
TW e8b2ca8b20
Merge pull request #6372 from ThomasWaldmann/unbundle-3rd-party
Unbundle most 3rd party code
2022-02-27 18:41:22 +01:00
Thomas Waldmann 8375d44328 create: skip with warning if opening the parent dir of recursion root fails, fixes #6374 2022-02-27 02:52:03 +01:00
TW 52f9bc926d
Merge pull request #6368 from braewoods/warnings
fix multiple warnings related to _hashindex.c
2022-02-26 23:03:38 +01:00
Thomas Waldmann 96d93dcf0e do not support bundled 3rd party code any more, fixes #6316 2022-02-26 22:18:12 +01:00
Björn Ketelaars 73c426497f Fix build with LibreSSL
#6338 introduces regression when building with LibreSSL (3.5.0).

```
cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -g -fPIC -O2 -pipe -g -O2 -pipe -g -O2 -pipe -fPIC -Isrc/borg/crypto -I/usr/local/include/python3.9 -c src/borg/crypto/low_level.c -o /tmp/ports/pobj/borgbackup-1.2.1/borg-eec359cf228caf00d9c72bde07bf939872e9d3fa/temp.openbsd-7.1-amd64-3.9/src/borg/crypto/low_level.o
src/borg/crypto/low_level.c:12439:48: error: use of undeclared identifier 'EVP_chacha20_poly1305'; did you mean 'EVP_aead_chacha20_poly1305'?
  __pyx_v_self->__pyx_base.__pyx_base.cipher = EVP_chacha20_poly1305;
                                               ^~~~~~~~~~~~~~~~~~~~~
                                               EVP_aead_chacha20_poly1305
/usr/include/openssl/evp.h:1161:17: note: 'EVP_aead_chacha20_poly1305' declared here
const EVP_AEAD *EVP_aead_chacha20_poly1305(void);
                ^
1 error generated.
```

Unfortunately `EVP_aead_chacha20_poly1305`, offered by LibreSSL, is not
a drop in replacement for `EVP_chacha20_poly1305`. More info on the
first can be found at https://man.openbsd.org/EVP_AEAD_CTX_init.3.
2022-02-26 22:03:00 +01:00
Thomas Waldmann 159bd06412 unbundle xxh64 3rd party code, see #6316 2022-02-26 21:37:34 +01:00
Thomas Waldmann ecd0865d9c unbundle zstd 3rd party code, see #6316 2022-02-26 21:36:30 +01:00
Thomas Waldmann 7c8f288a18 unbundle lz4 3rd party code, see #6316 2022-02-26 21:35:17 +01:00
James Buren 5c94c932bb src/borg/cache_sync/unpack.h: fix compiler warnings
The key argument being sent to hashindex_get and hashindex_set by
multiple functions is a different signedness from what the functions
expect. This resolves the issue by changing the key type in the
unpack_user struct to unsigned char.
2022-02-26 14:02:29 -06:00
James Buren 5f3d61e2f0 src/borg/_hashindex.c: fix compiler warnings
The value argument of hashindex_set is causing harmless pointer type
mismatches. This resolves the issue by changing the type to void*
which silences these types of warnings.
2022-02-26 14:02:29 -06:00
Thomas Waldmann 0f2a45b474 crypto: get rid of deprecated HMAC_* functions
These were deprecated in OpenSSL 3.0.
2022-02-26 20:20:11 +01:00
TW 8f945ea252
Merge pull request #6338 from ThomasWaldmann/drop-openssl10
drop openssl 1.0.x support
2022-02-26 20:18:56 +01:00
James Buren e27a3eff0a src/borg/compress.pyx: fix compiler warning, closes #6365
This resolves a compiler warning from the generated code that
resulted from a comparison of two local variables of different
signedness. The issue is resolved by changing the type of both
to int since this seems like the safest choice available.
2022-02-25 22:28:34 -06:00
Thomas Waldmann d1d3d1dfa4 crypto: remove support for: OpenSSL < 1.1.1, LibreSSL < 2.7.0
All these are unsupported since long.

Newer versions of LibreSSL have gained chacha20-poly1305 support,
but still lack aes256-ocb support.
Also they have the HMAC_CTX_new/free api now.

docs: openssl >= 1.1.1 is required now
anything older is out of support anyway.
2022-02-26 00:55:19 +01:00
TW cf4a1abb87
Merge pull request #6341 from ThomasWaldmann/require-msgpack10
Require msgpack 1.0.3
2022-02-26 00:51:53 +01:00
James Buren 0d2fae6e7b src/borg/crypto/low_level.pyx: fix compiler warning
The generated source code was producing a compiler warning due to
the pointers differing in constness. The called function expects
a non-const pointer while the generated code produces a const pointer
via a cast. This changes the cast to drop 'const' to make the compiler
happy.
2022-02-25 16:42:24 -06:00