our data structures need strict_map_key=False, which is not the
default of msgpack 1.0.0. i made it default in our wrapper API.
call our wrapper for performance profile creation/conversion also
to avoid msgpack compat issues.
remove encoding from wrapper api, we do not use it any more.
remove raw is True check, we need false for profiles
strict_map_key is only supported for msgpack >= 1.0.0.
- Created a batch file to build borg on windows
- Adjusted setup.py to be runnable on windows and build the windows
extension
- Extracted the free space check to a function in the platform module
- Created the minimal needed (dummy) functions for the windows platform
module
The build process can be controlled via environment variables
similar to other bundled libraries in borgbackup. The main difference
is probably that upstream does not provide a pkgconfig file for
xxhash.
Therefore borg will probably fail to detect the system-provided
version by default (tested on Fedora, seems like Debian and Ubuntu
do not ship a pkgconfig file either). I kept the pkgconfig lookup
code anyway to keep the code as similar as possible to
"setup_compress.py"/"setup_crypto.py".
Setting BORG_LIBXXHASH_PREFIX=/usr helps borg to detect xxhash
on my system (Fedora). You can force the use of the bundled
version of xxhash by setting BORG_USE_BUNDLED_XXHASH=1.
these are already used internally when the build system can not find
a system library (neither via pkginfo nor BORG_LIBXXX_PREFIX is given)
and then triggers usage of the bundled code via these env vars.
now they are also used to tell right from the beginning "use the
bundled code" and in that case it will not try to locate system libs
and headers.
1. BORG_*_PREFIX is checked (avoids lib detection via pkg-config).
2. pkg-config is tried
3. fallback to bundled C code (or failure in case of OpenSSL)
also:
- simplified code again
- removed (c) headers, nothing left of original code
- removed hardcoded lib search pathes
- to find system libs/headers, one must point these env vars to them:
BORG_OPENSSL_PREFIX
- moved some code from setup.py to setup_*.py
- removed hardcoded lib search pathes
- to find system libs/headers, one must point these env vars to them:
BORG_LIBLZ4_PREFIX, BORG_LIBZSTD_PREFIX, BORG_LIBB2_PREFIX
- moved some code from setup.py to setup_*.py
if you do not have python 3.6.x, you can still use borg 1.1.x or 1.0.x.
another option is to use the fat binary from github releases, which
includes python 3.6 and all other stuff needed.
MASTER BRANCH ONLY, DO NOT BACKPORT TO 1.1 or 1.0!
As there are no 0.4.x msgpack releases (under this new name, 0.4.x
used to be named "msgpack-python"), currently only 0.5.6 can be
recommended. See comments for other versions that are acceptable
under certain conditions.
We will add new msgpack versions later, after testing them.
Failure to use environment markers means the dependencies are
unconditionally added at build time based on the host instead of being
always present and evaluated at runtime on the target; e.g. wheels have
the wrong information.
Also use python_requires. This teaches pip how to natively comprehend
when the current version of python is not supported by borg.
Returns errors in the format:
borgbackup requires Python '>=3.5' but the running Python is $oldver
we temporarily used the updated, bundled lz4 and zstd code for testing
purposes, but now going back to using system provided libs by default.
(cherry picked from commit 66084c7234)
update build process to use newer Cython/setuptools APIs
Cython.Distutils.build_ext has been deprecated, and the recommended replacement
is Cython.Build.cythonize.
setuptools.Extension will convert the file extension .pyx -> .c so there's no
need to duplicate the functionality. This allows the removal of the special
Sdist logic when cython is available.
cythonize will compile .pyx -> .c, but the resulting Extension's files won't be
included in sdist unless the Extension is in ext_modules. We don't want the
platform extensions in ext_modules though because that will break compilation,
so the platform .c files were added to the manifest.
- removed some unnecessary distutils imports and moved some imports for better
visibility
- stop cythonize from running in some common use cases like 'clean'
maybe this is the easiest way for us to deal with msgpack compatibility.
0.5.0 release had some troubles:
- FutureWarning on stderr disturbing other output there, breaking tests
- pip install -U broken due to a pip issue with the transisition pkg
which was needed due to the package rename (ImportError for msgpack)
- some linux dists not packaging the transition pkg
setup_zstd.py modified so it is just amending the Extension() kwargs,
but the Extension is initialized by the caller.
this way, amending can happend multiple times (e.g. for multiple
compression algorithms).
also:
- move include/library dirs processing for system-library case
- move system zstd prefix detection to setup_zstd module
- cosmetic: setup.py whitespace fixes
- prefer system zstd option, document zstd min. requirement
based on willyvmm's work in PR #3116, but some changes:
- removed any mulithreading changes
- add zstandard in setup.py install_requires
- tests
- fix: minimum compression level is 1 (not 0)
- use 3 for the default compression level
- use ID 03 00 for zstd
- only convert to bytes if we don't have bytes yet
- move zstd code so that code blocks are ordered by ID
- other cosmetic fixes
1.1.2 was released with a sdist that included quite some files that
did not belong into the package (I tried nuitka at some time).
This was because the old MANIFEST.in / setup.py included all the stuff
in the package dir.
Now, setuptools_scm is to automatically deal with the INCLUDES and
MANIFEST.in only handles the EXCLUDES, so only committed files get
into the sdist (minus some we do not want).
Also, no .c .h .pyx files will be installed - they are not needed as
they have been compiled into binary files.
lgtm:
Nested loops in which the target variable is the same for each loop make
the behavior of the loops difficult to understand.
(not really here, just wanted to get rid of lgtm warning)
This is a (relatively) simple state machine running in the
data callbacks invoked by the msgpack unpacking stack machine
(the same machine is used in msgpack-c and msgpack-python,
changes are minor and cosmetic, e.g. removal of msgpack_unpack_object,
removal of the C++ template thus porting to C and so on).
Compared to the previous solution this has multiple advantages
- msgpack-c dependency is removed
- this approach is faster and requires fewer and smaller
memory allocations
Testability of the two solutions does not differ in my
professional opinion(tm).
Two other changes were rolled up; _hashindex.c can be compiled
without Python.h again (handy for fuzzing and testing);
a "small" bug in the cache sync was fixed which allocated too
large archive indices, leading to excessive archive.chunks.d
disk usage (that actually gave me an idea).
Thomas Waldmann