update build process to use newer Cython/setuptools APIs
Cython.Distutils.build_ext has been deprecated, and the recommended replacement
is Cython.Build.cythonize.
setuptools.Extension will convert the file extension .pyx -> .c so there's no
need to duplicate the functionality. This allows the removal of the special
Sdist logic when cython is available.
cythonize will compile .pyx -> .c, but the resulting Extension's files won't be
included in sdist unless the Extension is in ext_modules. We don't want the
platform extensions in ext_modules though because that will break compilation,
so the platform .c files were added to the manifest.
- removed some unnecessary distutils imports and moved some imports for better
visibility
- stop cythonize from running in some common use cases like 'clean'
maybe this is the easiest way for us to deal with msgpack compatibility.
0.5.0 release had some troubles:
- FutureWarning on stderr disturbing other output there, breaking tests
- pip install -U broken due to a pip issue with the transisition pkg
which was needed due to the package rename (ImportError for msgpack)
- some linux dists not packaging the transition pkg
setup_zstd.py modified so it is just amending the Extension() kwargs,
but the Extension is initialized by the caller.
this way, amending can happend multiple times (e.g. for multiple
compression algorithms).
also:
- move include/library dirs processing for system-library case
- move system zstd prefix detection to setup_zstd module
- cosmetic: setup.py whitespace fixes
- prefer system zstd option, document zstd min. requirement
based on willyvmm's work in PR #3116, but some changes:
- removed any mulithreading changes
- add zstandard in setup.py install_requires
- tests
- fix: minimum compression level is 1 (not 0)
- use 3 for the default compression level
- use ID 03 00 for zstd
- only convert to bytes if we don't have bytes yet
- move zstd code so that code blocks are ordered by ID
- other cosmetic fixes
1.1.2 was released with a sdist that included quite some files that
did not belong into the package (I tried nuitka at some time).
This was because the old MANIFEST.in / setup.py included all the stuff
in the package dir.
Now, setuptools_scm is to automatically deal with the INCLUDES and
MANIFEST.in only handles the EXCLUDES, so only committed files get
into the sdist (minus some we do not want).
Also, no .c .h .pyx files will be installed - they are not needed as
they have been compiled into binary files.
lgtm:
Nested loops in which the target variable is the same for each loop make
the behavior of the loops difficult to understand.
(not really here, just wanted to get rid of lgtm warning)
This is a (relatively) simple state machine running in the
data callbacks invoked by the msgpack unpacking stack machine
(the same machine is used in msgpack-c and msgpack-python,
changes are minor and cosmetic, e.g. removal of msgpack_unpack_object,
removal of the C++ template thus porting to C and so on).
Compared to the previous solution this has multiple advantages
- msgpack-c dependency is removed
- this approach is faster and requires fewer and smaller
memory allocations
Testability of the two solutions does not differ in my
professional opinion(tm).
Two other changes were rolled up; _hashindex.c can be compiled
without Python.h again (handy for fuzzing and testing);
a "small" bug in the cache sync was fixed which allocated too
large archive indices, leading to excessive archive.chunks.d
disk usage (that actually gave me an idea).
Thomas Waldmann