maybe this is the easiest way for us to deal with msgpack compatibility.
0.5.0 release had some troubles:
- FutureWarning on stderr disturbing other output there, breaking tests
- pip install -U broken due to a pip issue with the transisition pkg
which was needed due to the package rename (ImportError for msgpack)
- some linux dists not packaging the transition pkg
setup_zstd.py modified so it is just amending the Extension() kwargs,
but the Extension is initialized by the caller.
this way, amending can happend multiple times (e.g. for multiple
compression algorithms).
also:
- move include/library dirs processing for system-library case
- move system zstd prefix detection to setup_zstd module
- cosmetic: setup.py whitespace fixes
- prefer system zstd option, document zstd min. requirement
based on willyvmm's work in PR #3116, but some changes:
- removed any mulithreading changes
- add zstandard in setup.py install_requires
- tests
- fix: minimum compression level is 1 (not 0)
- use 3 for the default compression level
- use ID 03 00 for zstd
- only convert to bytes if we don't have bytes yet
- move zstd code so that code blocks are ordered by ID
- other cosmetic fixes
1.1.2 was released with a sdist that included quite some files that
did not belong into the package (I tried nuitka at some time).
This was because the old MANIFEST.in / setup.py included all the stuff
in the package dir.
Now, setuptools_scm is to automatically deal with the INCLUDES and
MANIFEST.in only handles the EXCLUDES, so only committed files get
into the sdist (minus some we do not want).
Also, no .c .h .pyx files will be installed - they are not needed as
they have been compiled into binary files.
lgtm:
Nested loops in which the target variable is the same for each loop make
the behavior of the loops difficult to understand.
(not really here, just wanted to get rid of lgtm warning)
This is a (relatively) simple state machine running in the
data callbacks invoked by the msgpack unpacking stack machine
(the same machine is used in msgpack-c and msgpack-python,
changes are minor and cosmetic, e.g. removal of msgpack_unpack_object,
removal of the C++ template thus porting to C and so on).
Compared to the previous solution this has multiple advantages
- msgpack-c dependency is removed
- this approach is faster and requires fewer and smaller
memory allocations
Testability of the two solutions does not differ in my
professional opinion(tm).
Two other changes were rolled up; _hashindex.c can be compiled
without Python.h again (handy for fuzzing and testing);
a "small" bug in the cache sync was fixed which allocated too
large archive indices, leading to excessive archive.chunks.d
disk usage (that actually gave me an idea).
hacky, but works. Better would be to make a separate docs/examples dir
with only the examples in them, separated by command.
Or, putting these different sections; DESCRIPTION, EXAMPLES and NOTES
into the --help doc, but separately of course, so that they can be aptly
formatted for different media (html, --help, man).
CRC slice by 8 for generic CPUs outperforms zlib CRC32 on ppc
and x86 (ARM untested but expected to as well).
PCLMULQDQ derived from Intel's zlib patches outperforms every other
CRC implementation by a huge margin.
This makes an surprisingly large difference. Test case: ~70000 empty files.
(Ie. little data shoveling, lots of metadata shoveling). Before: 9.1 seconds
+- 0.1 seconds. After: 8.4 seconds +- 0.1 seconds.). That's a huge
win for changing a few lines.
I'd expect that this improves performance in almost all areas that touch
the items (list, delete, prune).
I have discovered that PyPI is way more sensitive to RST warnings than
other platforms: warnings and errors will make the document not show
up correctly, which is currently the case here:
https://pypi.python.org/pypi/borgbackup/
the suggested changes remove Sphinx-specific markup from the output,
namely the badges and `|replacement|` patterns, but also the
`higlight` directive. this also requires adding tags to the README to
mark the badges to remove and removal of the `none` argument for the
`.. code-block::` element which was not having any significant in
Sphinx anyways.
literal blocks must be followed by an empty line
suppressed the non-local image uri warning via sphinx config
the links on the resources page must have different label texts
setup.py: do not generate pointless "::\n" - it does not create a literal block if the stuff below is not indented
Add /opt/pkg (used by pkgsrc on Mac OS X) to possible openssl and
lz4 header path.
from 2015Q2 onwards the prefix has changed to /opt/pkg
in order to be compatible with El Capitan's
"System Integrity Protection" feature.
-- pkgsrc.joynet.com
thanks to @weakish for finding this!
- Group options
- Nicer list of options in Sphinx
- Deduplicate 'Common options'
(including --help)
The latter is done by explicitly declaring --help in the common_parser,
which is then inherited by the sub-parsers; no change in observable
behaviour.
- add archiver.main_mount()
- provide borgfs behaviour when the monolithic binary is called via a
symlink called borgfs
- docs: update usage of mount subcommand, provide examples for borgfs and
add symlink creation to standalone binary installation
- run build_usage
- add entry point in setup.py
- patch helpers.py:get_keys_dir() to allow mounting fstab entries with
"user" option set
Without this, setuid() called at some point by mount changes the HOME
environment variable to '/root' and os.expanduser('~') would return
'/root' as well, thus the mount would fail with
PermissionError: [Errno 13] Permission denied: '/root/.config'
After setuid(), the HOME variable stays intact, so we still can
explicitly query USER's home.
Also, os.path.expanduser() behaves differently for '~' and '~someuser'
as parameters: when called with an explicit username, the possibly set
environment variable HOME is no longer respected. So we have to check if
it is set and only expand the user's home directory if HOME is unset.
- add myself to AUTHORS
this was making us require mock, which is really a test component and
shouldn't be part of the runtime dependencies. furthermore, it was
making the imports and the code more brittle: it may have been
possible that, through an environment variable, backups could be
corrupted because mock libraries would be configured instead of real
once, which is a risk we shouldn't be taking.
finally, this was used only to build docs, which we will build and
commit to git by hand with a fully working borg when relevant.
see #384.
This reverts commit 86487d192a.
We will instead commit the generated `.rst` usage and API files
directly into git. The setup commands remain to generate them when the
usage or API changes, but as things are the hoops required to generate
those RST files are way too complicated to justify simply build docs.
See #384.
Conflicts:
setup.py
it requires "mock" (and later also sphinx) and we do not (want to) have that in setup_requires.
I removed the build.sub_commands.append(...) because that made setup.py install run build_api
and build_usage automatically for production installs. the comment said it does not work on
readthedocs anyway.
build_api and build_usage can still be run manually.
it seems that what worked in the debug branch is not working in the
main branch, even though the commit IDs are exactly the same. the RTD environment doesn't seem really reliable...
besides, we want to build extensions before the rest, so should run it
first, in order to have msgpack loaded.
instead of applying this only to usage generation, use it as a generic
mechanism to disable loading of Cython code.
it may be incomplete: there may be other places where Cython code is
loaded that is not checked, but that is sufficient to build the usage
docs. the environment variable used is documented as such in the
docs/usage.rst.
we also move the check to a helper function and document it
better. this has the unfortunate side effect of moving includes
around, but I can't think of a better way.
this is such a crude hack it is totally embarrassing....
the proper solution would probably be to move the `build_parser()`
function out of `Archiver` completely, but this is such an undertaking
that i doubt it is worth doing since we're looking at switching to
click anyways.
the main problem in moving build_parser() out is that it references
`self` all the time, so it *needs* an archiver context that it can
reuse. we could make the function static and pass self in there by
hand, but it seems like almost a worse hack... and besides, we would
need to load the archiver in order to do that, which would break usage
all over again...
this is an unfortunate rewrite of the manpage creation code mentionned
in #208. ideally, this would be rewritten into a class that can
generate both man pages and .rst files.
first off, this required ticking the `Install your project inside a
virtualenv using setup.py install` box in the advanced config.
then, i had to disable all the C extensions build and disable some
checks, based on whether we are running on RTD or not.
still missing: usage builds and possibly other stuff that is in our
Makefile and not in setup.py.
Thomas Waldmann