Commit Graph

1480 Commits

Author SHA1 Message Date
Thomas Waldmann e818026440 acls (linux): remove int() cast
it seems like the (non-numerical) name or group might get into fields[3] if the name -> uid or group -> gid mapping is unknown.
2015-10-19 00:28:53 +02:00
Thomas Waldmann 1f14d1de19 acls (linux): use surrogatescape error handling for acl_append_numeric_ids and acl_numeric_ids
surrogatescape will decode/encode invalid utf-8 sequences (if we do not get utf-8) in a round-tripping way.
2015-10-19 00:01:19 +02:00
Thomas Waldmann 63ed5d9949 acls (posix platforms): use surrogatescape error handling for posix_acl_use_stored_uid_gid
surrogatescape will decode/encode invalid utf-8 sequences (if we do not get utf-8) in a round-tripping way.
2015-10-18 23:43:24 +02:00
Thomas Waldmann b2cffe00fe acls (linux): use surrogatescape error handling, fix test
surrogatescape will decode/encode invalid utf-8 sequences (if we do not get utf-8) in a round-tripping way.
2015-10-18 23:33:29 +02:00
Thomas Waldmann 55179fe64d acls (linux): add a test for acl_use_local_uid_gid and fix this function
this has never worked as intended as the function was not using the computed "fields[1]" value at all.
plus there were type issues after that was fixed.
2015-10-18 23:20:01 +02:00
Thomas Waldmann 45b04cd3c1 add non-ascii ACL test (linux)
still failing as the tested code does not yet work with non-ascii ACLs
2015-10-18 20:54:46 +02:00
Thomas Waldmann 723590df3d fix borg.exe presence detection for python3.2 2015-10-18 19:18:24 +02:00
Thomas Waldmann 60cdb5f716 make archiver tests run for the python source as well as for the pyinstaller binary, fixes #215
if the borg.exe binary is not available in PATH, binary tests are skipped.

source tests are run without forking (for better speed, esp. on travis).
binary tests need forking the binary, of course.

for source tests, some tests check for an exception to happen.
for a forked binary, we of course can only check the exit code, which is non-zero in that case.
2015-10-18 18:48:36 +02:00
Thomas Waldmann 03975016c5 upgrader: fix / reformat comments 2015-10-18 16:43:59 +02:00
TW 3c52f41132 Merge pull request #299 from anarcat/no-inplace
do not upgrade repositories in place by default
2015-10-18 16:36:07 +02:00
TW 10567fb327 Merge pull request #306 from anarcat/upgrader-index-fixes
fix cascading failure with the index conversion code
2015-10-18 14:54:49 +02:00
TW 2c4196ed54 Merge pull request #297 from ThomasWaldmann/help-usage-docs
include the help offered by borg help <topic> in the usage docs, fixes #293
2015-10-18 14:51:12 +02:00
TW 1b426c5e24 Merge pull request #300 from ThomasWaldmann/attic-changes
include a list of major changes compared to attic into the README, fixes #224
2015-10-18 14:50:40 +02:00
Antoine Beaupré 319fcbacc5 fix phrasings in "in place" refactoring 2015-10-17 22:04:00 -04:00
Antoine Beaupré 8c48015e70 try to work around Windows idiocy of open files
apparently, we may be able to rewrite an open file if we rename it first.

i have no way to test this, unfortunately
2015-10-17 21:59:51 -04:00
Antoine Beaupré e59f1d50d9 fix tests: umask is not defined in the repo yet 2015-10-17 21:56:25 -04:00
Antoine Beaupré ce72051284 rename cache variable to index for clarity 2015-10-17 21:50:16 -04:00
Antoine Beaupré 07e7f2dcad fixup warning message 2015-10-17 21:50:14 -04:00
Antoine Beaupré fb68c6ff70 always convert the chunks cache we find
it seems it is possible that the chunks files are copied but *not*
converted. this may have happened here because the conversion was
interrupted, although the specific scenario is still unclear (but it
did happen during manual tests here). therefore reproducing this
problem seems to be difficult, hence the lack of tests for this
specific issue.

since we consider the header replacement code to be safe, that we
always convert shouldn't pose any additional threat to the existing
borg chunk cache.
2015-10-17 21:44:50 -04:00
Antoine Beaupré c4599d8ea4 fix cascading failure with the index conversion code
this resolves bug #something where the index file could not be
converted, completely breaking conversion.

it seems that, during some refactoring, the index conversion code was
completely dropped. this was missed by the unit tests because the repo
can still be opened by the constructor even though the index is
invalid, so tests need improvements there.
2015-10-17 21:44:50 -04:00
anarcat 2bd2267969 Merge pull request #296 from ThomasWaldmann/one-file-system-option
use --one-file-system (as "du" tool does) instead of --do-not-cross-mountpoints
2015-10-17 21:24:40 -04:00
Antoine Beaupré 92ac120fb0 default progress display to true if on a tty
this makes --progress a toggle: if there's a terminal, it turns it
off, if there isn't, it forces it on.
2015-10-17 21:21:33 -04:00
Antoine Beaupré e4f325182e Revert "no need to flush stderr in my tests"
It is actually necessary, for now.

This reverts commit 8fdd1eddf9.

Conflicts:
	borg/helpers.py
2015-10-17 21:21:33 -04:00
Antoine Beaupré ce1aaa9dfa Revert "fix conflict between --stats and --progress"
the columns handling fixed that isssue more elegantly

This reverts commit 7f77778419.

Conflicts:
	borg/helpers.py
2015-10-17 21:21:33 -04:00
Antoine Beaupré 0d8525ad8f add missing lines 2015-10-17 21:21:33 -04:00
Antoine Beaupré d666c86bfc adjust display of --stats
it was broken by recent commits.

also remove the __format__() anti-pattern from cache as well.
2015-10-17 21:21:33 -04:00
Antoine Beaupré 28d0a9ce84 human-readable representation of stats 2015-10-17 21:21:32 -04:00
Antoine Beaupré 66bfc6fce8 get rid of print_() function
this function was over-coupling the cache system and the statistics
module. they are now almost decoupled insofar as the cache system has
its own rendering system now that is called separately.

furthermore, we have a much more flexible formatting system that is
used coherently between --progress and --stats

the degenerate case here is if we want to change the label in the
statistics summary: in this case we need to override the default
__str__() representation to insert our own label.
2015-10-17 21:21:32 -04:00
Antoine Beaupré 118ee8679f remove custom microformat from stats
i saw the errors in my ways: __format__ is only to customize the
"format mini-language", what comes after ":" in a new string
format. unfortunately, we cannot easily refer to individual fields in
there, short of re-implementing a new formatting language, which seems
silly.

instead, we use properties to extract human-readable versions of the
statistics. more intuitive and certainly a more common pattern than
the exotic __format__().

also add unit tests to prove this all works
2015-10-17 21:21:32 -04:00
Antoine Beaupré 6b547e6554 no need to flush stderr in my tests 2015-10-17 21:21:32 -04:00
Antoine Beaupré 75c993b875 simplify progress display
we stop enforcing a minimum width for fields, it changes only on
logarithmic boundaries, so not a big problem. string conversion is
implicit

this gives us a little more width for the path
2015-10-17 21:21:32 -04:00
Antoine Beaupré ecae163072 use all available columns for path in progress
we use the new get_terminal_size() function, with a fallback for
Python 3.2. we default to 80 columns.

then we generate the stats bit and fill the rest with the path, as
previously, but with a possibly larger field.

note that this works with resizes in my test (uxterm)
2015-10-17 21:21:32 -04:00
Antoine Beaupré 7f28244cfe fix conflict between --stats and --progress
the --stats output would be slightly garbled by --progress, because of
the \r that is output at the last line...

example:

    initializing cache
    reading files cache
    processing files
    ------------------------------------------------------------------------------ s/twotone
    Archive name: 2015-10-15-test
2015-10-17 21:21:32 -04:00
Antoine Beaupré e8cf28f9a0 only show all files found in debug
a single -v flag shouldn't flood the console with all the files in the
path specified, it makes -v basically useless

this way, -v can also be used with --progress to have nicer output:

initializing cache
reading files cache
processing files
  5.20 GB O   2.66 GB C  25.13 MB D 27576 N baz/...
2015-10-17 21:21:31 -04:00
Antoine Beaupré d8f8076984 factor out status output so it is consistent
as it was, surrogates were not always removed, for example

we may also want to output at different levels or control if we want
to print unchanged files and so on
2015-10-17 21:21:31 -04:00
Antoine Beaupré b120e5f119 output more progress information
without this, there would be a solid 20 seconds here without any sort
of output on the console, regardless of the verbosity level. this
makes nice incremental messages telling the user that borg is not
stalled (or waiting for a lock, for that matter)

the "processing files" message is a little clunky, as we somewhat
abuse the cache to figure out if we are just starting... but it helps
if there are problems reading the actual files: it tells us the
initialization is basically complete and we're going ahead with the
reading of all the files.
2015-10-17 21:21:31 -04:00
Antoine Beaupré 3827c2b107 also show the number of files processed in --progress 2015-10-17 21:21:31 -04:00
Antoine Beaupré 3eebea8745 explain what the --progress flag does exactly
an alternative to this would be to use more than a letter in the
output, for example:

 orig: 16.82 GB comp: 9.44 GB dedup: 25.86 MB home/
2015-10-17 21:21:31 -04:00
Thomas Waldmann 6712bdf456 include a list of major changes compared to attic into the README, fixes #224 2015-10-17 22:27:59 +02:00
Thomas Waldmann 908e944bdd include the help offered by borg help <topic> in the usage docs, fixes #293
it's not pretty yet, but better than nothing.
2015-10-17 21:53:09 +02:00
Thomas Waldmann fc78b44923 use --one-file-system (as "du" tool does) instead of --do-not-cross-mountpoints
the old name still works, but emits a deprecation warning suggesting the new name.

this is a followup to 4fd06e2634, which added "-x" (as seen in "du").
2015-10-17 20:55:07 +02:00
TW 0e9c524266 Merge pull request #292 from anarcat/rename-usage
add missing rename usage to docs, closes #279
2015-10-17 20:06:23 +02:00
TW 2dda41dc2a Merge pull request #294 from anarcat/x-option
add -x flag, common to GNU utilities
2015-10-17 20:06:02 +02:00
TW b0f66e84a9 Merge pull request #295 from yagebu/master
docs: restructure installation.rst + add OS X install instructions
2015-10-17 20:05:36 +02:00
Jakob Schnitzer 462bb41e57 docs: AUTHORS: one borg in the title is enough 2015-10-17 18:40:15 +02:00
Jakob Schnitzer 7cc3becad9 docs: restructure installation.rst, add OSX instructions
Moved the list of dependencies to the corresponding subsection.
Collected all preparation steps under one heading.
Added link to the Arch Linux AUR package.
Install docs for OS X.
2015-10-17 18:40:15 +02:00
Antoine Beaupré eacb0b9e83 Merge branch 'logging-refactor' into upstream 2015-10-17 12:29:52 -04:00
Antoine Beaupré 4fd06e2634 add -x flag, common to GNU utilities
it should also probably be --one-file-system for coherence with du(1)
2015-10-17 12:23:45 -04:00
Antoine Beaupré e6c2359a60 add missing rename usage to docs, closes #279 2015-10-17 12:13:59 -04:00
Antoine Beaupré 34d0e0641c make sure hardlink copy doesn't break perms 2015-10-17 00:41:20 -04:00