This is a fixup for #3155, which was broken on at least python <= 3.4.2.
Also clarify when to use which *func in set_defaults.
(cherry picked from commit 5ce4fa9704)
do no read/archive bsdflags: borg create --nobsdflags ...
do not extract/set bsdflags: borg extract --nobsdflags ...
use cases:
- fs shows wrong / random bsdflags (bug in filesystem)
- fs does not support bsdflags anyway
- already archived bsdflags are wrong / unwanted
- borg shows any sort of unwanted effect due to get_flags, esp. on Linux
the nodump flag ("do not backup this file") is not honoured any more by
default because this functionality (esp. if it happened by error or
unexpected) was rather confusing and unexplainable at first to users.
if you want that "do not backup NODUMP-flagged files" behaviour, use:
borg create --exclude-nodump ...
when doing in-file checkpointing, borg creates *.borg_part_N files.
complete_file = part_1 + part_2 + ... + part_N
the source item for recreate already has a precomputed (total) size
member, thus we must force recomputation from the (partial) chunks
list to correct the size to be the part's size only.
borg create avoided this problem by computing the size member after
writing all the parts. this is now not required any more.
the bug is mostly cosmetic, borg check will complain, borg extract on
a part file would also complain. but all the complaints only refer to
the wrong metadata of the part files, the part files' contents are
correct.
usually you will never extract or look at part files, but only deal
with the full file, which will be completely valid, all metadata and
content.
you can get rid of the archives with these cosmetic errors by running
borg recreate on them with a fixed borg version. the old part files
will get dropped (because they are usually ignored) and any new part
file created due to checkpointing will be correct.
opening a device file for a non-existing device can be very slow.
symlinks will make the open() call fail as it is using O_NOFOLLOW.
also: lstat -> stat(..., follow_symlinks=False) like everywhere else.
move --no-files-cache from common to borg create options, fixes#3146
for borg prune, just use do_files=False (it only needs the chunks
cache, not the files cache).
if it is not significantly better compressed, we just store lz4
compressed data (which we already have computed anyway), because
that at least decompressed super fast.
You can now control the files cache mode using this option:
--files-cache={ctime,mtime,size,inode,rechunk,disabled}*
(only some combinations are supported)
Previously, only these modes were supported:
- mtime,size,inode (default of borg < 1.1.0rc4)
- mtime,size (by using --ignore-inode)
- disabled (by using --no-files-cache)
Now, you additionally get:
- ctime alternatively to mtime (more safe), e.g.:
ctime,size,inode (this is the new default of borg >= 1.1.0rc4)
- rechunk (consider all files as changed, rechunk them)
Deprecated:
- --ignore-inodes (use modes without "inode")
- --no-files-cache (use "disabled" mode)
The tests needed some changes:
- previously, we use os.utime() to set a files mtime (atime) to specific
values, but that does not work for ctime.
- now use time.sleep() to create the "latest file" that usually does
not end up in the files cache (see FAQ)
refactor: make a generally usable function
fix: remove support code for ancient pyinstaller
the "else" branch was needed for pyinstaller < 20160820 because it did
not have the LD_LIBRARY_PATH_ORIG env var, so we just killed LDLP
because we had no better way.
but with borg tests running under fakeroot, this is troublesome as
fakeroot uses this also and can't find its library without it.
so, just remove it, we do not need to support old pyinstaller.
due to block buffering (in borg, pipes, sshd, ssh) partial lines might
be received. for plain text, this causes cosmetic issues, for json it
causes tracebacks due to parse errors.
the code now makes sure handle_remote_line() only gets called with a
complete line (which is terminated by any universal newline char, a
pure \r seems to be needed for remote progress displays).
it also fixes a yet undiscovered partial utf-8-sequence decoding issue
that might occur for the same reason.
the client_supports_log_v3 flag was added to differentiate 1.1.0 beta3
to beta5 clients (which did not support parsing json log format from
server) from >= 1.1.0beta6 clients (which support it).
for clients older than 1.1.0b3, no json log format will be negotiated
anyway.
by removing the client_supports_log_v3 flag support, we drop support for
clients using 1.1.0beta3..5.
thus, a client is now expected to either support old log format (like
borg 1.0.x) or new json format (like borg 1.1.0 >= beta6).
client server comment
===========================================
any 0.29+ uses $LOG plain remote log format
any 1.0.x uses $LOG plain remote log format
1.0.x 1.1.0 uses $LOG plain remote log format
1.1.0b1/b2 1.1.0 (uses $LOG plain remote log format)
1.1.0b3-b5 1.1.0 (malfunction)
1.1.0b6 1.1.0 (uses json remote log format)
1.1.0rc 1.1.0 uses json remote log format
1.1.x 1.1.0 uses json remote log format
(beta testing is over and betas are unsupported now)
Note: client_supports_log_v3 flag was added in changeset
18a2902c9c
if borg stderr is not connected to a tty, but to ssh (when using
borg client/server), sys.stderr is block buffered (tty: line buffered).
thus we better flush explicitly after emitting a line as the receiving
side can not handle partial json at the end of the block.
also, it might solve some delays, when output didn't arrive at
receiving side in time.
Antti Aalto