use posix_fadvise to avoid spoiling the OS cache

note:
 - we call this frequently AFTER re-filling the chunker buffer,
so even big input files have little cache impact.
- there is still some cache impact due to output files caching,
if the repository is on a locally mounted filesystem.
This commit is contained in:
Thomas Waldmann 2015-04-11 01:09:03 +02:00
parent 7ad1093951
commit c7d232c4ce
1 changed files with 9 additions and 0 deletions

View File

@ -1,4 +1,5 @@
#include <Python.h>
#include <fcntl.h>
/* Cyclic polynomial / buzhash: https://en.wikipedia.org/wiki/Rolling_hash */
@ -153,6 +154,14 @@ chunker_fill(Chunker *c)
// some error happened
return 0;
}
#if ( _XOPEN_SOURCE >= 600 || _POSIX_C_SOURCE >= 200112L )
// We tell the OS that we do not need the data of this file any more
// that it maybe has in the cache. This avoids that we spoil the
// complete cache with data that we only read once and (due to cache
// size limit) kick out data from the cache that might be still useful
// for the OS or other processes.
posix_fadvise(c->fh, (off_t) 0, (off_t) 0, POSIX_FADV_DONTNEED);
#endif
}
else {
// no os-level file descriptor, use Python file object API