.TH bup-memtest 1 "2013-12-26" "Bup debian/0.25-1~bpo70+1" .SH NAME .PP bup-memtest - test bup memory usage statistics .SH SYNOPSIS .PP bup memtest [options...] .SH DESCRIPTION .PP \f[C]bup\ memtest\f[] opens the list of pack indexes in your bup repository, then searches the list for a series of nonexistent objects, printing memory usage statistics after each cycle. .PP Because of the way Unix systems work, the output will usually show a large (and unchanging) value in the VmSize column, because mapping the index files in the first place takes a certain amount of virtual address space. However, this virtual memory usage is entirely virtual; it doesn\[aq]t take any of your RAM. Over time, bup uses \f[I]parts\f[] of the indexes, which need to be loaded from disk, and this is what causes an increase in the VmRSS column. .SH OPTIONS .TP .B -n, --number=\f[I]number\f[] set the number of objects to search for during each cycle (ie. before printing a line of output) .RS .RE .TP .B -c, --cycles=\f[I]cycles\f[] set the number of cycles (ie. the number of lines of output after the first). The first line of output is always 0 (ie. the baseline before searching for any objects). .RS .RE .TP .B --ignore-midx ignore any \f[C]\&.midx\f[] files created by \f[C]bup\ midx\f[]. This allows you to compare memory performance with and without using midx. .RS .RE .TP .B --existing search for existing objects instead of searching for random nonexistent ones. This can greatly affect memory usage and performance. Note that most of the time, \f[C]bup\ save\f[] spends most of its time searching for nonexistent objects, since existing ones are probably in unmodified files that we won\[aq]t be trying to back up anyway. So the default behaviour reflects real bup performance more accurately. But you might want this option anyway just to make sure you haven\[aq]t made searching for existing objects much worse than before. .RS .RE .SH EXAMPLE .IP .nf \f[C] $\ bup\ memtest\ -n300\ -c5 PackIdxList:\ using\ 1\ index. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ VmSize\ \ \ \ \ \ VmRSS\ \ \ \ \ VmData\ \ \ \ \ \ VmStk\ \ \ \ \ \ \ \ \ 0\ \ \ \ 20824\ kB\ \ \ \ 4528\ kB\ \ \ \ 1980\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ \ 300\ \ \ \ 20828\ kB\ \ \ \ 5828\ kB\ \ \ \ 1984\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ \ 600\ \ \ \ 20828\ kB\ \ \ \ 6844\ kB\ \ \ \ 1984\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ \ 900\ \ \ \ 20828\ kB\ \ \ \ 7836\ kB\ \ \ \ 1984\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ 1200\ \ \ \ 20828\ kB\ \ \ \ 8736\ kB\ \ \ \ 1984\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ 1500\ \ \ \ 20828\ kB\ \ \ \ 9452\ kB\ \ \ \ 1984\ kB\ \ \ \ \ \ 84\ kB\ $\ bup\ memtest\ -n300\ -c5\ --ignore-midx PackIdxList:\ using\ 361\ indexes. \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ VmSize\ \ \ \ \ \ VmRSS\ \ \ \ \ VmData\ \ \ \ \ \ VmStk\ \ \ \ \ \ \ \ \ 0\ \ \ \ 27444\ kB\ \ \ \ 6552\ kB\ \ \ \ 2516\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ \ 300\ \ \ \ 27448\ kB\ \ \ 15832\ kB\ \ \ \ 2520\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ \ 600\ \ \ \ 27448\ kB\ \ \ 17220\ kB\ \ \ \ 2520\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ \ 900\ \ \ \ 27448\ kB\ \ \ 18012\ kB\ \ \ \ 2520\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ 1200\ \ \ \ 27448\ kB\ \ \ 18388\ kB\ \ \ \ 2520\ kB\ \ \ \ \ \ 84\ kB\ \ \ \ \ \ 1500\ \ \ \ 27448\ kB\ \ \ 18556\ kB\ \ \ \ 2520\ kB\ \ \ \ \ \ 84\ kB\ \f[] .fi .SH DISCUSSION .PP When optimizing bup indexing, the first goal is to keep the VmRSS reasonably low. However, it might eventually be necessary to swap in all the indexes, simply because you\[aq]re searching for a lot of objects, and this will cause your RSS to grow as large as VmSize eventually. .PP The key word here is \f[I]eventually\f[]. As long as VmRSS grows reasonably slowly, the amount of disk activity caused by accessing pack indexes is reasonably small. If it grows quickly, bup will probably spend most of its time swapping index data from disk instead of actually running your backup, so backups will run very slowly. .PP The purpose of \f[C]bup\ memtest\f[] is to give you an idea of how fast your memory usage is growing, and to help in optimizing bup for better memory use. If you have memory problems you might be asked to send the output of \f[C]bup\ memtest\f[] to help diagnose the problems. .PP Tip: try using \f[C]bup\ midx\ -a\f[] or \f[C]bup\ midx\ -f\f[] to see if it helps reduce your memory usage. .PP Trivia: index memory usage in bup (or git) is only really a problem when adding a large number of previously unseen objects. This is because for each object, we need to absolutely confirm that it isn\[aq]t already in the database, which requires us to search through \f[I]all\f[] the existing pack indexes to ensure that none of them contain the object in question. In the more obvious case of searching for objects that \f[I]do\f[] exist, the objects being searched for are typically related in some way, which means they probably all exist in a small number of packfiles, so memory usage will be constrained to just those packfile indexes. .PP Since git users typically don\[aq]t add a lot of files in a single run, git doesn\[aq]t really need a program like \f[C]bup\ midx\f[]. bup, on the other hand, spends most of its time backing up files it hasn\[aq]t seen before, so its memory usage patterns are different. .SH SEE ALSO .PP \f[C]bup-midx\f[](1) .SH BUP .PP Part of the \f[C]bup\f[](1) suite. .SH AUTHORS Avery Pennarun .