Original bug ID: 614
Reporter: administrator
Status: closed
Resolution: fixed
Priority: normal
Severity: minor
Category: ~DO NOT USE (was: OCaml general)

Bug description

Hello,

I have started using your Unison program. I would like to thank
you for the program and congratulate on the work well done. Unison
is a great utility and very impressive. I have found it via
a search at Google.

I have a fairly complex environment with several machines at work and
several at home. Each of the machines is used for a specific purpose,
so a straightforward file synchronization does not work for me. However,
a workable Unison based solution with different combinations of path and
ignore commands is easy.

Your documentation helped enormously, especially Common Problems.
I have hit:
* The text mode user interface fails with Uncaught exception Sys_blocked_io'' when running over ssh2. * The command line unison work ssh://remote.dcs.ed.ac.uk/work fails, with fatal error: could not connect to server.''
The first one was simple. The second one required more
investigation. It looks that ssh does not execute shell initialization
sequence, which sets up user specific PATH environment variable. On
some systems, my solution is to set the PATH in the .bashrc file. However,
this does not seem to work on all systems. A general solution is to
copy the unison executable to a directory with system executables,
/usr/bin is a good choice. A drawback is that this solution requires
system root privileges.

My total environment is quite large, consisting of around 170,000 files
occupying over 10Gb of storage. While 130,000 files works OK, I have
started to hit some limits with 170,000 files. I have encountered the
following problems:

    - unison core dumps. This happens during the "Looking for changes"
      phase in the following scenario:
            - synchronize roots with a sequence of ignore and 
		path commands
            - change ignore and path commands, but keep 
		the same .prf file
            - run unison again
            - unison core dumps
      I am using Unison 2.7.7 on Linux. Since I can overcome 
  this problem by having multiple .prf files, I have not looked 
  at details.

    - unison enters an infinite loop during the "Reconciling changes"
      phase. The process uses almost 100% of CPU time, while its
  memory consumption stays constant. This problem looked
  interesting, so I gathered some more details.

I took Ocaml 3.0.2 and Unison 2.7.7 sources and rebuilt both executables.
In Unison, I changed line 264 in recon.ml from:
(* debug (fun() -> Util.msg1 "reconcile %s\n" (path2localString path)); *)
to:
debug (fun() -> Util.msg1 "reconcile %s\n" (Path.toString path));
Unison enters an infinite loop while reconciling around 8,500th file.
Different runs fail at a different file, but the number around 8,500
is fairly consistent. At that point, the Unison memory consumption
stands at around 60Mb.

It seems that the problem is with the Ocaml garbage collection, not
with the Unison. When I attach to the Unison process with gdb, I get
the following printout:
Attaching to program: /home/rok/bin/unison, process 887
Reading symbols from /lib/libm.so.6...done.
Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done.
add_to_heap (mem=0x42c3e000) at memory.c:126
126 memory.c: No such file or directory.
(gdb) where
#0 add_to_heap (mem=0x42c3e000) at memory.c:126
#1 0x809f170 in expand_heap (request=6) at memory.c:162
#2 0x809f293 in alloc_shr (wosize=6, tag=250) at memory.c:230
#3 0x809ebe6 in oldify (v=1074922940, p=0xbffff814) at minor_gc.c:97
#4 0x809da1e in oldify_local_roots () at roots.c:180
#5 0x809eca0 in empty_minor_heap () at minor_gc.c:131
#6 0x809ed4e in minor_collection () at minor_gc.c:159
#7 0x809de05 in garbage_collection () at signals.c:100
#8 0x80a7d70 in caml_call_gc ()
#9 0x1 in ?? ()
#10 0x416c6138 in ?? ()
#11 0x1 in ?? ()
#12 0x7ff in ?? ()
Cannot access memory at address 0x3c9.
(gdb) step
127 in memory.c
(gdb)
128 in memory.c
(gdb) print cur
$1 = 0x41576000 ""
(gdb) step
126 in memory.c
(gdb)
127 in memory.c
(gdb)
128 in memory.c
(gdb)
126 in memory.c
(gdb)
127 in memory.c
(gdb)
128 in memory.c
(gdb) print cur
$2 = 0x41576000 ""
The infinite loop is in the following code in memory.c, since cur does not
change its value during loop iterations:
/* Chain this heap block. */
{
char **last = &heap_start;
char *cur = *last;

while (cur != NULL && cur < m){
  last = &(Chunk_next (cur));
  cur = *last;
}
Chunk_next (m) = cur;
*last = m;

}

If you have any ideas on how to proceed from here, please let me know.
I am also sending this report to the Ocaml bugs mailing list.

Thanks again,
Rok