English version
Accueil     À propos     Téléchargement     Ressources     Contactez-nous    

Ce site est rarement mis à jour. Pour les informations les plus récentes, rendez-vous sur le nouveau site OCaml à l'adresse ocaml.org.

Browse thread
copy the rest of a file after scanning
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: 2008-03-17 (23:30)
From: viktor tron <viktor.tron.ml@g...>
Subject: copy the rest of a file after scanning
Dear list,
I have the funniest problem.

I use Scan to scan a file, and output an edited variant, when edits are done
I just need to copy the remainder of the file.

This ludicrious task is proving more elusive to handle than the whole

The problem is that when I finish editing, the scanning buffer is active and
not empty and
my reading position in the input channel is not where I am currently at in

I don't want to use scanner to consume the remaining 3 terrabytes digesting
it line by line. I can't seem to be able to scan bigger chunks than lines
either unless I can name a character
that certainly does not appear in the text I am reading.
If there is a character like this, say @, then I am ok with

scan "%s@@%!" (fun s -> Printf.fprintf corpus_out "%s" s)

this one reads till the next @ character which is ignored or the end of the
input, which is checked with putting %! explicitly.
This passes my tests, but horribly ugly, since there is no character that I
can guarantee this way.
Plus I might not have memory for passing this whole chunk as one string if
the file is large.

So as an alternative I did this:

(* we set the input channel reading position to where we are in scanning *)
let _ = scan "%n" (fun x -> seek_in corpus_in (x - 1)) in
(* and then dump the rest trivially in chunks of buf_size chars *)
let buf = String.create buf_size in
let rec dump () =
let len = input corpus_in buf 0 buf_size in
if len > 0 then (output corpus_out buf 0 len; dump () )

or in one go

let end_pos = in_channel_length corpus_in
let len = end_pos - pos_in corpus_in in
let s = String.create len in
let _ = really_input corpus_in s 0 len in
Printf.fprintf corpus_out "%s" s;

On my mac and linux, all works smoothly, till I used it on windows.
3) does not work on windows, since in_channel_length and seek do not take
into account the newline translations that take
place at reading.
Or in other words, the scan module reports character positions incorrectly
since CRLF=\013\010 is counted as one character and matched by \n.

But then again, 2 seems to work, but have no clue why seek is not the same
as in_channel_length when it comes to counting chars.
In which case it should not work either.

Any thoughts?