More generally, you've got parentheses, comments, and string literals,
and you need to know to ignore whitespace within any of those -- and
to ignore e.g. an open parenthesis that occurs within a comment, or a
close comment that occurs within a string literal. So inevitably
you've got to lex and parse at some level to make this work for a
practical language.
There are still some byzantine cases that ocaml+twt doesn't handle
properly, and I think it probably gets pretty close to the minimally
complex yet practically usable approach to this.

Mike

On Tue, Jun 30, 2009 at 2:58 PM, Yitzhak
Mandelbaum<yitzhak@research.att.com> wrote:
> To restart this thread, do your solutions handle the following (legal)
> variation of the original example?
> if True:
>    x = 3+4
>    y = (2 +
> 4 + 5)
>    z = 5
> else:
>    x = 5
>    if False:
>        x = 8
>        z = 2
>
> Notice that the assignment of y wraps onto the next line at an *earlier*
> column. This is legal b/c it is surrounded by parens. However, it seems that
> the preprocessing approaches will fail for this example. Do you have a
> workaround?
> --Yitzhak
>
> On Jun 12, 2009, at 11:43 AM, Andreas Rossberg wrote:
>
> On Jun 12, 2009, at 10.20 h, Andrej Bauer wrote:
>
> I think I understand the general idea of inserting "virtual" tokens,
> but the details confuse me still. So starting with
>
> if True:
>
>    x = 3
>
>    y = (2 +
>
>      4 + 5)
>
> else:
>
>    x = 5
>
>    if False:
>
>        x = 8
>
>        z = 2
>
> Martin suggests the following:
>
> {
>
> if True:
>
> ;
>
>   {
>
>   x = 3
>
>   ;
>
>   y = (2 +
>
>   ;
>
>     {
>
>     4 + 5)
>
>     }
>
>   }
>
> ;
>
> else:
>
> ;
>
>   {
>
>   x = 5
>
>   ;
>
>   if False:
>
>   ;
>
>       {
>
>       x = 8
>
>       ;
>
>       z = 2
>
>       }
>
>   }
>
> }
>
> I have two questions. Notice that the { ... } and ( ... ) need not be
> correctly nested (in the top half), so how are we going to deal with
> this? The second question is, why are there the separators after and
> just before "else:". I would expect separators inside { .... }, but
> not around "else".
>
> It depends on how exactly you define your layout rules. The usual approach
> is to tie start of layout-sensitive blocks to particular keywords -- this is
> essentially what Python and Haskell do. In that case, the binding to y is
> not affected. Haskell's rules for optional layout would rewrite your
> original program as
>
> if True:
>
>    {x = 3
>
>    ;y = (2 +
>
>      4 + 5)
>
> }else:
>
>    {x = 5
>
>    ;if False:
>
>        {x = 8
>
>        ;z = 2
>
> }}
>
> The basic rules are fairly simple:
> 1. Insert "{" (assume width 0) before the first token following a layout
> keyword (usually ":" in Python). This opens a block.
> 2. As long as inside a block, insert ";" before each token that is on the
> _same_ column as the current (i.e. innermost) "{".
> 3. A block ends as soon as you see a line whose first token is _left_ of the
> current "{". Insert "}" before that token.
> Blocks can be nested, so you need to maintain a stack of starting columns in
> the parser. Note that rule 3 may end several blocks at once. EOF is treated
> as a token at column 0.
> The way I implemented this is by wrapping the ocamllex-generated lexer with
> a function that compares each token's column with the top of the layout
> stack and inserts auxiliary tokens as necessary.
> Haskell has another rule for inserting "}" if there would be a parse error
> without it (this is to allow inline blocks). This rule is pretty fudgy, and
> almost impossible to implement properly with a conventional parser
> generator. IMO, the only sane way to reformulate this rule is again to tie
> it to specific keywords, e.g. insert "}" before "else" if missing. This can
> be implemented in the parser by making closing braces optional in the right
> places.
> - Andreas
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
> --------------------------------------------------
> Yitzhak Mandelbaum
> AT&T Labs - Research
> http://www.research.att.com/~yitzhak
>
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>