> Is it impossible to have modules and classes combined recursively?

Currently, yes.  A module definition cannot be mutually recursive with
any other definition (type, class, or another module).  This raises a
number of difficult typechecking and compilation problems that are
still largely open.  I agree this is probably the most irritating
restriction of ML modules.

> If yes, what would be a good workaround?

I'm afraid you just have to define your own "set" type, without using
the Set.Make functor.  E.g.

        type 'a myset = Empty | Node of myset * 'a * myset

        (* an efficient implementation of myset can easily be
           lifted from the sources of the "Set" standard library
           module *)

        class foo =
          object
            method bar = (... : foo myset)
            ...
          end

This isn't very satisfactory, but I'm afraid we can't do better until
recursive modules are better understood.

Regards,

- Xavier Leroy