Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006704OCamltoplevelpublic2014-12-11 21:172019-01-11 09:11
Assigned To 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0006704: Expose more compiler-libs internals in Toploop
DescriptionThe toplevel is expunged, which means that even if I require compiler-libs.common, I get duplicate modules that are often unusable due to:

  1) exceptions having a different internal ID (this affects the entirety of Location),
  2) mutable cells having different identity.

Since 4.02 now includes module aliases, would it be possible to have Toploop re-export the entirety of compiler-libs that it expunges?
    module Compiler_libs = struct
      module Location = Location

This would greatly help those developing loadable modules that extend the toplevel, both the regular one and custom ones like utop (as utop is expunged in a similar way).
TagsNo tags attached.
Attached Files

- Relationships
related to 0006692closed Identifiers in Unicode 
related to 0007589acknowledged Improve the toplevel API 

-  Notes
gasche (administrator)
2014-12-12 19:09
edited on: 2014-12-12 19:09

I didn't know about expunge before this PR, and while I vaguely understand the comment at the top of toplevel/ (am I correct that the "" there is a search-and-replace typo for "map"?), it says what the module does but not *why* it does it. Would you care to explain why?

My understanding of the "what" is as follows: the script removes the module that are part of the standard library from the symbol table of the produced bytecode executable (this is done in the "ocaml" target of the Makefile at the root of the distribution, including the list of stdlib modules from stdlib/StdlibModules).

(How do you know that utop is expunge? I quickly looked at its Makefile and found no trace of such a process.)

whitequark (developer)
2014-12-12 19:17

Yes, it is a typo. The idea behind expunging is simple: the toplevel itself depends on compiler-libs, however we do not want many of the generic names from compiler-libs to pollute the module namespace. In other words we want to keep the module namespace almost same as in ocamlc (Toploop and Topdirs are also provided).

Expunging does this by removing the *non*-stdlib modules from the internal symbol table of the runtime, which is also used by Dynlink, and by Dynlink-like code that loads the newly compiled bytecode into the toplevel. The effect of this is that the toplevel code itself still refers to compiler-libs, however if you load compiler-libs explicitly, you have a "split brain" situation: Dynlink acts as if compiler-libs was never loaded, loads a new copy, and resolves all future references to it.

However, the toplevel itself leaks outside a few implementation details of compiler-libs, such as: Lexer.Error and Syntaxerr.Error (which would appear if you try to invoke !Toplevel.parse_toplevel_phrase); the many references to Outcometree, Path, Env and Types from Toploop; the behavior of Toplevel-internal Location module, unable to catch any exceptions registered with Location.register_error_of_exn; probably more.

I now actually wonder if expunging is needed at all, given that compiler-libs are not in default cmi search path anymore (I'm assuming they were).

utop is expunged by: [^]
doligez (administrator)
2015-01-09 00:00

The point of expunging is to give a clean environment to the toplevel user, but it's probably a good idea to reexport compiler-libs under one module name.

typo: fixed in trunk (15771).
garrigue (manager)
2015-01-09 01:37

Since module aliases are just aliases, you cannot use them after expunging: the referenced module is no longer there.
A better approach would be to build a packed version of the compiler libraries, and use it to build the toplevel. Then there would be no need to expunge to start with.
whitequark (developer)
2015-01-09 04:57

I really like the idea of removing expunging entirely.
whitequark (developer)
2015-03-04 02:08

Another argument against expunging is that it fails if you try to load `compiler-libs.toplevel` in a rather unsound way: [^]
dbuenzli (reporter)
2016-10-11 18:17

FWIW I'm running into the same problem in another project that tries to programatically invoke toplevel directives for loading byte code in the toplevel.

My problem is 1) that whitequark mentions, basically trying to invoke Topdirs.dir_load might raise exceptions that you are neither able to catch nor to print. In my case I can't even apply horrible hacks like [^] since I don't know how to programatically generate a Symtable.Error.

Note that Toploop.{parse_toplevel_phrase,preprocessor_phrase,excute_phrase} suffer from the same problems. So as exposed the current Toploop API is completely unusable from the toplevel itself.

The smallest improvement that could be made is to provide a

  Toploop.catch : formatter -> ('a -> 'b) -> 'a -> 'b option

That catches and prints the errors all these functions may raise. I thought the current Topdir.print_exception_outcome would do the job but that's apparently not the case (Not sure what it does exactly).
dbuenzli (reporter)
2017-07-17 23:41

Also it seems that 4.04 introduced a new API in Env that allows to intercept cmi lookups.

It was suggested to me to combine this with `odig` to enable autoloading libraries (see [^] for a rough prototype on utop-full) somehow it would be nice to expose that API in the Toploop API.
nojebar (developer)
2019-01-09 13:43

What about "wrapping" compiler-libs and getting rid of expunge? Using the main Makefile terminology, I am thinking something along the following lines:

  [ $(COMMON) modules ]
    [ $(BYTECOMP) modules ]
    [ $(OPTCOMP) modules ]
    [ $(TOPLEVEL) modules ]

It would break all users of compiler-libs; we could soften the blow by distributing both the wrapped and unwrapped versions for a while.

dim (developer)
2019-01-09 14:47

I'm in favour of wrapping the compiler-libs. That would be an improvement for all users of compiler-libs as well as they define a lot of common names such as Path, Ident, Misc, ...
gasche (administrator)
2019-01-09 14:54

This is precisely what [^] does, right?
nojebar (developer)
2019-01-09 14:57

Do we prefer a single top-level module ("Compiler_libs") or several ("Ocaml_common", "Ocaml_bytecomp", "Ocaml_optcomp", "Ocaml_toplevel") ?
dim (developer)
2019-01-09 15:38

@gasche, indeed. The annoying thing is that we need to do a weird shadowing trick to hide the toplevel names. If the compiler libs where prefixed right from the start, that would make things simpler.

@nojebar, one important property to enforce is that if you use a module from bytecomp but only specify a dependency on common, then you should get a failure at compilation time rather than at link time. This is a common pitfall for users. I believe this would be easier to enforce with several top-level names, so my preference is for several top-level names.
nojebar (developer)
2019-01-11 09:10 [^]

- Issue History
Date Modified Username Field Change
2014-12-11 21:17 whitequark New Issue
2014-12-12 19:09 gasche Note Added: 0012776
2014-12-12 19:09 gasche Note Edited: 0012776 View Revisions
2014-12-12 19:17 whitequark Note Added: 0012777
2014-12-13 22:01 gasche Relationship added related to 0006692
2014-12-19 03:57 whitequark Summary Expose more compiler-libs internals Toploop => Expose more compiler-libs internals in Toploop
2015-01-09 00:00 doligez Note Added: 0013036
2015-01-09 00:01 doligez Status new => acknowledged
2015-01-09 00:01 doligez Target Version => undecided
2015-01-09 01:37 garrigue Note Added: 0013037
2015-01-09 04:57 whitequark Note Added: 0013038
2015-01-13 23:06 doligez Target Version undecided => 4.02.3+dev
2015-03-04 02:08 whitequark Note Added: 0013376
2015-05-24 19:49 gasche Severity minor => feature
2015-07-10 17:46 doligez Target Version 4.02.3+dev => 4.03.0+dev / +beta1
2016-04-18 21:08 gasche Target Version 4.03.0+dev / +beta1 => 4.03.1+dev
2016-10-11 18:17 dbuenzli Note Added: 0016400
2017-02-16 14:01 doligez Target Version 4.03.1+dev => undecided
2017-02-23 16:36 doligez Category OCaml general => -OCaml general
2017-03-01 14:38 doligez Category -OCaml general => toplevel
2017-03-01 14:38 doligez Target Version undecided =>
2017-07-17 23:41 dbuenzli Note Added: 0018087
2019-01-09 13:43 nojebar Note Added: 0019538
2019-01-09 14:47 dim Note Added: 0019539
2019-01-09 14:54 gasche Note Added: 0019540
2019-01-09 14:57 nojebar Note Added: 0019541
2019-01-09 15:38 dim Note Added: 0019542
2019-01-11 09:10 nojebar Note Added: 0019546
2019-01-11 09:11 nojebar Relationship added related to 0007589

Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker