New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bad compressibility of marshalled OCaml-data #4056
Comments
Comment author: @mmottl A quick update to this issue: the patch for "absolute sharing" contains a copyright notice for Jane Street Holding, LLC. As far as I know INRIA has a policy that copyright must be transferable to INRIA, otherwise code cannot be integrated into official distributions. Since this patch would otherwise be of rather limited use to INRIA and hence other OCaml-users, including us, we (Jane Street Holding, LLC) herewith grant INRIA the right to redistribute or use any code contained in the above file "absolute.patch" in any form without any restrictions, with our without any reference to our authorship. We, Jane Street Holding, LLC, disclaim any and all responsibility or liability arising from the use of this patch and its contained code as stated in the original patch. I hope this will make it easier for you to accept this patch. The implemented feature would be very helpful for many users who have to store large amounts of marshaled data. The only way around this problem right now is to essentially maintain a patched copy of the affected runtime files, which is quite inconvenient. The proposed patch is very simple and should not break existing code. It would be great if it could be added in an upcoming release. Thanks! |
Comment author: @mshinwell This patch from the depths (almost the dawn) of time I think no longer has the support of its (corporate) author, so let's close this issue. |
Instead of the default relative references. As noted in ocaml#4056, absolute references compress better. (We get a 15% size reduction on typical .cmt files.) Closes: ocaml#4056
Instead of the default relative references. As noted in ocaml#4056, absolute references compress better. (We get a 15% size reduction on typical .cmt files.) Closes: ocaml#4056
Instead of the default relative references. As noted in ocaml#4056, absolute references compress better. (We get a 15% size reduction on typical .cmt files.) Closes: ocaml#4056
Instead of the default relative references. As noted in ocaml#4056, absolute references compress better. (We get a 15% size reduction on typical .cmt files.) Closes: ocaml#4056
- Introduce a new header format for compressed marshaled data (see runtime/caml/intext.h) - Add a `Marshal.Compression` flag to the `Marshal.to_` functions, instructing them to produce compressed marshaled data if Zlib support is available. Otherwise, normal data is silently produced. - Unmarshaling functions (`input_value`, etc) recognize compressed marshaled data and transparently uncompress before unmarshaling, or raise an error if Zlib support is not available. - Zlib is autodetected in `configure` and can be turned off with `configure --without-zlib`. * Use compressed marshaling to write .cmt and .cmti files The resulting files are 35-40% the size of the uncompressed files. * For compressed marshaling, use absolute shared object references Instead of the default relative references. As noted in ocaml#4056, absolute references compress better. (We get a 15% size reduction on typical .cmt files.) Closes: ocaml#4056
- Introduce a new header format for compressed marshaled data (see runtime/caml/intext.h) - Add a `Marshal.Compression` flag to the `Marshal.to_` functions, instructing them to produce compressed marshaled data if ZSTD support is available. Otherwise, normal data is silently produced. - Add function `Marshal.compression_supported` to determine whether ZSTD compression is effective. - Unmarshaling functions (`input_value`, etc) recognize compressed marshaled data and transparently uncompress before unmarshaling, or raise an error if Zlib support is not available. - ZSTD is autodetected in `configure` and can be turned off with `configure --without-zstd` or made into a hard requirement with `configure --with-zstd`. - For compressed marshaling, use absolute shared object references, instead of the default relative references. As noted in ocaml#4056, absolute references compress better. (We get a 15% size reduction on typical .cmt files.) - Add `compression_supported` line to the output of `ocamlrun -config`
Original bug ID: 4056
Reporter: @mmottl
Status: closed (set by @mshinwell on 2016-12-12T15:55:49Z)
Resolution: won't fix
Priority: normal
Severity: feature
Version: 3.09.3
Category: ~DO NOT USE (was: OCaml general)
Tags: patch
Monitored by: warwick dvaillancourt BenediktGrundmann @mshinwell ogasawara jm letaris @oandrieu @mmottl
Bug description
I have just finished (and attached) a patch for the latest CVS-release
of OCaml, which adds a new marshalling flag. This flag generates a
different marshalling format, which uses absolute addresses to refer
to shared values. This fixes the problem of bad compressibility of
marshalled OCaml data. See the patch header for more information.
Since the patch is quite small and does not break any existing code,
we'd be very grateful if it could be made part of the next
OCaml-release. Thanks!
File attachments
The text was updated successfully, but these errors were encountered: