Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad compressibility of marshalled OCaml-data #4056

Closed
vicuna opened this issue Jun 28, 2006 · 2 comments
Closed

Bad compressibility of marshalled OCaml-data #4056

vicuna opened this issue Jun 28, 2006 · 2 comments

Comments

@vicuna
Copy link

vicuna commented Jun 28, 2006

Original bug ID: 4056
Reporter: @mmottl
Status: closed (set by @mshinwell on 2016-12-12T15:55:49Z)
Resolution: won't fix
Priority: normal
Severity: feature
Version: 3.09.3
Category: ~DO NOT USE (was: OCaml general)
Tags: patch
Monitored by: warwick dvaillancourt BenediktGrundmann @mshinwell ogasawara jm letaris @oandrieu @mmottl

Bug description

I have just finished (and attached) a patch for the latest CVS-release
of OCaml, which adds a new marshalling flag. This flag generates a
different marshalling format, which uses absolute addresses to refer
to shared values. This fixes the problem of bad compressibility of
marshalled OCaml data. See the patch header for more information.

Since the patch is quite small and does not break any existing code,
we'd be very grateful if it could be made part of the next
OCaml-release. Thanks!

File attachments

@vicuna
Copy link
Author

vicuna commented Apr 14, 2008

Comment author: @mmottl

A quick update to this issue: the patch for "absolute sharing" contains a copyright notice for Jane Street Holding, LLC. As far as I know INRIA has a policy that copyright must be transferable to INRIA, otherwise code cannot be integrated into official distributions.

Since this patch would otherwise be of rather limited use to INRIA and hence other OCaml-users, including us, we (Jane Street Holding, LLC) herewith grant INRIA the right to redistribute or use any code contained in the above file "absolute.patch" in any form without any restrictions, with our without any reference to our authorship. We, Jane Street Holding, LLC, disclaim any and all responsibility or liability arising from the use of this patch and its contained code as stated in the original patch.

I hope this will make it easier for you to accept this patch. The implemented feature would be very helpful for many users who have to store large amounts of marshaled data. The only way around this problem right now is to essentially maintain a patched copy of the affected runtime files, which is quite inconvenient. The proposed patch is very simple and should not break existing code. It would be great if it could be added in an upcoming release. Thanks!

@vicuna
Copy link
Author

vicuna commented Dec 12, 2016

Comment author: @mshinwell

This patch from the depths (almost the dawn) of time I think no longer has the support of its (corporate) author, so let's close this issue.

@vicuna vicuna closed this as completed Dec 12, 2016
xavierleroy added a commit to xavierleroy/ocaml that referenced this issue Feb 11, 2023
Instead of the default relative references.
As noted in ocaml#4056, absolute references compress better.
(We get a 15% size reduction on typical .cmt files.)

Closes: ocaml#4056
xavierleroy added a commit to xavierleroy/ocaml that referenced this issue Feb 11, 2023
Instead of the default relative references.
As noted in ocaml#4056, absolute references compress better.
(We get a 15% size reduction on typical .cmt files.)

Closes: ocaml#4056
xavierleroy added a commit to xavierleroy/ocaml that referenced this issue Feb 12, 2023
Instead of the default relative references.
As noted in ocaml#4056, absolute references compress better.
(We get a 15% size reduction on typical .cmt files.)

Closes: ocaml#4056
xavierleroy added a commit to xavierleroy/ocaml that referenced this issue Feb 12, 2023
Instead of the default relative references.
As noted in ocaml#4056, absolute references compress better.
(We get a 15% size reduction on typical .cmt files.)

Closes: ocaml#4056
xavierleroy added a commit to xavierleroy/ocaml that referenced this issue Feb 17, 2023
- Introduce a new header format for compressed marshaled data
    (see runtime/caml/intext.h)

- Add a `Marshal.Compression` flag to the `Marshal.to_` functions,
  instructing them to produce compressed marshaled data if Zlib support
  is available.  Otherwise, normal data is silently produced.

- Unmarshaling functions (`input_value`, etc) recognize compressed
  marshaled data and transparently uncompress before unmarshaling,
  or raise an error if Zlib support is not available.

- Zlib is autodetected in `configure` and can be turned off with
  `configure --without-zlib`.

* Use compressed marshaling to write .cmt and .cmti files

The resulting files are 35-40% the size of the uncompressed files.

* For compressed marshaling, use absolute shared object references

Instead of the default relative references.
As noted in ocaml#4056, absolute references compress better.
(We get a 15% size reduction on typical .cmt files.)

Closes: ocaml#4056
xavierleroy added a commit to xavierleroy/ocaml that referenced this issue Feb 26, 2023
- Introduce a new header format for compressed marshaled data
    (see runtime/caml/intext.h)

- Add a `Marshal.Compression` flag to the `Marshal.to_` functions,
  instructing them to produce compressed marshaled data if ZSTD support
  is available.  Otherwise, normal data is silently produced.

- Add function `Marshal.compression_supported` to determine
  whether ZSTD compression is effective.

- Unmarshaling functions (`input_value`, etc) recognize compressed
  marshaled data and transparently uncompress before unmarshaling,
  or raise an error if Zlib support is not available.

- ZSTD is autodetected in `configure` and can be turned off with
  `configure --without-zstd` or made  into a hard requirement
  with `configure --with-zstd`.

- For compressed marshaling, use absolute shared object references,
  instead of the default relative references.
  As noted in ocaml#4056, absolute references compress better.
  (We get a 15% size reduction on typical .cmt files.)

- Add `compression_supported` line to the output of `ocamlrun -config`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant