Bigarray reshape wrong for 64 bit systems #5511
Comments
|
Comment author: gerd On the second glance, I think the test "size > 0x7FFFFFFFL" (for 32 bit only) belongs into caml_ba_alloc, after multiplying the dimensions. The real problem is that "size" (and num_elts) is unsigned, but the dimensions are signed. So, when reshaping, it may happen that a positive size is interpreted as a negative dimension (e.g. when a 1000 * 3_000_000 char array is reshaped into a one-dimension 3_000_000_000 array). IMHO, this should better be caught in caml_ba_alloc, by just disallowing that large arrays. I'm not absolutely sure whether such a case actually exists, because one has to pass the new dimensions in to reshape, and is limited by max_int anyway. |
|
Comment author: @xavierleroy The intent of this test was indeed to guarantee that bigarray dimensions always fit in 32-bit signed, for compatibility between 32- and 64-bit platforms. As discussed in the related PR, this guarantee is useless anyway, and the test in Bigarray.reshape should have been removed. Now it is (SVN commit 12180). |
Original bug ID: 5511
Reporter: gerd
Status: closed (set by @xavierleroy on 2013-08-31T10:48:35Z)
Resolution: fixed
Priority: normal
Severity: minor
Version: 3.12.1
Fixed in version: 3.13.0+dev
Category: otherlibs
Related to: #4457
Bug description
Just found these lines in bigarray_stubs.c, in caml_ba_reshape:
if (dim[i] < 0 || dim[i] > 0x7FFFFFFFL)
caml_invalid_argument("Bigarray.reshape: negative dimension");
The second condition, dim[i] > 0x7FFFFFFFL, can never be true on 32 bit platforms (because dimensions are signed), and is apparently wrong on 64 bit platforms. If I'm not totally blind, this test can just be deleted.
Steps to reproduce
On a 64 bit system:
open Bigarray;;
let g = Genarray.create char c_layout [| 0x80000000 |];;
val g :
(char, Bigarray.int8_unsigned_elt, Bigarray.c_layout) Bigarray.Genarray.t =
reshape_1 g 0x80000000;;
Exception: Invalid_argument "Bigarray.reshape: negative dimension".
The text was updated successfully, but these errors were encountered: