Null target_class DoS

How it was found

The Object#instance_exec method in mrbgems/mruby-object-ext/src/object.c executes a block in the context of an object. It sets the VM's target_class pointer to the singleton class of this object. target_class is used as the definition target for constants and methods.

If a singleton class cannot be created for an object, target_class is set to NULL. The OP_CLASS and OP_MODULE opcodes in the VM assume target_class is not null when defining new classes and modules.

This causes a null pointer dereference and segfaults the mruby VM.

Sample code:

1.instance_exec { class X; end }