Uninitialized memory in float out values can lead to crashes in mozjs gc code later on
When calling a C function with a float
out value that exits early due to an error before writing anything to the out value, this can lead to a crash in mozjs during GC later. I assume the uninitialized memory happens to correspond to one of the NaN values that indicate a JS::Value
type that contains a pointer that needs to be freed when garbage collected. If that is indeed the cause, maybe NaN values could be normalized before being passed to mozjs or maybe out values could be zero initialized for floats/doubles.
The effect of this could be seen in gnome-shell-extensions#469 (closed) which was caused by the extension calling LayoutManager::findIndexForActor()
-> clutter_actor_get_transformed_position()
-> clutter_actor_apply_transform_to_point()
-> _clutter_actor_fully_transform_vertices()
before the actor had been added to the stage, leading to the out value being uninitialized memory. This then lead to crashes with backtraces like this one found in https://bugs.archlinux.org/task/80132#comment223433:
#0 js::gc::Cell::storeBuffer() const (this=<optimized out>) at /usr/src/debug/js115/firefox-115.4.0/js/src/gc/Cell.h:399
#1 js::HeapSlot::post(js::NativeObject*, js::HeapSlot::Kind, unsigned int, JS::Value const&) (this=0xd6f09b73640, owner=0xd6f09b73628, kind=js::HeapSlot::Slot, slot=0, target=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/gc/Barrier.h:1004
#2 js::HeapSlot::set(js::NativeObject*, js::HeapSlot::Kind, unsigned int, JS::Value const&) (this=0xd6f09b73640, owner=0xd6f09b73628, kind=js::HeapSlot::Slot, slot=0, v=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/gc/Barrier.h:993
#3 js::NativeObject::setSlot(unsigned int, JS::Value const&) (this=0xd6f09b73628, slot=0, value=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/NativeObject.h:1170
#4 AddOrChangeProperty<(IsAddOrChange)1>(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::PropertyDescriptor>, js::PropertyResult*) (cx=0x55ba49d92750, obj=..., id=..., existing=0x7fff6eb28688, desc=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/NativeObject.cpp:1331
#5 js::NativeDefineProperty(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::PropertyDescriptor>, JS::ObjectOpResult&) (cx=<optimized out>, obj=..., id=..., desc_=..., result=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/NativeObject.cpp:1763
#6 0x00007f2c9b41dcd4 in js::DefineDataProperty(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::Value>, unsigned int, JS::ObjectOpResult&) (cx=0x55ba49d92750, obj=..., id=..., value=..., attrs=<optimized out>, result=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/JSObject.cpp:2094
#7 js::DefineDataProperty(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::PropertyKey>, JS::Handle<JS::Value>, unsigned int) (cx=0x55ba49d92750, obj=..., id=..., value=..., attrs=<optimized out>) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/JSObject.cpp:2115
#8 js::DefineDataProperty(JSContext*, JS::Handle<JSObject*>, js::PropertyName*, JS::Handle<JS::Value>, unsigned int) (cx=0x55ba49d92750, obj=..., name=<optimized out>, value=..., attrs=<optimized out>) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/JSObject.cpp:2129
#9 0x00007f2c9b10b499 in js::InitPropertyOperation(JSContext*, unsigned char*, JS::Handle<JSObject*>, JS::Handle<js::PropertyName*>, JS::Handle<JS::Value>) (cx=0x55ba49d92750, pc=<optimized out>, obj=..., name=..., rhs=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter-inl.h:296
#10 js::Interpret(JSContext*, js::RunState&) (cx=<optimized out>, state=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter.cpp:4067
#11 0x00007f2c9b0fb55d in MaybeEnterInterpreterTrampoline(JSContext*, js::RunState&) (cx=0x55ba49d92750, state=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter.cpp:400
#12 js::RunScript(JSContext*, js::RunState&) (cx=0x55ba49d92750, state=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter.cpp:458
#13 js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) (cx=0x55ba49d92750, args=..., construct=js::NO_CONSTRUCT, reason=<optimized out>) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter.cpp:612
#14 InternalCall(JSContext*, js::AnyInvokeArgs const&, js::CallReason) (cx=0x55ba49d92750, args=..., reason=<optimized out>) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter.cpp:647
#15 js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) (cx=0x55ba49d92750, fval=..., thisv=..., args=..., rval=..., reason=<optimized out>) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/Interpreter.cpp:679
#16 0x00007f2c9b80c3a7 in JS_CallFunctionValue(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) (cx=0x55ba49d92750, obj=..., fval=..., args=..., rval=...) at /usr/src/debug/js115/firefox-115.4.0/js/src/vm/CallAndConstruct.cpp:53
#17 0x00007f2c9cfbd797 in JS::Call (rval=..., args=..., fun=..., thisObj=..., cx=<optimized out>) at /usr/include/mozjs-115/js/CallAndConstruct.h:92
#18 Gjs::Closure::invoke(JS::Handle<JSObject*>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) (this=0x55ba4c21a840, this_obj=..., args=..., retval=...) at ../gjs/gi/closure.cpp:184
#19 0x00007f2c9cffaeaf in Gjs::Closure::marshal(_GValue*, unsigned int, _GValue const*, void*, void*) (this=0x55ba4c21a840, return_value=0x0, n_param_values=2, param_values=0x7fff6eb29410, invocation_hint=<optimized out>, marshal_data=<optimized out>) at /usr/include/mozjs-115/js/RootingAPI.h:614
#20 0x00007f2c9d6fb6c0 in g_closure_invoke (closure=0x55ba4c21a840, return_value=0x0, n_param_values=2, param_values=0x7fff6eb29410, invocation_hint=0x7fff6eb29360) at ../glib/gobject/gclosure.c:832
#21 0x00007f2c9d729a36 in signal_emit_unlocked_R.isra.0 (node=node@entry=0x7fff6eb29500, detail=detail@entry=0, instance=instance@entry=0x55ba4a1d9d40, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7fff6eb29410) at ../glib/gobject/gsignal.c:3980
#22 0x00007f2c9d71aa42 in signal_emit_valist_unlocked (instance=instance@entry=0x55ba4a1d9d40, signal_id=signal_id@entry=241, detail=detail@entry=0, var_args=var_args@entry=0x7fff6eb29660) at ../glib/gobject/gsignal.c:3612
#23 0x00007f2c9d71ac77 in g_signal_emit_valist (instance=0x55ba4a1d9d40, signal_id=241, detail=0, var_args=var_args@entry=0x7fff6eb29660) at ../glib/gobject/gsignal.c:3355
#24 0x00007f2c9d71ad34 in g_signal_emit (instance=instance@entry=0x55ba4a1d9d40, signal_id=<optimized out>, detail=detail@entry=0) at ../glib/gobject/gsignal.c:3675
#25 0x00007f2c9cb02305 in meta_workspace_add_window (workspace=0x55ba4a1d9d40, window=0x55ba4bfd18c0) at ../mutter/src/core/workspace.c:387
...