Gnumeric (introspection enabled) hangs associated with Python plugins loading GObject
Submitted by Dean McCarron
Link to original bug (#779731)
Description
There appears to be an issue with the python loader for python plugins that make use of gobject introspection (and, in some cases, other libraries as well.)
Due to the nature of the bug the first instance of the embedded python interpreter will perform as expected, however all subsequent interpreters will have the fault demonstrated. Due to the plugin-loader's setup, the first instance of the interpreter is always assigned to the python console, and later interpreters are assigned to plugins, all which will have unexpected behavior.
The workaround is somewhat painful, which is to run all code via the python console's default interpreter. This precludes any type of automatic loading, nor does it allow separation.
To replicate the issue enable both the python loader and python functions (or any other python plugin) in the plugin manager.
Then launch the python console and execute in the "Default interpreter":
from gi.respository import GObject
the command will be successful.
Now switch to any alternate interpreter context (e.g. "Python Functions" instead of "Default Interpreter" -- it may be necessary to put a formula like =py_capwords("test me") in a spreadsheet cell to activate the context so it may be accessed from the python console.)
executing the same command in "Python Functions":
from gi.respository import GObject
will hang gnumeric. This is also true within actual plugin code loaded from a file, and loading other libraries may trigger similar behavior. This will even happen with an empty plugin.
This same issue is likely behind this older issue mentioned in https://mail.gnome.org/archives/gnumeric-list/2012-May/msg00014.html (note that this is distinct from opening two differing libraries intentionally, as the rest of the thread and bug https://bugzilla.gnome.org/show_bug.cgi?id=675698 is discussing), however bug #675698 is in fact showing a similar issue, e.g. conflicts happening between libraries in two distinct interpreter contexts, even if the particular libraries used were ill-advised.
A tangentially related issue is getting plugins to load with complete features (e.g. UI functions), because the plugin-loader is checking for _PyGObject_API to be present -- since the only way to get it to be present is to load GObject, there's no way for this to happen without a hang unless the test for _PyGObject API is removed.
This thread ( https://mail.gnome.org/archives/gnumeric-list/2013-October/msg00006.html ) is a bit of a red herring, but I think it shows someone else enountering the UI problem, though the solution was a brute force one that didn't address the actual cause of the issue (the lack of the presence of library with _PyGObject_API ). Though the incorrect solution, the patch which eliminates the check (though deleted it can be accessed at https://sourceforge.net/p/fborg/code/320/tree//gnumeric/py-ui-fix.patch ) does indeed allow the UI functions to be restored as long as the associated python code never attempts to import GObject, which would cause a hang.
So that's the bug, and associated issues.
I am by no means a expert in C coding, but I spent the past few days with the Gnumeric source inserting breakpoints, and found the point where the hang occurs is (unsurprisingly) at the PyRun_String function call in run_print_string function in the gnm-py-interpreter.c. This call is part of the Python C API -- suggesting this is a python issue. The fact that this call doesn't hang on the first instance of the interpreter (e.g. the python console) but does on other instances is I think a big clue to the problem. Indeed, after a dirty hack that forced all plugins to use the same (e.g. first) context as the console, the hang caused by loading GObject did not occur.
The Gnumeric python-loader implements multiple python interpreters presumably with the intent to have distinctly separate contexts, e.g. so that plugin #1's code doesn't interfere with plugin #2. However, rather than creating new, distinct python interpreter objects, python-loader uses sub-interpreters (and indeed a substantial portion of the code in gnm-py-interpreter is dedicated to managing and switching these sub-interpreters.) I strongly suspect this is the source of the strange behavior, noted here -- https://docs.python.org/2/c-api/init.html#bugs-and-caveats -- the separation is imperfect, and some extensions may not work properly, and that certain things like inits are not called on second can later instances of interpreters; the overwhelming evidence is that GObject in the gi.repository is one such extension. Elsewhere there are reports that sub-interpreters are inconsistent and their use appears discouraged by seasoned python coders. The Python C API document also indicates that getting non-Python calls thread-safe with sub-interpreters (possibly part of our problem) is not supported. I would suggest sub-interpreters (today, at least) are the wrong choice of implementation of separate python interpreter contexts.
While none of this is a fault of Gnumeric per se, if the above is true, the implementation of sub-interpreters enables this fault. I suspect the best fix to the bug is for python-loader to be rewritten to eliminate sub-interpreters, e.g. just create a unique python interpreter object per plugin, this fault would go away (and a side effect, the interpreter code in the plugin would likely be much simpler.) The downside is there would be unique instances of python running for each plugin, increasing memory use, however it's been noted that such an implementation would likely lend itself to multiprocessing better than the (seemingly broken) python threading used with the sub-interpreters.
An alternate and relatively easy fix would be to simply have a single interpreter (e.g. the console and plugins would be completely co-mingled and no attempt would be made to switch context from the first one created for the console, nor would there be any code or namespace separation between plugins) but this could have consequences to existing code. However doing this in combination with an introspection-specific python loader might make sense.
Yet another alternative fix would be to correct whatever code is in gi.repository GObject that is triggering the issue, but this would leave the loader open to a recurrence with some other libraries. It seems unlikely that the python sub-interpreter issue will be corrected on the python side.
As a secondary suggestion, introspection does seem to open up the ability for the python code to handle much of the functionality python-loader does. Perhaps it would make sense to have a much simpler python-introspection loader that simply brings up unique python interpreter objects running the plugin code and little else, leaving UI, function plugin management, etc up to introspection calls from the Python side.
additional notes: Gnumeric 1.12.32, Python 2.7.11
Version: 1.12.x