[BZ#725507] write a new JSON tokenizer
Submitted by Emmanuele Bassi (:ebassi) <<eba..@..com>>
Assigned to json-glib-maint@gnome.bugs
Link to original bug (#725507)
Description
JsonParser uses a fork of GScanner called JsonScanner for the tokenization of the input.
JsonScanner is generally okay-ish, but its implementation has quirks that make it harder to work with. the main stumbling block is error reporting: some errors in the tokenization (especially when it comes to Unicode parsing) result in assertions, and in other cases the flow bubbles up to the JsonParser, which can then correctly handle them as GError.
another issue with JsonScanner is the amount of code we string along from GScanner, and that we don't use.
we should drop our GScanner fork, and use an ad hoc tokenizer for JSON.