-
Carlos Garnacho authored
Our old stale copy of the FTS3/4 module is now deleted, replaced by a shinier FTS5 embedded module. If at configure time we detect that SQLite doesn't offer the FTS5 module, we will load our own, just as we used to do with FTS4. FTS5 brings a few differences in the ways it's meant to be extended, the tokenizer has been updated to cope with the differences. Also, FTS5 offers no offsets() builtin function, nor matchinfo() which we used to implement ranking. It offers though ways to implement additional functions, and builtin rank support which can be tweaked to achieve the same functional results than we did. Other than that, the ways to interact with the FTS virtual table are roughly similar to those in FTS4, insertions and deletions have been updated to do things the FTS5 way. Since it's not worth to bump the database format (data is reproducted from the journal, so we drop some embedded data such as nie:plainTextContent), the nco:hobby property has been modified to no longer be fulltext indexed, AFAIK there's no users ever setting/ accessing that, and the FTS properties change will trigger the regeneration of the FTS view and virtual tables, resulting in a seamless update to FTS5. However, we don't leave completely unscathed from the fts3_tokenizer() change. Since the older FTS3/4 tokenizer is not registered, we can't just drop the older FTS table. So it is left dangling and never accessed again, in favor of the newer fts5 table. This is obviously not a problem when creating the database from scratch. In the way, a few bugs were found. per-property weights in ranking were being given in a scrambled way (although stable across database generations). And deletion of FTS properties (or entire rows) could result in the tokens not being fully removed from the FTS table, resulting in confused searches. These are now fixed. Impact to users of tracker should be none. All the FTS Sparql-to-SQL translation has been updated to just use FTS5 syntax and tables.
e6bd45c2