• Carlos Garnacho's avatar
    Update to FTS5 · e6bd45c2
    Carlos Garnacho authored
    Our old stale copy of the FTS3/4 module is now deleted, replaced by
    a shinier FTS5 embedded module. If at configure time we detect that
    SQLite doesn't offer the FTS5 module, we will load our own, just as
    we used to do with FTS4.
    
    FTS5 brings a few differences in the ways it's meant to be extended,
    the tokenizer has been updated to cope with the differences. Also,
    FTS5 offers no offsets() builtin function, nor matchinfo() which we
    used to implement ranking. It offers though ways to implement
    additional functions, and builtin rank support which can be tweaked
    to achieve the same functional results than we did.
    
    Other than that, the ways to interact with the FTS virtual table
    are roughly similar to those in FTS4, insertions and deletions have
    been updated to do things the FTS5 way.
    
    Since it's not worth to bump the database format (data is reproducted
    from the journal, so we drop some embedded data such as
    nie:plainTextContent), the nco:hobby property has been modified to
    no longer be fulltext indexed, AFAIK there's no users ever setting/
    accessing that, and the FTS properties change will trigger the
    regeneration of the FTS view and virtual tables, resulting in a
    seamless update to FTS5.
    
    However, we don't leave completely unscathed from the fts3_tokenizer()
    change. Since the older FTS3/4 tokenizer is not registered, we can't
    just drop the older FTS table. So it is left dangling and never
    accessed again, in favor of the newer fts5 table. This is obviously
    not a problem when creating the database from scratch.
    
    In the way, a few bugs were found. per-property weights in ranking
    were being given in a scrambled way (although stable across database
    generations). And deletion of FTS properties (or entire rows) could
    result in the tokens not being fully removed from the FTS table,
    resulting in confused searches. These are now fixed.
    
    Impact to users of tracker should be none. All the FTS Sparql-to-SQL
    translation has been updated to just use FTS5 syntax and tables.
    e6bd45c2
sqlite-builtin-fts5.m4 1.06 KB