1. 19 Jan, 2021 2 commits
    • Michael Gratton's avatar
      ImapDb.Database: Register new ICU-based tokeniser for FTS · 7e381982
      Michael Gratton authored and Michael Gratton's avatar Michael Gratton committed
      The SQLite tokeniser does not deal with scripts that do not use spaces
      for word breaking (CJK, Thai, etc), thus searching in those languages
      does not work well.
      
      This adds a custom SQLite tokeniser based on ICU that breaks words for
      all languages supported by that library, and uses NFKC_Casefold
      normalisation to handle normalisation, case folding, and dropping of
      ignorable characters.
      
      Fixes #121
      7e381982
    • Michael Gratton's avatar
      engine: Convert from SQLite FTS3/4 to FTS5 for full-text-search · 4fe0d921
      Michael Gratton authored and Michael Gratton's avatar Michael Gratton committed
      Add SQL migration that drops the old FTS4 MessageSearchTable table,
      re-create as a FTS5 table, clean up the column names a bit, and adds a
      flags column so unread/starred queries can be made fast.
      
      Define a SQLite FTS5 extension function `geary_matches()` to replace
      the FTS3 `offsets()` function which no longer exists in FTS5, based on
      Tracker's implementation.
      
      Update code to FTS5 conventions (docid -> rowid, etc), use new column
      names, populate and update the flags column as the email's flags
      change, and use new match function for getting matching tokens.
      
      Advanced searches are probably currently broken, these will be fixed
      by subsequent commits.
      4fe0d921
  2. 14 Oct, 2020 1 commit
  3. 03 Oct, 2020 1 commit