Use BM25 as FTS ranking algorithm

Merged Carlos Garnacho requested to merge carlosg/tracker:wip/carlosg/bm25 into master

This MR includes:

  • The introduction of BM25 as the FTS ranking algorithm, using the SQLite implementation. This replaces our own ranking function. A database version bump was required to make the change transparently in existing databases.
  • The dropping of stop words entirely, since BM25 weighs relative to the frequency of words. There is enough literature hinting that BM25 make stop word lists moot, or even prone to worse results.
  • Everything worthwhile salvaging from !592 (closed), the TrackerLanguage cleanups and the improved stemmer initialization based on user language.
  • Some additional tests to stress a bit results based on frequency of words, and former stop words.

This MR supersedes !592 (closed).

Edited by Carlos Garnacho

Merge request reports