1. 20 Nov, 2016 1 commit
  2. 21 Jun, 2016 1 commit
  3. 30 May, 2016 2 commits
  4. 08 May, 2016 1 commit
    • Kevin Haller's avatar
      libtracker-data: Support regular expressions for fn:replace(). · 60d0f54f
      Kevin Haller authored
      Extends the sqlite database by a new function (with the name
      SparqlReplace). The function makes use of the g_regex_replace() function
      of glib.
      
      To fullfill the XPath 2.0 standard some constraints must be checked for
      fn:replace(input, pattern, replacement, flags). The given pattern must
      not match a zero-length string. The given replacement string have to use
      $ followed by a number for backreferences. If the dollar sign shall be used
      "as is", it must be escaped (\$).
      
      For checking and interpreting the given replacement string of fn:replace()
      some regular expressions are needed. This expressions are precompiled and
      saved in the function_regex hashset of the TrackerDBInterface. The
      pre-compilation and initialization of the hashset are done by the
      prepare_database() method.
      
      The glib method g_regex_replace() make use of the backslash followed by a
      number to inidcate backreferences. So the dollar signs must be interpreted
      - the backslashes can be still used for this purpose.
      
      In the sparql expression class the corresponding section is adapted, so
      that the new SparqlReplace function is used for fn:replace(..) statements.
      
      https://bugzilla.gnome.org/show_bug.cgi?id=754961
      60d0f54f
  5. 01 Mar, 2016 1 commit
    • Carlos Garnacho's avatar
      Update to FTS5 · e6bd45c2
      Carlos Garnacho authored
      Our old stale copy of the FTS3/4 module is now deleted, replaced by
      a shinier FTS5 embedded module. If at configure time we detect that
      SQLite doesn't offer the FTS5 module, we will load our own, just as
      we used to do with FTS4.
      
      FTS5 brings a few differences in the ways it's meant to be extended,
      the tokenizer has been updated to cope with the differences. Also,
      FTS5 offers no offsets() builtin function, nor matchinfo() which we
      used to implement ranking. It offers though ways to implement
      additional functions, and builtin rank support which can be tweaked
      to achieve the same functional results than we did.
      
      Other than that, the ways to interact with the FTS virtual table
      are roughly similar to those in FTS4, insertions and deletions have
      been updated to do things the FTS5 way.
      
      Since it's not worth to bump the database format (data is reproducted
      from the journal, so we drop some embedded data such as
      nie:plainTextContent), the nco:hobby property has been modified to
      no longer be fulltext indexed, AFAIK there's no users ever setting/
      accessing that, and the FTS properties change will trigger the
      regeneration of the FTS view and virtual tables, resulting in a
      seamless update to FTS5.
      
      However, we don't leave completely unscathed from the fts3_tokenizer()
      change. Since the older FTS3/4 tokenizer is not registered, we can't
      just drop the older FTS table. So it is left dangling and never
      accessed again, in favor of the newer fts5 table. This is obviously
      not a problem when creating the database from scratch.
      
      In the way, a few bugs were found. per-property weights in ranking
      were being given in a scrambled way (although stable across database
      generations). And deletion of FTS properties (or entire rows) could
      result in the tokens not being fully removed from the FTS table,
      resulting in confused searches. These are now fixed.
      
      Impact to users of tracker should be none. All the FTS Sparql-to-SQL
      translation has been updated to just use FTS5 syntax and tables.
      e6bd45c2
  6. 02 Dec, 2015 3 commits
  7. 19 Oct, 2015 5 commits
  8. 18 Oct, 2015 9 commits
  9. 12 Mar, 2015 1 commit
  10. 21 Jan, 2014 2 commits
    • Aleksander Morgado's avatar
      libtracker-data: new 'tracker:unaccent' method · b3fb86ea
      Aleksander Morgado authored
      https://bugzilla.gnome.org/show_bug.cgi?id=722254
      
      This method allows removing combining diacritical marks (accents) from strings
      used in SPARQL queries. It expects a single argument, the string to be
      unaccented.
      
      Note that the output string will also be NFKD-normalized.
      
      Example:
      
      1) First, insert a new element which has accents in the nie:title. In the
      example we insert the word 'école' which in UTF-8 NFC looks like
      "0xC3 0xA9 0x63 0x6F 0x6C 0x65":
      
          $ tracker-sparql -u -q "
              INSERT { <abc> a         nie:InformationElement .
                       <abc> nie:title 'école' }"
      
      2) Second, get hexdump of querying nie:title, we should get the original string
      in UTF-8 and NFC normalization:
      
          $ tracker-sparql -q "
              SELECT ?title
              WHERE { <abc> nie:title ?title }" | hexdump
          0000000 6552 7573 746c 3a73 200a c320 63a9 6c6f
          0000010 0a65 000a
          0000013
      
      Or, without the hexdump...
      
          $ tracker-sparql -q "
              SELECT ?title
              WHERE { <abc> nie:title ?title }"
          Results:
            école
      
      3) Last, apply the unaccenting method. The expected string should look like
      "0×65 0×63 0x6F 0x6C 0×65" (i.e. without the combining diacritical mark):
      
          $ tracker-sparql -q "
              SELECT tracker:unaccent(?title)
              WHERE { <abc> nie:title ?title }" | hexdump
          0000000 6552 7573 746c 3a73 200a 6520 6f63 656c
          0000010 0a0a
          0000012
      
      Or, without the hexdump...
      
          $ tracker-sparql -q "
              SELECT tracker:unaccent(?title)
              WHERE { <abc> nie:title ?title }"
          Results:
            ecole
      b3fb86ea
    • Aleksander Morgado's avatar
      libtracker-data: new 'tracker:normalize' method · 8e00e181
      Aleksander Morgado authored
      https://bugzilla.gnome.org/show_bug.cgi?id=722254
      
      This method allows normalizing the strings used in SPARQL queries. It expects
      two arguments: First, the string to be normalized, and second, one of "nfc",
      "nfd", "nfkc" or "nfkd" specifying the type of normalization to apply to the
      string.
      
      Example:
      
      1) First, insert a new element which has accents in the nie:title. In the
      example we insert the word 'école' which in UTF-8 NFC looks like
      "0xC3 0xA9 0x63 0x6F 0x6C 0x65":
      
          $ tracker-sparql -u -q "
              INSERT { <abc> a         nie:InformationElement .
                       <abc> nie:title 'école' }"
      
      2) Second, get hexdump of querying nie:title, we should get the original string
      in UTF-8 and NFC normalization:
      
          $ tracker-sparql -q "
              SELECT ?title
              WHERE { <abc> nie:title ?title }" | hexdump
          0000000 6552 7573 746c 3a73 200a c320 63a9 6c6f
          0000010 0a65 000a
          0000013
      
      3) Third, now apply explicitly NFC normalization, we should get the same output:
      
          $ tracker-sparql -q "
              SELECT tracker:normalize(?title,'nfc')
              WHERE { <abc> nie:title ?title }" | hexdump
          0000000 6552 7573 746c 3a73 200a c320 63a9 6c6f
          0000010 0a65 000a
          0000013
      
      4) Last, apply a NFD decomposition, the expected decomposed string should look
      like "0×65 0xCC 0x81 0×63 0x6F 0x6C 0×65":
      
          $ tracker-sparql -q "
              SELECT tracker:normalize(?title,'nfkd')
              WHERE { <abc> nie:title ?title }" | hexdump
          0000000 6552 7573 746c 3a73 200a 6520 81cc 6f63
          0000010 656c 0a0a
          0000014
      8e00e181
  11. 08 Apr, 2013 1 commit
    • Carlos Garnacho's avatar
      Fix AS ?foo handling in FTS queries · 0ea1d5f9
      Carlos Garnacho authored
      FTS queries implicitly add an "AS var" clause to the translated
      SQL select query so values can be matched with the outer query that
      accesses FTS tables, which resulted in doubly added AS clauses if
      it was specified explicitly in SPARQL too.
      
      So, make sure the clause is just added once.
      0ea1d5f9
  12. 04 Feb, 2013 3 commits
    • Carlos Garnacho's avatar
      Handle FTS queries more generically · 1943dfb0
      Carlos Garnacho authored
      FTS functions and other SQL functions like COUNT() don't
      mix well, so handle it more generically by joining with the
      FTS table at the outmost level, so those functions are handled
      separately.
      1943dfb0
    • Carlos Garnacho's avatar
      libtracker-data: Implement fts:snippet() · 1f435c87
      Carlos Garnacho authored
      this function takes up to 3 optional parameters after the
      object, the first 2 parameters are the starting/ending text
      for the match (defaults to <b></b>), and the third one
      modifies the ellipsis text (defaults to <b>...</b>)
      1f435c87
    • Carlos Garnacho's avatar
      libtracker-data: Perform FTS matching in subquery · 3a3f7801
      Carlos Garnacho authored
      The outer query will call the functions that require a full row
      scan only with the elements returned by the inner query, so
      it would perform better with OFFSET and LIMIT
      3a3f7801
  13. 01 Dec, 2011 1 commit
  14. 23 Nov, 2011 1 commit
  15. 16 Nov, 2011 2 commits
  16. 11 Oct, 2011 1 commit
  17. 21 Jul, 2011 1 commit
  18. 20 Jul, 2011 1 commit
  19. 08 Apr, 2011 1 commit
  20. 28 Mar, 2011 2 commits