1. 01 Aug, 2018 2 commits
  2. 31 Jul, 2018 2 commits
  3. 27 Jul, 2018 1 commit
  4. 26 Jul, 2018 5 commits
  5. 23 Jul, 2018 3 commits
  6. 22 Jul, 2018 2 commits
  7. 21 Jul, 2018 4 commits
    • Sam Thursfield's avatar
      functional-tests: Remove unused code from helpers.py · 2bcf53a2
      Sam Thursfield authored
      This functionality is alive and well in the tracker-miners.git fork of
      this code, and has seen lots of fixes. Rather than let this version get
      out of sync, let's remove it until something actually needs it.
      2bcf53a2
    • Carlos Garnacho's avatar
      da971357
    • Carlos Garnacho's avatar
      tests: Adapt TrackerFileNotifier tests to internal behavioral change · e4e7766e
      Carlos Garnacho authored
      Before commit 68381c1d, ensure_parents() would stop before the indexing
      root in the assumption that it was already notified upon, that commit made
      it so those folders are ensured to be notified too.
      
      This internal behavioral change is normally evened out by TrackerMinerFS,
      but shows at the tests for the internal TrackerFileNotifier object as
      there is nothing there to set the IRI for those files.
      e4e7766e
    • Carlos Garnacho's avatar
      libtracker-miner: Coalesce 2 CREATED events · 58cc6255
      Carlos Garnacho authored
      It is not even clear this is possible in real life cases, however the
      standalone tracker-file-notifier tests fall into this (due to IRI not being
      ever set, still this is an async op). In the case of 2 consecutive CREATED
      events on the same file, it would be dealt with in TrackerMinerFS as CREATED+
      UPDATED. This was already harmless, but we can do better and swallow one of
      such events.
      58cc6255
  8. 20 Jul, 2018 21 commits
    • Carlos Garnacho's avatar
      libtracker-direct: Use specific include · 9329e254
      Carlos Garnacho authored
      This is an internal header, so we don't need pointing to the general
      tracker-data.h header. Fixes build from scratch on meson as some files
      pulled from there are unexpectedly in the build directory.
      9329e254
    • Carlos Garnacho's avatar
      libtracker-direct: Avoid WAL checkpoint on readonly connections · d5c79e87
      Carlos Garnacho authored
      There is no point in doing WAL checkpoints from a readonly connection, so
      avoid it altogether.
      d5c79e87
    • Carlos Garnacho's avatar
      libtracker-data: Fetch shared connection on failure to create one · 85e58a52
      Carlos Garnacho authored
      On stress situations (tests/functional-tests/ipc/test-bus-query-cancellation
      is known to trigger this) there may be too many opened FDs all around (dbus
      related, fds passed for resultsets, DB connections, ...) to keep it all
      together.
      
      In those cases, we may attempt to create an extra interface to cater for
      the incoming request, but it will just fail underneath us. In those cases
      it is preferrable to fetch a connection from the pool and have it shared
      across threads than tripping into critical warnings and undefined behavior.
      85e58a52
    • Sam Thursfield's avatar
      add37bb9
    • Sam Thursfield's avatar
      dbce390a
    • Carlos Garnacho's avatar
      tracker-store: Use TrackerDirectConnection underneath · 693f89de
      Carlos Garnacho authored
      Instead of the lower level TrackerDataManager object directly.
      The only additional thing that tracker-store does is signal
      emission for writeback and GraphUpdated, the internal
      TrackerDataManager object is still accessed to implement those
      features. This makes libtracker-direct the only place where
      queries/updates are queued, performed and dispatched.
      
      There's other indirect benefit from this, update queue handling
      no longer needs to hit the main thread in order to schedule the
      next update. Besides the very unlikely thread contention situations
      described in previous commits, this should maximize throughput
      of the updates queue.
      693f89de
    • Carlos Garnacho's avatar
      libtracker-data: Move notify_transaction() down to libtracker-data · 09fae200
      Carlos Garnacho authored
      Now that we don't need upper layers' information to find out the
      right CommitType, push this call down together with the handling
      of the insert/delete/rollback callbacks.
      
      One particularity is that the commit callbacks will now be called
      from within the update thread, just like all the other callbacks.
      And this is fine, with all the preparation work from the previous
      commits.
      09fae200
    • Carlos Garnacho's avatar
      tracker-store: Push TrackerData hooks down to Tracker.Store · 1d59f036
      Carlos Garnacho authored
      Move this out of the Resources object, which is basically a view
      of the internal Store object. All event accounting and signaling
      is now performed by the Store object, to which the Resources
      DBus object connects to in order to implement GraphUpdated and
      Writeback signals.
      
      Only one handler of these events is possible at the moment, would
      be nice to consider doing something marginally better on the
      Steroids interface at some point, at least wrt the amount of data
      sent through the bus.
      
      Instead of trying to schedule the timeout across threads (the
      TrackerData hooks run in the thread performing the updates, and
      we want signaling done from the main/dbus thread), the code now
      just sets up a timeout on the main thread that keeps running as
      long as there are pending updates. When the task for the last
      batched update returns, it will be safe for the timeout to do
      signaling one last time and turn itself down, all of this happening
      in the main thread.
      1d59f036
    • Carlos Garnacho's avatar
      tracker-store: Protect ready writeback events with mutex · adfa177c
      Carlos Garnacho authored
      Just like with ready GraphUpdated events, this will be potentially accessed
      by both the thread performing updates, and the thread doing the DBus
      dispatching and signaling.
      
      Just like there, the chances of contention are rather low, since emission
      is checked just once per second by default.
      adfa177c
    • Carlos Garnacho's avatar
      38e362da
    • Carlos Garnacho's avatar
      tracker-store: Give ownership of writeback events on get_ready() · 10915fd2
      Carlos Garnacho authored
      Instead of doing get_ready() and then reset_ready(), just give
      ownership of the events hashtable on get_ready() while resetting
      the internal one.
      10915fd2
    • Carlos Garnacho's avatar
      tracker-store: Protect event batches with a mutex · 683035a5
      Carlos Garnacho authored
      While the pending data and event counter are only accessed by
      the updates thread, the ready events will be potentially accessed
      by both the updates and the dbus thread.
      
      That said, chances of locking will be minimal, since the
      get_pending() call only happens once a second (by default) or after
      the pending buffer grew big enough.
      683035a5
    • Carlos Garnacho's avatar
      tracker-store: Do immediate GraphUpdated emission checks on commit hook · 0d808351
      Carlos Garnacho authored
      Triggering those on insert/delete callbacks isn't right for two reasons:
      there could still be a rollback of the just notified data, and it's done
      from the wrong thread (the one performing updates instead of the main
      thread).
      
      To fix the first, only call this from the commit hook, we can only notify
      of data that was successfully stored. To fix the second, do the call on
      an idle that will ensure the main thread running the main loop and doing
      the DBus dispatching is the one handling the actual emission. At the
      moment the commit hook is actually executed on that same thread, but that
      won't stay as-is.
      0d808351
    • Carlos Garnacho's avatar
      libtracker-data: Move TrackerClass event maintenance to tracker-store · dffa19e7
      Carlos Garnacho authored
      This is solely used by tracker-store to keep the backlog of pending
      GraphUpdated events. This event tracking can move to tracker-store
      itself, implemented atop libtracker-data's insert/delete/commit/rollback
      callbacks.
      dffa19e7
    • Carlos Garnacho's avatar
      libtracker-data: Drop CommitType · 7b25b9a2
      Carlos Garnacho authored
      This basically exists to allow deferring GraphUpdated signals
      while there's pending batch updates. This is arguably wrong,
      the priorities should totally affect the order in which updates
      are processed, but for the sake of interactivity once the data
      is in the database it makes sense to let the users know ASAP.
      
      Now all commits shall set up a timer for GraphUpdated emission
      is none is set yet.
      7b25b9a2
    • Carlos Garnacho's avatar
      libtracker-direct: Add sync() call · 05383d97
      Carlos Garnacho authored
      This will flush all updates and trigger the WAL hook.
      05383d97
    • Carlos Garnacho's avatar
      libtracker-direct: Add internal function to set additional DBManager flags · cb3ef04c
      Carlos Garnacho authored
      This will be useful for tracker-store, and the flags that make sense there
      don't make as much sense to add to the public TrackerSparqlConnectionFlags
      set.
      cb3ef04c
    • Carlos Garnacho's avatar
      libtracker-data: Remove TRACKER_DB_MANAGER_REMOVE_CACHE flag · ed8e9870
      Carlos Garnacho authored
      It's unused and unhandled. As the TrackerDBManager type is internal
      API, just remove the flag and shuffle the numbers.
      ed8e9870
    • Carlos Garnacho's avatar
      0a4c9213
    • Carlos Garnacho's avatar
      libtracker-direct: Add internal TrackerDataManager getter · da5283d9
      Carlos Garnacho authored
      This will make internal users able to access all the gory details
      that TrackerDataManager has to offer. Will help deduplicate code
      in tracker-store that is essentially the same than this.
      da5283d9
    • Carlos Garnacho's avatar
      libtracker-direct: Rewrite in C · a5332e93
      Carlos Garnacho authored
      In today's chapter of gratuituous rewrites. The instigator of this
      rewrite is vala's bug https://bugzilla.gnome.org/show_bug.cgi?id=789249.
      One might argue that bugs are bugs and eventually get fixed, but there's
      two things that make me think it won't happen soon:
      - Vala behavior of possibly iterating the main loop from the async task
        until the task is complete is very much deliberate in order to support
        the Generator pattern without a main loop, as seen at:
        https://wiki.gnome.org/Projects/Vala/AsyncSamples#Generator_example
      - OTOH, glib docs specify that a GAsyncReadyCallback must happen at a
        later iteration of the main context, presumably to ensure the task
        is not finished before the async function dispatching the task returns.
        This is precisely what trips Vala into starting the same task again.
      
      I don't see either changing anytime soon, and in the mean time Tracker
      is largely affected by it, both in perceived bugs (All those nie:url
      constraint warnings out of the blue had a reason, this) and in real bugs
      (It's sometimes attempting to insert things twice, and it may even succeed
      if the query does not break any cardinality/unique constraints). This
      affects Tracker in too fundamental ways to just shrug it away, unlike the
      Vala code this C/glib code works just as it looks.
      
      Now about the code... It's a pretty boring rewrite, there's a thread pool
      to dispatch select queries, and a single exclusive thread to handle updates.
      The WAL hook can possibly use an additional thread to perform non-blocking
      writes. All very much alike the older code.
      
      Future commits will make tracker-store use this connection implementation,
      so there's only this piece of code handling SPARQL updates.
      a5332e93