1. 21 Oct, 2016 1 commit
  2. 19 Dec, 2014 1 commit
    • Jim Nelson's avatar
      Periodic database & attachments garbage collection: Bug #714134 · 23511dc3
      Jim Nelson authored
      See the ticket (comment #2) for more information on the thinking and
      strategy here, but in a nutshell this will remove from the Geary
      database all emails no longer accessible via any folder and not seen
      on the server in over 30 days.  It also deletes those messages
      attachment(s) and removes any empty directories in the attachment/
      directory to prevent clutter.  If enough messages are garbage
      collected, Geary will vacuum the database at startup, which will
      lower its disk footprint and reduce fragmentation, potentially
      increasing performance.
      23511dc3
  3. 16 Dec, 2014 1 commit
    • Jim Nelson's avatar
      Improved search experience: Bug #720361 · 533ab75e
      Jim Nelson authored
      This introduces a new full-text search algorithm that attempts to
      curb the effects of overstemming in the Porter Snowball stemmer.
      The FTS table will be regenerated with this update.
      
      The crux of this new algorithm is a configurable heuristic that
      reduces stemmed matching.  The configuration is not available via the
      UI (I suspect it will only confuse users) but can be changed by power
      users via GSettings.  More information is available at:
      
      https://wiki.gnome.org/Apps/Geary/FullTextSearchStrategy
      533ab75e
  4. 13 Aug, 2014 1 commit
  5. 12 Aug, 2014 1 commit
    • Jim Nelson's avatar
      Show attachments lacking a Content-Disposition: Bug #713830 · a9db6e8c
      Jim Nelson authored
      Attachments without Content-Disposition are now generated and shown
      in the client.  This requires a database upgrade as well as rescanning
      all messages to generate the previously missing attachments.
      
      In addition, this upgrade now stores the attachments' Content-ID in
      the database.  This makes it much easier for the client to associate
      a particular MIME section in the RFC822 message with an attachment in
      the database and on disk.
      a9db6e8c
  6. 12 Mar, 2014 1 commit
    • Charles Lindsay's avatar
      Fix a number of database hiccups · c3e52705
      Charles Lindsay authored
      1) Use docid instead of id in search table.
      
      We had previously included an 'id INTEGER PRIMARY KEY' column in the
      MessageSearchTable, assuming it would get the same rowid alias treatment
      as it does in non-FTS tables.  That assumption was wrong: it was being
      created as a FTS column.  This fixes it so we use docid everywhere.
      
      To fix the old incorrect docid values, we simply blow away the search
      table and let the natural search table population process, which now has
      the correct docid insertion code, fix the problem.
      
      This also removes the id column from the search table creation SQL, but
      this will only affect new users.  Upgraders will see an empty, vestigal
      id column in their search table.  Since SQLite doesn't easily let you
      remove columns, it's just easier to ignore the column than go through
      all the work to fix it.
      
      2) Do as many rowid lookups as possible in batches, instead of doing
      them individually in loops.  This speeds up working with large sets of
      email.
      
      3) Rejigger indices on the MessageLocationTable to make certain queries
      faster.  This creates a new covering index in particular for the email
      prefetcher, which previously had to sort using a temp table.  The new
      index should work in the general case too, as we should never be looking
      at ordering without folder_id (and since folder_id comes first, it works
      as an index on just folder_id, too).
      
      4) For bonus measure, log all slow queries (> 1s execution time) to
      debug output.
      
      Closes: bgo #725929
      c3e52705
  7. 04 Mar, 2014 1 commit
    • Charles Lindsay's avatar
      Don't multithread db upgrades · 85d1a971
      Charles Lindsay authored
      Turns out for long-running upgrades we were running them all in
      parallel, which thrashes the disk pretty hard.  This adds a simple mutex
      lock around each upgrade, so at least the computer is usable while it's
      going on.  A more robust solution would be to have a single-thread
      thread pool where we enqueue upgrades, but that's too much change this
      late in the release cycle.
      
      Also it turns out that the nullifying of the internaldate_time_t column
      before we repopulate it was very costly, and unnecessary.  So, this also
      should speed things up for upgrading users.
      
      Closes: bgo #724475
      85d1a971
  8. 21 Feb, 2014 1 commit
  9. 15 Feb, 2014 1 commit
    • Charles Lindsay's avatar
      Fix wrong internaldate_time_t column · 4dc5d28c
      Charles Lindsay authored
      We had a bug in our DateTime to time_t conversion logic where all
      time_ts would end up in the year 3800.  This fixes that, and repopulates
      the internaldate_time_t column with the new, correct time_t values.
      
      Closes: bgo #724335
      4dc5d28c
  10. 30 Jan, 2014 2 commits
    • Charles Lindsay's avatar
      Install version-017.sql · 6631ed91
      Charles Lindsay authored
      Closes: bgo #723281
      6631ed91
    • Charles Lindsay's avatar
      Add option to save sent mail · cce04b81
      Charles Lindsay authored
      This adds the ability for Geary to push sent mail up to the account's
      Sent Mail folder (if available).  There's an accompanying account option
      that defaults to on (meaning: push sent mail).
      
      The current implementation will leave messages in the Outbox (though
      they won't be sent again) if they fail to be pushed to Sent Mail.  This
      isn't the best solution, but it at least means you have a way of seeing
      the problem and hopefully copying the data elsewhere manually if you
      need to save it.
      
      Note that Geary might not always recognize an account's Sent Mail
      folder.  This is the case for any "Other" accounts that don't support
      the "special use" or "xlist" IMAP extensions.  In this case, Geary will
      either throw an error and leave messages in the Outbox, or erase the
      message from the Outbox when it's sent, depending on the value of the
      account's save sent mail option.  Better support for detecting the Sent
      Mail folder in every case is coming soon.
      
      Closes: bgo #713263
      cce04b81
  11. 02 Oct, 2013 1 commit
  12. 26 Sep, 2013 1 commit
  13. 25 Sep, 2013 2 commits
  14. 19 Jul, 2013 1 commit
  15. 25 Jun, 2013 1 commit
  16. 22 Jun, 2013 1 commit
  17. 10 Jun, 2013 1 commit
    • Charles Lindsay's avatar
      Limit search results from DB; fix #7049 · a7787462
      Charles Lindsay authored
      This caps the search results at 1000 emails, due to our unfortunate
      requirement of constructing an object for each search result.  A better
      way to proceed here would be to do the search only as items were loaded
      in the SearchFolder, but that gets complicated when the search phrase
      gets updated.
      a7787462
  18. 17 May, 2013 1 commit
    • Charles Lindsay's avatar
      Basic search table implementation; fix #6766 · a4f680b4
      Charles Lindsay authored
      This is a limited implementation, so please backup your database before
      running this search feature branch from now on as we may change things.
      
      It's using a Unicode Snowball stemming tokenizer available from
      https://github.com/littlesavage/sqlite3-unicodesn, also handily
      available in src/sqlite3-unicodesn in Geary.  If you want to look at the
      search tables on the command line, cd into the unicodesn source folder,
      run make and make install, then load sqlite3 like:
      
         sqlite3 -cmd '.load unicodesn.sqlext' /path/to/geary.db
      a4f680b4
  19. 15 May, 2013 1 commit
  20. 11 Mar, 2013 1 commit
  21. 26 Feb, 2013 1 commit
    • Jim Nelson's avatar
      Prefetch mail in background according to age of message: Closes #6365 · f54f8055
      Jim Nelson authored
      This introduces a background account synchronizer into Geary that
      prefetches email folder-by-folder to a user-configurable epoch.  The
      current default is 15 days.
      
      Additional work to make this user-visible is coming, in particular with
      
      The primary purpose for this feature is to allow "full" conversations
      (#4293), which needs more of the mailbox stored locally to do searching.
      f54f8055
  22. 25 Feb, 2013 1 commit
    • Charles Lindsay's avatar
      Properly decode Unicode folder names; fix #5217 · cee5a81d
      Charles Lindsay authored
      Previously, we were taking folder names as they came off the wire.
      Turns out IMAP specifies that folder names with 8 bit code points are
      encoded in a crazy scheme unique to IMAP.  Now, we properly decode that
      scheme to the correct UTF-8 folder names to be displayed to the user.
      
      There's also now a database upgrade path that converts all existing
      mailboxes to the decoded version, so your existing database should just
      keep working.
      cee5a81d
  23. 26 Aug, 2012 1 commit
  24. 16 Aug, 2012 1 commit
  25. 11 Jul, 2012 1 commit
    • Jim Nelson's avatar
      Remove SQLHeavy: Closes #5034 · 0e2a5334
      Jim Nelson authored
      It is done.
      
      Initial implementation of the new database subsystem
      
      These pieces represent the foundation for ticket #5034
      
      Expanded transactions, added VersionedDatabase
      
      Further expansions of the async code.
      
      Moved async pool logic into Database, where it realistically
      belongs.
      
      Further improvements.  Introduced geary-db-test.
      
      Added SQL create and update files for Geary.Db
      
      version-001 to version-003 are exact copies of the SQLHeavy scripts
      to ensure no slight changes when migrating.  version-004 upgrades
      the database to remove the ImapFolderPropertiesTable and
      ImapMessagePropertiesTable, now that the database code is pure
      IMAP.
      
      When we support other messaging systems (such as POP3), those
      subsystems will need to code their own database layers OR rely on
      the IMAP schema and simply ignore the IMAP-specific fields.
      
      ImapDB.Account fleshed out
      
      ImapDB.Folder is commented out, however.  Need to port next.
      
      ImapDB.Folder fleshed out
      
      MessageTable, MessageLocationTable, and AttachementTable are now
      handled inside ImapDB.Folder.
      
      chmod -x imap-db-database.vala
      
      OutboxEmailIdentifier/Properties -> SmtpOutboxEmailIdentifier/Properties
      
      Moved SmtpOutboxFolderRoot into its own source file
      
      SmtpOutboxFolder ported to new database code
      
      Move Engine implementations to ImapDB.
      
      Integration and cleanup of new database code with main source
      
      This commit performs the final integration steps to move Geary
      completely over to the new database model.  This also cleans out
      the old SQLHeavy-based code and fixes a handful of small bugs that
      were detected during basic test runs.
      
      Moved Outbox to ImapDB
      
      As the Outbox is tied to the database that ImapDB runs, move the
      Outbox code into that folder.
      
      Outbox fixes and better parameter checking
      
      Bumped Database thread pool count and made them exclusive
      
      My reasoning is that there may be a need for a lot of threads at
      once (when a big batch of commands comes in, especially at
      startup).  If performance looks ok, we might consider relaxing
      this later.
      0e2a5334
  26. 21 Jun, 2012 1 commit
  27. 13 Jun, 2012 1 commit
    • Eric Gregory's avatar
      Send outgoing messages via Outbox folder: Closes #4569 · 6963063f
      Eric Gregory authored
      Squashed commit of many patches that merged Eric's outbox patch
      as well as additional changes to upgrade the database rather than
      require it be wiped and some refactoring suggested by the Outbox
      implementation.  Also updated Outbox to be fully atomic via
      Transactions.
      6963063f
  28. 08 Jun, 2012 1 commit
  29. 24 Apr, 2012 1 commit
  30. 09 Jan, 2012 1 commit
  31. 17 Nov, 2011 1 commit
    • Jim Nelson's avatar
      Fetches only a small portion of the message for previews: Closes #4254, Closes #3799 · 18716ae6
      Jim Nelson authored
      Before we were fetching the entire message body (including attachments) to get the
      preview text.  This patch now offers the ability to fetch a small (128 byte) preview
      of the email.
      
      Also, since this ticket is about speeding up performance, I've introduced NonblockingBatch,
      which allows for multiple async operations to be executed in parallel easily.  I've added
      its use in a few places to speed up operations, including one that was causing the lag
      in #3799, which is why this commit closes that ticket.
      18716ae6
  32. 18 Oct, 2011 1 commit
    • Jim Nelson's avatar
      FETCH BODY[section]<partial> support. · db62ed5d
      Jim Nelson authored
      This adds support for retrieving partial header and body blocks straight from the email, and
      therefore support to pull the References header from a message (which, for some reason, IMAP
      doesn't support or include in the FETCH ENVELOPE command).  This is necessary for email conversations (#3808).
      
      This required a change to the database schema, meaning old databases will need to be blown
      away before starting.
      db62ed5d
  33. 15 Jul, 2011 2 commits
    • Jim Nelson's avatar
      Further work on detecting message removal when folder first selected: #3805 · 0533bc97
      Jim Nelson authored
      Needed to rethink storage strategies as I researched this and realized that a true scarce database -- where the database is sparsely populated both in columns and rows -- is not feasible due to IMAP's UID rules.  The strategy now means that the database rows are contiguous from the highest (newest) message to the oldest *requested by the user*.  This is a better situation than having to download the UID for the entire folder.
      0533bc97
    • Jim Nelson's avatar
      Implemented IMAP-specific folder and message properties tables: #3805 · 6b8951bf
      Jim Nelson authored
      This commit adds support for IMAP-specific properties, of which UIDValidity is crucial toward completing #3805.  The additional code is to integrate these tables into the SQLite Geary backend and to make sure this information is requested from the IMAP server.
      
      NOTE: This commit changes the database schema.  Old databases will need to be blown away before running.
      6b8951bf
  34. 24 Jun, 2011 1 commit
    • Jim Nelson's avatar
      Persist messages locally: #3742 · d179cb9b
      Jim Nelson authored
      This completes the heavy lifting of persisting messages locally.  The strategy is that the local database may be sparsely populated, both in the availability of messages in a folder and the fields of a message that is partially stored.  As data is pulled from the remote server it's always stored in the database.  Future requests will always go to the database first, preventing unnecessary network traffic.
      
      Also, this patch will detect when a message is stored in multiple folders on the server.  The database uses soft links from the folder to the message, so the message is stored only once in the database.  This technique relies heavily on the availability and validity of the Message-ID header, but we expect this to be reliable the vast majority of the time.
      d179cb9b
  35. 22 Jun, 2011 1 commit
  36. 16 Jun, 2011 1 commit
    • Jim Nelson's avatar
      Addition of MessageTable and MessageLocationTable toward fixing #3742. · 9221937e
      Jim Nelson authored
      Much of the API between the local and net stores had to be reworked for consistency as well as planning ahead for how messages will be retrieved and stored efficiently.  This work also attempts to keep in mind that other mail sources (POP, etc.) may be required in the future, and hopefully can be added without major rework.
      9221937e
  37. 11 Jun, 2011 1 commit
    • Jim Nelson's avatar
      First stab at implementing persistent storage of IMAP data: #3695. · 4b8ac568
      Jim Nelson authored
      This large diff represents a growth of the architecture to persist IMAP data as its downloaded.  When listing folders, a local database is consulted first to immediately feed to the caller.  In the background, network calls fetch the "real" list.  The two are collated for differences which are reported to the caller via signals, who are then responsible for updating the user interface appropriately.  No other synchronization work is represented in this diff.
      
      Note that this breaks functionality: when a folder is selected, no messages appear in the message list.  Fixing this requires more work, and this patch was already large enough.  It's ticketed here: #3741
      4b8ac568