tracker-miner-fs: Always delete graph nie:InformationElement on create/update

See merge request !392

There are some situations where the file monitors cannot distinguish between
a file being created where none existed before, or a file newly created
replacing a previously existing file.

Treat all create/update events the same WRT trimming the previously
existing nie:InformationElements, in order to ensure these updates that
pass as creates also result in the file being reindexed by the
metadata extractor. This only applies to files that would have metadata
extracted.

While at it, simplify the SPARQL and move the code so that it is not
scattered across the function.

tracker-extract: Check field type on IPTC data embedded in TIFFs

Closes tracker#364

See merge request !391

The field containing IPTC metadata can come either as TIFF_LONG (32-bit
ints, endianness dependent), or TIFF_UNDEFINED (a byte string). We currently
handle everything as TIFF_LONG, which may cause memory corruptions if we
deal with a file where we must perform endianness swapping and receive
non-long-aligned data.

Ensure to handle only these 2 types as it is defined for the TIFFTAG_RICHTIFFIPTC
tag, and only perform byte swapping for non-byte data (i.e. TIFF_LONG).

This is more in line with what other (e.g. ImageMagick) do when dealing with
byte-swapping and IPTC data.

Fixes: https://gitlab.gnome.org/GNOME/tracker/-/issues/364

libtracker-miner: Properly detect fanotify failures

Closes #217

See merge request !390

commit 28142bdb makes
trackers file monitoring code fall back to GLib file monitors,
if fanotify doesn't work.

Unfortunately, fanotify_mark failures aren't propagated up,
so the fall back code doesn't get triggered in some cases.

This commit adds the plumbing to make the fall back code get
propagated.

Closes #217

cli: Fix inverted condition

See merge request !389

Fix thinko in boolean handling introduced in commit 71234ebf,
since the variable name started off as "escape_sequences" that
suggested the inverted value. We want to use escape sequences
when not piped to a file.

Improvements to "tracker3 status" CLI

See merge request !388

If the output is being dumped to a file, avoid the escape sequences
to change font color/appearance in output, so the file is readable
without "weird" characters in between.

Handle "tracker3 status" output being redirected to a file showing the
full error reports for all existing errors, instead of the clamped/paged
list meant for interactive navigation.

As every race condition is possible in filesystems, it is possible that a
file disappears under tracker-extract-3 feet after existence checks, but
before the extractor module opens the file for metadata extraction.

In that case we could generate error reports for files that do no longer
exist, and whose report won't be automatically deleted ever again.

Ensure to trim these stale files during "tracker3 status" output
generation, so there's something able to remove these from the filesystem
and they don't confuse users into thinking these are legit errors.

tracker-extract: Initialize cache for stable content URNs at startup

Closes tracker#355

See merge request !386

This cache initialization may incur in unexpected ioctls to probe
floppy/cdrom devices, and is nowadays left up to the first extractor
thread that queries a content identifier. This may result in seccomp
issues.

Perform this initialization on the main thread early during initialization
so the extractor threads find a populated cache when querying content
identifiers.

Fixes: tracker#355

libtracker-miner: Handle FAN_DELETE[_SELF] being emitted separately on dirs

See merge request !384

If there is a monitor on a dir and its parent, fanotify used to be able to
coalesce the event so FAN_DELETE and FAN_DELETE_SELF would be both set on
an uniquely sent event.

This seems no longer the case on more recent linux versions, where these
generate 2 separate events, first FAN_DELETE_SELF for the folder being
deleted, and then FAN_DELETE from the parent dir to notify of the parent
folder structure change. This double DELETE event emission is inconsistent
with TrackerMonitor behavior and causes test failures.

Handle these 2 events, by caching both and being able to merge them, but
flushing on the one that is expected last. Fixes tracker-file-notifier-test
on recent linux versions (locally, 5.17.0).

functional-tests: Initial test of removable media handling

Closes #191

See merge request !350

Sam Thursfield authored Aug 18, 2021 and Carlos Garnacho committed Mar 05, 2022

This is a basic test that removable device indexing works as expected.

Sam Thursfield authored Aug 17, 2021 and Carlos Garnacho committed Mar 05, 2022

This is our best option for testing Tracker's removable device handling
code, see: https://discourse.gnome.org/t/testing-removable-usb-devices/7297

You can test-drive the module by running this command in the build tree:

    env GIO_USE_VOLUME_MONITOR=mockvolumemonitor GIO_MODULE_DIR=$(pwd)/tests/functional-tests/mockvolumemonitor gio mount --monitor

The module is operated over D-Bus, to create a fake mount you can do
this:

     gdbus call --session --dest org.freedesktop.Tracker3.MockVolumeMonitor \
                --object-path /org/freedesktop/Tracker3/MockVolumeMonitor \
                --method org.freedesktop.Tracker3.MockVolumeMonitor.AddMount file:///tmp/mymount

OpenBSD: implement find_command

See merge request !366

tracker-extract: ignore subtrack titles for videos

Closes #202

See merge request !382

Ignacy Kuchciński authored Feb 22, 2022 and Carlos Garnacho committed Mar 04, 2022

Add a test that lets us check the resulting title of a video that
containes audio or subtitle tracks with their own title tags.

Ignacy Kuchciński authored Feb 22, 2022 and Carlos Garnacho committed Mar 04, 2022

Currently, the resulting nie:title includes titles from the individual
tracks due to both gstreamer not differentiating between
global/container tags and track tags until introducing
gst_discoverer_container_info_get_tags() as new API and tracker-extract
gstreamer backend looping through available tracks and collecting all
tags it could find.

As a result, in cases where there is no title tag in the container, but
there are some title tags in subtitle or audio tracks, they are included
in the nie:title and prevent the file name based fallback title in grilo
and cause the video to show up with nonsense titles in totem.

To fix this, replace gst_discoverer_info_get_tags() with the new API,
and ignore title tags from subtracks if the file is a video.

Fixes #202

ci: Update ci-templates dependency

See merge request !381

docs: Fix generation of manpages with asciidoc 10

See merge request !380

This contains fixes for Fedora >34 that we want here.

We should use asciidoc's `a2x` driver instead of running xsltproc
ourselves. asciidoc 10 ships its stylesheets in a private directory.
Copied from a similar update in Tracker by Jan Alexander Steffens.

Related: tracker#347

libtracker-miner: Make FANotify monitor a subclass of GLib monitor

Closes #212

See merge request !379

And fallback into GLib for the situations that FANotify cannot set
up a mark despite being detected at runtime.

Unfortunately, a growingly common situation for this are btrfs
subvolumes. These are currently problematic with FANotify due to
filesystem ID mismatches, thus EXDEV is currently returned. See
https://lore.kernel.org/all/CAOQ4uxjt2+BECu60aisZs_D6pgHK3VbBjju6hzmvG=Ls88UqUA@mail.gmail.com/
for a description of the problem.

Hopefully this situation will be resolved at some point, in the
mean time, transparently fallback to the old situation for users
of btrfs.

Closes: #212

These will be necessary to stack one subclass into the other, so
it can work as a fallback.

libtracker-miners-common: Use better stable filesystem identifiers

See merge request !374

libtracker-miner: Do not recurse into mount points unless configured

Closes tracker#85

See merge request !375

Avoid deprecated API in tracker-extract-libav

See merge request !378

cli: Fix --show-files in "tracker3 tag"

Closes #213

See merge request !377

This query was missing the split between file and content data. Since
nao:hasTag can be set on any of these, return either the URI, or the URN.

Closes: #213