GitLab

Embarassingly, we were running the extractor tests, checking if the
output was what we expected, then ignoring errors due to a missing
'raise' statement. This is a regression from fc281ca8.

build: Remove old extract rules from build tree when configuration changes

See merge request GNOME/tracker-miners!71

functional-tests: Don't depend on GSettings schemas from the host

Closes #45

See merge request GNOME/tracker-miners!61

functional-tests: Show a better error when we can't extract a file

See merge request GNOME/tracker-miners!72

See merge request GNOME/tracker-miners!72

Update README

See merge request GNOME/tracker-miners!70

See the related Tracker core merge request for more information:
GNOME/tracker!92

API additions for the 2.3 series are now possible.

The functional-tests assume that all necessary dependencies are
installed. If you are missing one, you'll get an error.

Previously the error looked like this:

    ======================================================================
    ERROR: nctional-tests/test-extraction-data/images/xmp-loaded-1.expe (__main__.GenericExtractionTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/parallels/newrepos/tracker-miners/tests/functional-tests/common/utils/extractor.py", line 73, in get_tracker_extract_jsonld_output
        data = json.loads(output)
      File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "./400-extractor-metadata.py", line 85, in generic_test_extraction
        result = get_tracker_extract_jsonld_output(self.file_to_extract)
      File "/home/parallels/newrepos/tracker-miners/tests/functional-tests/common/utils/extractor.py", line 76, in get_tracker_extract_jsonld_output
        "Output was: %s" % (e, output))
    RuntimeError: tracker-extract did not return valid JSON data: Expecting value: line 1 column 1 (char 0)
    Output was:

Now it is a bit clearer:

    ======================================================================
    ERROR: nctional-tests/test-extraction-data/images/test-image-1.expe (__main__.GenericExtractionTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "./400-extractor-metadata.py", line 85, in generic_test_extraction
        result = get_tracker_extract_jsonld_output(self.file_to_extract)
      File "/home/sam/src/tracker-miners/tests/functional-tests/common/utils/extractor.py", line 77, in get_tracker_extract_jsonld_output
        "Error output was: %s" % error_output)
    RuntimeError: tracker-extract didn't return any data.
    Error output was: file:///home/sam/src/tracker-miners/tests/functional-tests/test-extraction-data/images/test-image-1.jpg: No metadata or extractor modules found to handle this file

For the functional-tests we run Tracker entirely from the source tree.
We need to maintain a directory of .rule files for tracker-extract to
use, and we must only have the .rule files that are enabled in the
current build configuration. This wasn't working properly because we
didn't remove old .rule files when reconfiguring, so disabling an
extractor at configure-time would sometimes not work as expected.

This is fixed by preparing the uninstalled rule files in a shell
script rather than from Meson, allowing us to delete the rule files
each time that we reconfigure.

Add testcase for cue sheet extractor

See merge request GNOME/tracker-miners!58

We recently discovered a long standing regression in this feature:
GNOME/tracker-miners#60

As the saying goes: "if it's not tested, it doesn't work"[1].

Depends on GNOME/tracker!85

1. https://yakking.branchable.com/posts/truism-4-if-it-is-not-tested/

If tracker-miners is configured with the `-Dtracker_core=subproject` option,
it should use GSettings schemas from the ./subprojects/tracker build
tree. We weren't doing that, which meant that if you ran `dnf remove
tracker` on your test image and then ran the test, you would see
failures like this in the functional-tests:

    (process:8450): GLib-GIO-ERROR **: 09:43:34.625: Settings schema 'org.freedesktop.Tracker.Store' is not installed

This fix is part of GNOME/tracker#24

tracker-extract-text: Correctly report errors back to the caller

Closes #56

See merge request GNOME/tracker-miners!62

The tracker_extract_get_metadata() function should return FALSE if
an error occurs reading the file. The tracker-extract-text module would
return TRUE in all cases.

This was causing intermittent failures in the functional-tests, as the
following situation could occur:

  1. file2.txt is created
  2. tracker-miner-fs sees file2.txt and processes it
  3. file2.txt is deleted (or moved into an unmonitored directory)
  4. tracker-miner-fs sees the deletion and removes its resource from
     the store
  5. tracker-extract sees the created notification for file2.txt and
     tries to process it
  6. the tracker_extract_get_metadata() incorrectly returns TRUE
     (success), so tracker-extract recreates the deleted resource

This problem was being detected in the functional tests and was
causing intermittent failures.

This hopefully fixes: GNOME/tracker-miners#56

This reverts commit 60145ddd.

Some projects that include Tracker haven't yet updated to Meson 0.50. This
change broke gnome-build-meta for example:
https://gitlab.gnome.org/GNOME/gnome-build-meta/-/jobs/286671

The install: arg isn't actually required as for configure_file() targets
Meson will infer whether an install is needed by the presence of the
install_dir: arg.

This reverts commit ad664049.

I accidentally pushed this to master without review, when I pushed the
trivial Meson change.

The merge request is here:
GNOME/tracker-miners!61

Meson helpfully warns us that this is needed:

    WARNING: Project specifies a minimum meson_version '>=0.47' but uses features which were added in newer versions:
     * 0.50.0: {'install arg in configure_file'}

If tracker-miners is configured with the `-Dtracker_core=subproject` option,
it should use GSettings schemas from the ./subprojects/tracker build
tree. We weren't doing that, which meant that if you ran `dnf remove
tracker` on your test image and then ran the test, you would see
failures like this in the functional-tests:

    (process:8450): GLib-GIO-ERROR **: 09:43:34.625: Settings schema 'org.freedesktop.Tracker.Store' is not installed

This fix is part of GNOME/tracker#24

tracker-extract: Use libz to process ps.gz files

Closes #61

See merge request GNOME/tracker-miners!60

Properly handle (atend) when parsing Postscript file. Also add a test
case.

Process ps.gz files using GZlibDecompressor instead of spawing gunzip. Because
tracker-extract runs the file parsers inside a seccomp sandbox, spawning an
external process during parsing is not a good idea because it leaves us with
little control on which syscalls are used.

Closes: GNOME/tracker-miners#61

Fix memory leaks/corruptions when running tests

Closes #59

See merge request GNOME/tracker-miners!54

tracker-extract: Relate contained audio tracks and container file both ways

Closes #60

See merge request GNOME/tracker-miners!57

This is the most generic class that still triggers GraphUpdated, we need
to delegate on the extractor module to figure out whether the file is a
music piece on its own or a container of music pieces.

In the latter case the role belongs on the contained elements, not on the
file resource itself, so avoid setting it here for all.

If there is a TOC, the nmm:* information belongs in the child nmm:MusicPiece
elements, not on the container file.

The migration to TrackerResource broke the handling of flac/cue, as
tracker_resource_print_*() do expect all related resources to be directly
referenced from the root resource, but the gstreamer extractor just added
backreferences from the nmm:MusicPieces to the parent nfo:FileDataObject.

Add a relation both ways through nie:hasLogicalPart/nie:isLogicalPartOf
so the graph is complete, and the tracks referenced from the root element.

Closes: GNOME/tracker-miners#60

Switch to README.md

See merge request GNOME/tracker-miners!56

Move tests to python3 and reformat with autopep8

See merge request GNOME/tracker-miners!55

Python has a recommended code style, known as PEP-8. The tests now follow this
style.

This commit was created using `autopep8` version 1.4.3, run with this command:

	find . -name "*.py" -exec autopep8 --ignore E501 --in-place {} \;

Closes: GNOME/tracker-miners#59

Closes: GNOME/tracker-miners#59

Closes: GNOME/tracker-miners#59