tracker issueshttps://gitlab.gnome.org/GNOME/tracker/-/issues2022-08-03T13:56:23Zhttps://gitlab.gnome.org/GNOME/tracker/-/issues/239Allow content apps to choose where Tracker looks for data2022-08-03T13:56:23ZSam ThursfieldAllow content apps to choose where Tracker looks for data# Problem
GNOME expects users to keep media content in XDG content dirs (`~/Music`, `~/Pictures`, etc.) so Tracker will index them and core apps will show them. However, users may not want to store content like this (often due to storag...# Problem
GNOME expects users to keep media content in XDG content dirs (`~/Music`, `~/Pictures`, etc.) so Tracker will index them and core apps will show them. However, users may not want to store content like this (often due to storage constraints, e.g. media collection on external hard drive).
Users can customize Tracker index locations using GNOME control centre ([docs](https://help.gnome.org/users/gnome-help/unstable/files-search.html.en). However, some apps (Music, Photos) limit results to XDG content dirs while querying so this doesn't always get the desired behaviour.
Related discussions:
* https://discourse.gnome.org/t/music-photos-videos-import-from-folder/3868/
# Possible solutions
The general idea is ...
* apps allow user to choose folder(s) to index
* on startup, apps call org.freedesktop.Tracker3.Miner.Files.Control.IndexLocation for each chosen folder, to ensure Tracker processes it.
* access from inside Flatpak is controlled via a portal
When a Flatpak app bundles Tracker, there needs to be a way to expose chosen content to the app, so Tracker can index them and the app can access the content.
Related discussions:
* https://gitlab.gnome.org/GNOME/totem/-/merge_requests/154#note_882149Tracker 3.4https://gitlab.gnome.org/GNOME/tracker/-/issues/236Update documentation for Flatpak app developers2020-10-13T11:52:29ZSam ThursfieldUpdate documentation for Flatpak app developersWe don't have documentation for Flatpak app packagers who want to add correct policies to their apps.
We do provide an example at https://gitlab.gnome.org/GNOME/tracker/-/tree/master/examples/flatpak
At minimum, we should...
* remove...We don't have documentation for Flatpak app packagers who want to add correct policies to their apps.
We do provide an example at https://gitlab.gnome.org/GNOME/tracker/-/tree/master/examples/flatpak
At minimum, we should...
* remove existing content at https://gnome.pages.gitlab.gnome.org/tracker/docs/api-preview/libtracker-sparql-3/tracker-private-store.html
* add a link from the docs to https://gitlab.gnome.org/GNOME/tracker/-/blob/master/examples/flatpak/org.example.TrackerSandbox.json
* document the --domain-ontology flag in the manpages of the relevant daemons
* mention how to use this to start app-local instances of Tracker Minershttps://gitlab.gnome.org/GNOME/tracker/-/issues/234tracker_sparql_connection_bus_new() should return better errors on failure2020-08-15T15:24:55ZSam Thursfieldtracker_sparql_connection_bus_new() should return better errors on failureSince merging https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/180 I noticed this can happen (from https://gitlab.gnome.org/GNOME/gnome-music/-/merge_requests/732):
```
(org.gnome.Music.Devel:2): org.gnome.Music-DEBUG: 00:26:00.1...Since merging https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/180 I noticed this can happen (from https://gitlab.gnome.org/GNOME/gnome-music/-/merge_requests/732):
```
(org.gnome.Music.Devel:2): org.gnome.Music-DEBUG: 00:26:00.163: (trackerwrapper.py, _setup_host_miner_fs, 84) Connecting to session-wide Tracker indexer at org.freedesktop.Tracker3.Miner.Files
Traceback (most recent call last):
File "/app/bin/gnome-music", line 124, in <module>
sys.exit(main())
File "/app/bin/gnome-music", line 119, in main
return run_application()
File "/app/bin/gnome-music", line 107, in run_application
app = Application('org.gnome.Music.Devel')
File "/app/lib/python3.8/site-packages/gnomemusic/application.py", line 70, in __init__
self._coregrilo = CoreGrilo(self)
File "/app/lib/python3.8/site-packages/gnomemusic/coregrilo.py", line 77, in __init__
self._tracker_wrapper = TrackerWrapper(application.get_application_id())
File "/app/lib/python3.8/site-packages/gnomemusic/trackerwrapper.py", line 79, in __init__
self._setup_host_miner_fs()
File "/app/lib/python3.8/site-packages/gnomemusic/trackerwrapper.py", line 87, in _setup_host_miner_fs
self._miner_fs = Tracker.SparqlConnection.bus_new(
gi.repository.GLib.Error: g-dbus-error-quark: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name is not activatable (2)
```
This is a generic error message that doesn't give any idea of what went wrong. We should raise a Tracker specific error message that says what happened, i.e. "I tried to contact org.freedesktop.Tracker3.Miner.Files and org.freedesktop.portal and neither of those are available, so Tracker 3 Miners aren't available on this system"https://gitlab.gnome.org/GNOME/tracker/-/issues/233Automated testing for xdg-portal2020-07-18T12:32:12ZSam ThursfieldAutomated testing for xdg-portalThe following discussion from !180 should be addressed:
- [ ] @sthursfield started a [discussion](https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/180#note_716975): (+3 comments)
> How can we ensure ongoing testing of the p...The following discussion from !180 should be addressed:
- [ ] @sthursfield started a [discussion](https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/180#note_716975): (+3 comments)
> How can we ensure ongoing testing of the portal ? It might be too complicated to have a fully automated test, but at minimum we should have something in the HACKING file that says - if you make a change to the portal code, run steps X, Y, and Z to test your change.https://gitlab.gnome.org/GNOME/tracker/-/issues/221Make more file-related metadata available in content-specific graphs2021-07-05T09:38:19ZSam ThursfieldMake more file-related metadata available in content-specific graphsOur sandboxing model is that a photos app will only see the tracker:Pictures graph, a music app will only see the tracker:Audio graph, and so on.
GNOME Photos queries the filename, mtime and ctime of each photo in order to display this ...Our sandboxing model is that a photos app will only see the tracker:Pictures graph, a music app will only see the tracker:Audio graph, and so on.
GNOME Photos queries the filename, mtime and ctime of each photo in order to display this info in the UI. These are stored in the tracker:FileSystem graph as nfo:fileName, nfo:fileLastModified and nfo:fileCreated respectively.
Photos can request access to the tracker:FileSystem graph to fetch this data, but this means it can see the whole structure of the home directory as indexed by Tracker - this is exposing more data than the app needs to see.https://gitlab.gnome.org/GNOME/tracker/-/issues/220`tracker export` doesn't show data that's in the default graph2020-06-02T14:07:08ZSam Thursfield`tracker export` doesn't show data that's in the default graphI'm perhaps still confused about the default graph in Tracker 3.0.
I can insert data into the default graph like this:
```
INSERT DATA { <urn:example> a rdfs:Resource ; rdfs:label "Example" }
```
The data I inserted will not be return...I'm perhaps still confused about the default graph in Tracker 3.0.
I can insert data into the default graph like this:
```
INSERT DATA { <urn:example> a rdfs:Resource ; rdfs:label "Example" }
```
The data I inserted will not be returned by this query (as done by `tracker export` at the moment):
```
SELECT ?g ?u ?p ?v
{
GRAPH ?g {
?u ?p ?v
}
}
```
Do we have a way to say "query all graphs including the default graph" ? The SPARQL standard isn't clear about this, but reading https://stackoverflow.com/questions/53106635/sparql-querying-the-default-graph suggests at least one database allows `DEFAULT` to refer the default graph.
Alternately, perhaps we should prevent inserting data into the default graph?https://gitlab.gnome.org/GNOME/tracker/-/issues/219Various crashes with invalid ontologies.2024-03-18T09:46:52ZSam ThursfieldVarious crashes with invalid ontologies.Here are some examples of bad ontologies that cause crashes in libtracker-sparql.
```
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2...Here are some examples of bad ontologies that cause crashes in libtracker-sparql.
```
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix nautilus: <https://gitlab.gnome.org/GNOME/nautilus#> .
@prefix tracker: <http://tracker.api.gnome.org/ontology/v3/tracker#> .
nautilus: a tracker:Namespace, tracker:Ontology ;
tracker:prefix "nautilus" ;
tracker:lastModified "2020-05-01T10:00:00Z" .
nautilus:FileReference a rdfs:Class ;
rdfs:comment "..." ;
rdfs:subClassOf rdfs:Resource .
nautilus:starred a rdf:Property ;
rdfs:comment "Marks resources that are starred in Nautilus." ;
rdfs:domain Nautilus:File ; # this is invalid
rdfs:range xsd:boolean .
```
This triggers a crash here:
```
#0 0x00007ffff7e70b6a in __strlen_sse2 () at /lib64/libc.so.6
#1 0x00007ffff570f599 in tracker_data_ontology_process_statement
(manager=0x5555559a8440 [TrackerDataManager], subject=0x555555a8b850 "https://gitlab.gnome.org/GNOME/nautilus#starred", predicate=0
x555555a78e20 "http://www.w3.org/2000/01/rdf-schema#domain", object=0x0, in_update=0)
at ../subprojects/tracker/src/libtracker-data/tracker-data-manager.c:1997
#2 0x00007ffff570f723 in import_ontology_file (manager=0x5555559a8440 [TrackerDataManager], file=0x5555559b0820, in_update=0)
at ../subprojects/tracker/src/libtracker-data/tracker-data-manager.c:2040
#3 0x00007ffff5714523 in tracker_data_manager_initable_init (initable=0x5555559a8440, cancellable=0x0, error=0x7fffffffd478)
at ../subprojects/tracker/src/libtracker-data/tracker-data-manager.c:4201
#4 0x00007ffff57d86a5 in tracker_direct_connection_initable_init (initable=0x5555559a6790, cancellable=0x0, error=0x7fffffffd538)
at ../subprojects/tracker/src/libtracker-direct/tracker-direct.c:303
#5 0x00007ffff57bd05f in tracker_sparql_connection_new
(flags=TRACKER_SPARQL_CONNECTION_FLAGS_NONE, store=0x5555559b0f20, ontology=0x5555559b0f40, cancellable=0x0, error=0x7fffffffd788)
```
Another example of a malformed ontology:
```
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix nautilus: <https://gitlab.gnome.org/GNOME/nautilus#> .
@prefix tracker: <http://tracker.api.gnome.org/ontology/v3/tracker#> .
nautilus: a tracker:Namespace, tracker:Ontology ;
tracker:prefix "nautilus" ;
tracker:lastModified "2020-05-01T10:00:00Z" .
nautilus:FileReference a rdf:Class ; # should be rdfs:Class
rdfs:comment "..." ;
rdfs:subClassOf rdfs:Resource .
nautilus:starred a rdf:Property ;
rdfs:comment "Marks resources that are starred in Nautilus." ;
rdfs:domain nautilus:File ;
rdfs:range xsd:boolean .
```
This causes various warnings and criticals:
```
(process:21195): Tracker-CRITICAL **: 20:13:38.173: file:///opt/tracker3/share/nautilus/ontology/nautilus.ontology: Unknown class https://gitlab.gnome.org/GNOME/nautilus#FileReference
(process:21195): Tracker-CRITICAL **: 20:13:38.173: tracker_ontologies_get_class_by_uri: assertion 'class_uri != NULL' failed
(process:21195): Tracker-CRITICAL **: 20:13:38.173: file:///opt/tracker3/share/nautilus/ontology/nautilus.ontology: Unknown class (null)
(process:21195): Tracker-CRITICAL **: 20:13:38.218: Class 'http://www.w3.org/1999/02/22-rdf-syntax-ns#Class' not found in the ontology
(process:21195): Tracker-CRITICAL **: 20:13:38.218: Subject `https://gitlab.gnome.org/GNOME/nautilus#FileReference' is not in domain `rdfs:Resource' of property `rdfs:comment'
(process:21195): Tracker-CRITICAL **: 20:13:38.218: Subject `https://gitlab.gnome.org/GNOME/nautilus#FileReference' is not in domain `rdfs:Class' of property `rdfs:subClassOf'
fish: “env TRACKER_DEBUG=sparql ./naut…” terminated by signal SIGSEGV (Address boundary error)
```
Another ontology that triggers a crash:
```
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix nrl: <http://tracker.api.gnome.org/ontology/v3/nrl#> .
@prefix nautilus: <https://gitlab.gnome.org/GNOME/nautilus#> .
nautilus: a nrl:Namespace, nrl:Ontology ;
nrl:prefix "nautilus" ;
nrl:lastModified "2020-05-01T10:00:00Z" .
nautilus:File a rdfs:Class ;
rdfs:comment "Represents a file on disk by its URL. Equivalent to http://tracker.api.gnome.org/ontology/v3/nfo#FileDataObject" ;
rdfs:subClassOf rdfs:Resource .
nautilus:starred a rdf:Property ;
rdfs:comment "Marks files that are starred in Nautilus." ;
rdfs:domain nfo:FileDataObject ; # should be nautilus:File
rdfs:range xsd:boolean .
```
Triggers a critical here:
```
(org.gnome.Nautilus:295067): Tracker-CRITICAL **: 14:11:50.716: tracker_ontologies_get_class_by_uri: assertion 'class_uri != NULL' failed
```Tracker 3.2https://gitlab.gnome.org/GNOME/tracker/-/issues/217Error is ignored in SERVICE query if D-Bus name is invalid2020-06-13T13:38:48ZSam ThursfieldError is ignored in SERVICE query if D-Bus name is invalidThis following update causes the database engine to hang:
```
$ tracker3 sparql --database /tmp/tttt -u -q 'INSERT { ?u a rdfs:Resource } WHERE { SERVICE <dbus://invalid> { ?u a rdfs:Resource }}'
(tracker sparql:9803): GLib-GIO-CRITICA...This following update causes the database engine to hang:
```
$ tracker3 sparql --database /tmp/tttt -u -q 'INSERT { ?u a rdfs:Resource } WHERE { SERVICE <dbus://invalid> { ?u a rdfs:Resource }}'
(tracker sparql:9803): GLib-GIO-CRITICAL **: 12:58:56.664: g_dbus_message_new_method_call: assertion 'name == NULL || g_dbus_is_name (name)' failed
(tracker sparql:9803): GLib-GIO-CRITICAL **: 12:58:56.664: g_dbus_message_set_body: assertion 'G_IS_DBUS_MESSAGE (message)' failed
(tracker sparql:9803): GLib-GIO-CRITICAL **: 12:58:56.664: g_dbus_message_set_unix_fd_list: assertion 'G_IS_DBUS_MESSAGE (message)' failed
(tracker sparql:9803): GLib-GIO-CRITICAL **: 12:58:56.664: g_dbus_connection_send_message_with_reply: assertion 'G_IS_DBUS_MESSAGE (message)' failed
```https://gitlab.gnome.org/GNOME/tracker/-/issues/216tracker 2.3: UNION problems2020-08-25T23:55:16ZSam Thursfieldtracker 2.3: UNION problemsFeel free to close as WONTFIX since this is for 2.3, but I've hit some issues with `UNION` statements.
This query works fine:
```
$ tracker sparql -q "SELECT ?u ?p ?collection {
{
SELECT ?u (nie:isPartOf AS ?p) ?collection
...Feel free to close as WONTFIX since this is for 2.3, but I've hit some issues with `UNION` statements.
This query works fine:
```
$ tracker sparql -q "SELECT ?u ?p ?collection {
{
SELECT ?u (nie:isPartOf AS ?p) ?collection
{
?collection a nfo:DataContainer .
?u nie:isPartOf ?collection .
} LIMIT 1
}
}"
Results:
urn:uuid:68407b6f-6759-453b-9e98-d911ae76877b, http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf, urn:uuid:1bd24a0f-603a-4677-8484-ab4e4fcca90c
```
If I add a UNION statement, things go wrong.
```
tracker sparql -q "SELECT ?u ?p ?v {
{
SELECT ?u ?p ?v
{
?u a nfo:DataContainer .
?u ?p ?v .
} LIMIT 1
}
UNION
{
SELECT ?u (nie:isPartOf AS ?p) ?collection
{
?collection a nfo:DataContainer .
?u nie:isPartOf ?collection .
} LIMIT 1
}
}"
Results:
http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#image-category-screenshot, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, http://www.w3.org/2000/01/rdf-schema#Resource
urn:uuid:68407b6f-6759-453b-9e98-d911ae76877b, http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf, 100005
```
Notice how the value of `?collection` in the second query is reported as a resource ID, rather than the URN.
We can also use `AS` to cause a different issue:
```
tracker sparql -q "SELECT ?u ?p ?v {
{
SELECT ?u ?p ?v
{
?u a nfo:DataContainer .
?u ?p ?v .
} LIMIT 2
}
UNION
{
SELECT ?u (nie:isPartOf AS ?p) (?collection AS ?v)
{
?collection a nfo:DataContainer .
?u nie:isPartOf ?collection .
} LIMIT 1
}
}"
Results:
http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#image-category-screenshot, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, (null)
http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#image-category-screenshot, http://www.w3.org/1999/02/22-rdf-syntax-ns#type, (null)
urn:uuid:68407b6f-6759-453b-9e98-d911ae76877b, http://www.semanticdesktop.org/ontologies/2007/01/19/nie#isPartOf, urn:uuid:1bd24a0f-603a-4677-8484-ab4e4fcca90c
```
Notice how in the first query, the value is ow `(null)`.
I've not yet tested to see if this is fixed in master.https://gitlab.gnome.org/GNOME/tracker/-/issues/214Unbound variable in FILTER clause doesn't raise an error2020-05-27T10:48:34ZSam ThursfieldUnbound variable in FILTER clause doesn't raise an errorThe following query returns no results:
```
SELECT ?urn
WHERE {
{ SELECT ?urn
WHERE {
?urn a nmm:Photo ; nie:isStoredAs ?file
}
}
...The following query returns no results:
```
SELECT ?urn
WHERE {
{ SELECT ?urn
WHERE {
?urn a nmm:Photo ; nie:isStoredAs ?file
}
}
FILTER (fn:contains (nie:isStoredAs(?file), 'file://'))
}
```
It is in fact invalid, as the `?file` variable is unbound in the outer graph pattern. The same occurs if the FILTER is changed to `FILTER (fn:contains (?file, 'file://'))`.
If we use a variable in the filter that's not mentioned in the inner SELECT, we do get a `Could not run query, Use of undefined variable 'random'` error, so I think the issue is related to use of a nested query.https://gitlab.gnome.org/GNOME/tracker/-/issues/213Pick a unicode library implementation2022-03-07T18:12:06ZCarlos GarnachoPick a unicode library implementationWe have 2 backends: libicu and libunistring (complemented by enca).
Both have pros and cons:
- libicu
- pros
- good encoding detection, provides a confidence percentage
- cons
- Works with UTF16 internally
- it's huge
- ...We have 2 backends: libicu and libunistring (complemented by enca).
Both have pros and cons:
- libicu
- pros
- good encoding detection, provides a confidence percentage
- cons
- Works with UTF16 internally
- it's huge
- libunistring
- pros
- Works with UTF8 natively
- Smaller size
- cons
- no encoding detection. Enca is used for this, but has slightly worse detection, and doesn't hint the confidence on the result.
It seems we are down to trading dependency sizes for encoding detection. Given the latter is mostly useful for ID3v1 tags and other old places where encodings are underdefined, it'd seem the odds are in favor of libunistring.https://gitlab.gnome.org/GNOME/tracker/-/issues/209Document how apps should use the tracker-sandbox tool in their testsuites2023-08-19T14:57:29ZSam ThursfieldDocument how apps should use the tracker-sandbox tool in their testsuitesCurrently the only documentation seems to be this commit message: https://gitlab.gnome.org/GNOME/tracker/commit/1a787fa4fd667bb7727995fd26e35b2214f3f1f7
We should add something to HACKING.mdCurrently the only documentation seems to be this commit message: https://gitlab.gnome.org/GNOME/tracker/commit/1a787fa4fd667bb7727995fd26e35b2214f3f1f7
We should add something to HACKING.mdhttps://gitlab.gnome.org/GNOME/tracker/-/issues/207build: Don't hardcode path to asciidoc stylesheets2020-05-14T11:26:10ZSam Thursfieldbuild: Don't hardcode path to asciidoc stylesheetsSee the discussion at https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/209#note_799266See the discussion at https://gitlab.gnome.org/GNOME/tracker/-/merge_requests/209#note_799266https://gitlab.gnome.org/GNOME/tracker/-/issues/193Decompose SERVICE queries to pass in bindings2020-03-31T13:09:13ZSam ThursfieldDecompose SERVICE queries to pass in bindingsI was playing with remote queries using dbpedia, and my goal was to make this query work:
SELECT ?artist ?artist_name ?dbpedia_artist ?dbpedia_name {
?artist a nmm:Artist ; dc:title ?artist_name .
SERVICE <http://dbp...I was playing with remote queries using dbpedia, and my goal was to make this query work:
SELECT ?artist ?artist_name ?dbpedia_artist ?dbpedia_name {
?artist a nmm:Artist ; dc:title ?artist_name .
SERVICE <http://dbpedia.org/sparql> {
{ ?dbpedia_artist a dbo:Band } UNION { ?dbpedia_artist a dbo:MusicalArtist } .
?dbpedia_artist foaf:name STRLANG(?artist_name, "en") .
}
}
The idea is to return the dbpedia artist resource for each artist in the local music library. However, it fails because the ?artist_name binding from the outer query is not passed to the inner query. Instead, Tracker runs the inner query first -- which tries to list *every musical artist in the world*.
Here's a simpler example which also fails for the same reason:
SELECT ?artist { BIND ("Blur"@en AS ?artist_name) . SERVICE <http://dbpedia.org/sparql> { ?artist a dbo:Artist ; foaf:name ?artist_name . } }
The SPARQL 1.1 Federated Query standard mentions this problem in a section titled [interplay of SERVICE and VALUES](https://www.w3.org/TR/2013/REC-sparql11-federated-query-20130321/#values).
> An implementation of a query planner for federated queries may decide to decompose the query into two queries instead, where first the bindings from the local default graph are evaluated ... Next, dispatch to the remote endpoint <http://example.org/sparql> a constrained query with the solutions.
The suggestion is that the query optimiser separates the outer and inner parts of the query and runs them separately, injecting the bindings into the remote query using VALUES. My example query above would be broken down as follows:
1. `SELECT ?artist_name { BIND ("Blur"@en AS ?artist_name) }` -> save `?artist_name`
2. `SELECT ?artist ?artist_name { SERVICE <http://dbpedia.org/sparql> { ?artist a dbo:Band ; foaf:name ?artist_name . } VALUES (?artist_name) { ("Blur"@en) } } }`
The latter query doesn't work right now in Tracker, perhaps because it doesn't pass the VALUES part of the query to the remote service.
The problem is most visible with remote SPARQL connections but this optimization might be needed to effectively query DBus SPARQL connections too.https://gitlab.gnome.org/GNOME/tracker/-/issues/152Modernize coding style and document it in the Git tree2021-05-25T05:22:09ZSam ThursfieldModernize coding style and document it in the Git treeWe should document our coding style in the Git tree (currently it's at https://wiki.gnome.org/Projects/Tracker/Documentation/CodingStyle)
We may also want to modernize some things:
* do we recommend or disallow `g_auto` and `g_autopt...We should document our coding style in the Git tree (currently it's at https://wiki.gnome.org/Projects/Tracker/Documentation/CodingStyle)
We may also want to modernize some things:
* do we recommend or disallow `g_auto` and `g_autoptr`? (We use it in a couple of places already; it means we are GCC/Clang only)
* do we use C99 standard or C89 standard? (it means we drop support for MSVC)
* We currently avoid mixing code and variable declarations, we could start to do this if we allowed C99
My opinion is that nobody has ever expressed interest in building Tracker on Windows, so it's fine to require GCC/Clang and we should embrace autoptrs and C99.https://gitlab.gnome.org/GNOME/tracker/-/issues/147Unexpected results with non aggregate expressions2020-08-15T15:27:37ZCarlos GarnachoUnexpected results with non aggregate expressionsThe behavior of Tracker with a query like:
```
SELECT ?c ?u {
?u nfo:belongsToContainer ?c
}
GROUP BY ?c
```
is non-canon. We essentially do the equivalent to:
```
SELECT ?c (SAMPLE(?u) AS ?sample) {
?u nfo:belongsToContainer ?c
}...The behavior of Tracker with a query like:
```
SELECT ?c ?u {
?u nfo:belongsToContainer ?c
}
GROUP BY ?c
```
is non-canon. We essentially do the equivalent to:
```
SELECT ?c (SAMPLE(?u) AS ?sample) {
?u nfo:belongsToContainer ?c
}
GROUP BY ?c
```
According to https://www.w3.org/TR/sparql11-query/#aggregateRestrictions, ```In a query level which uses aggregates, only expressions consisting of aggregates and constants may be projected, (besides) [...] GROUP BY [...] simple expressions consisting of just a variable.```
And [wikidata](https://query.wikidata.org/#%23Cats%0ASELECT%20%3Fu%20%3Fc%0AWHERE%20%0A%7B%0A%20%20%3Fu%20rdfs%3AsubClassOf%20%3Fc.%0A%7D%0Agroup%20by%20%3Fc) seems to agree, returning a BadAggregate error.
I don't think this behavior was consciously intended, so it might make sense to adopt sparql1.1 behavior, now that we implement `SAMPLE` for it...https://gitlab.gnome.org/GNOME/tracker/-/issues/146Decide about syntax extensions2020-08-15T15:27:57ZCarlos GarnachoDecide about syntax extensionsWith 3.0 on the horizon, it would be a good moment to ponder about the syntax extensions Tracker gained in the past. Those may be found in https://gitlab.gnome.org/GNOME/tracker/blob/master/src/libtracker-data/tracker-sparql.c by searchi...With 3.0 on the horizon, it would be a good moment to ponder about the syntax extensions Tracker gained in the past. Those may be found in https://gitlab.gnome.org/GNOME/tracker/blob/master/src/libtracker-data/tracker-sparql.c by searching for "TRACKER EXTENSION".
I think those can be classified in 4 groups:
1. Slightly different syntax. Most usually gained as the previous sparql parser was developed sometime between sparql1.0 and 1.1, so those represent either old syntax, or some from unfinished working documents. IMHO the main benefit of dropping those is readability, examples:
- Tracker makes ';' separator between inserts/deletes/updates optional. Given delete/insert are sometimes part of a single statement, this makes the resulting parse expression tree less unequivocal too.
- Tracker makes parentheses in eg. `SELECT (?u as ?a) (?x as ?b) ...` and `ORDER BY (?a - ?b) (?c * ?d)` optional.
2. `INSERT OR REPLACE` as a whole. This is the sole user of `tracker_data_update_statement*`, accounting for roughly a third of the code in tracker-data-update.c. That on itself is not bad, but IMO `DELETE {} INSERT {} WHERE {}` is a more versatile replacement.
3. INSERT/DELETE allow `SILENT` keyword. The circumstances in which they can be rightfully ignored are pretty narrow, and it's always possible to pass a NULL error.
4. Expressions can be subqueries. This multiplies the places where we can step into a subquery, it has some implicit semantics that are probably easy to foresee (eg. the subselect must return a single column, for it to fit as a expression), but has other that are not as clear (eg. propagation of variables' values from the parent query).
I'm not saying they all should go, but we should see which do bring something of value, and ponder what to do with the others.https://gitlab.gnome.org/GNOME/tracker/-/issues/140Warnings during functional-02-sparql-bugs2019-09-14T11:44:54ZSam ThursfieldWarnings during functional-02-sparql-bugsLooking at [test logs](https://gnome.pages.gitlab.gnome.org/-/tracker/-/jobs/432290/artifacts/build/meson-logs/testlog.txt) I see various warnings and criticals in one of the tests. The test passes, but perhaps these need fixing?
```
2...Looking at [test logs](https://gnome.pages.gitlab.gnome.org/-/tracker/-/jobs/432290/artifacts/build/meson-logs/testlog.txt) I see various warnings and criticals in one of the tests. The test passes, but perhaps these need fixing?
```
2/42 functional-02-sparql-bugs OK 6.57 s
--- command ---
TRACKER_LANGUAGE_STOP_WORDS_DIR='/builds/GNOME/tracker/src/libtracker-common/stop-words' TRACKER_FUNCTIONAL_TEST_CONFIG='/builds/GNOME/tracker/build/tests/functional-tests/configuration.json' PYTHONPATH='/builds/GNOME/tracker/tests/functional-tests/../../utils' TRACKER_DB_ONTOLOGIES_DIR='/builds/GNOME/tracker/src/ontologies/nepomuk' TRACKER_TEST_DOMAIN_ONTOLOGY_RULE='/builds/GNOME/tracker/src/tracker-store/default.rule' /usr/bin/python3 02-sparql-bugs.py
--- stderr ---
test_01_NB217566_union_exists_filter (__main__.TrackerStoreSparqlBugsTests) ... ok
test_02_NB217636_delete_statements (__main__.TrackerStoreSparqlBugsTests) ... (tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gtype.c:4265: type id '0' is invalid
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: can't peek value table for type '<invalid>' which is not currently referenced
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gvalue.c:184: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
(tracker-store:1334): GLib-GObject-CRITICAL **: 10:43:03.518: g_value_copy: assertion 'G_IS_VALUE (src_value)' failed
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gtype.c:4265: type id '0' is invalid
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: can't peek value table for type '<invalid>' which is not currently referenced
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gvalue.c:184: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
(tracker-store:1334): GLib-GObject-CRITICAL **: 10:43:03.518: g_value_copy: assertion 'G_IS_VALUE (src_value)' failed
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gtype.c:4265: type id '0' is invalid
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: can't peek value table for type '<invalid>' which is not currently referenced
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gvalue.c:184: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
(tracker-store:1334): GLib-GObject-CRITICAL **: 10:43:03.518: g_value_copy: assertion 'G_IS_VALUE (src_value)' failed
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gtype.c:4265: type id '0' is invalid
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: can't peek value table for type '<invalid>' which is not currently referenced
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gvalue.c:184: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
(tracker-store:1334): GLib-GObject-CRITICAL **: 10:43:03.518: g_value_copy: assertion 'G_IS_VALUE (src_value)' failed
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gtype.c:4265: type id '0' is invalid
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: can't peek value table for type '<invalid>' which is not currently referenced
(tracker-store:1334): GLib-GObject-WARNING **: 10:43:03.518: ../gobject/gvalue.c:184: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
(tracker-store:1334): GLib-GObject-CRITICAL **: 10:43:03.518: g_value_copy: assertion 'G_IS_VALUE (src_value)' failed
(tracker-store:1334): Tracker-WARNING **: 10:43:03.519: Unknown type for binding: (null)
(tracker-store:1334): Tracker-WARNING **: 10:43:03.519: Unknown type for binding: (null)
(tracker-store:1334): Tracker-WARNING **: 10:43:03.519: Unknown type for binding: (null)
(tracker-store:1334): Tracker-WARNING **: 10:43:03.519: Unknown type for binding: (null)
(tracker-store:1334): Tracker-WARNING **: 10:43:03.519: Unknown type for binding: (null)
ok
test_03_NB222645_non_existing_class_resource (__main__.TrackerStoreSparqlBugsTests) ... ok
test_04_NB224760_too_long_filter (__main__.TrackerStoreSparqlBugsTests) ... ok
test_05_NB281201_insert_replace_and_superproperties (__main__.TrackerStoreSparqlBugsTests) ... ok
----------------------------------------------------------------------
Ran 5 tests in 6.304s
OK
```https://gitlab.gnome.org/GNOME/tracker/-/issues/129Cleanup indirect resources2019-08-28T08:34:59ZMarinus Schraalmschraal@gnome.orgCleanup indirect resourcesIndirect resources (like `nmm:MusicAlbum`, `nmm:Artist`) stay around forever, even when unused. This means it also does not signal the resource is removed.
In Music at the moment on every changed signal we have to query to check if any ...Indirect resources (like `nmm:MusicAlbum`, `nmm:Artist`) stay around forever, even when unused. This means it also does not signal the resource is removed.
In Music at the moment on every changed signal we have to query to check if any of the albums has no songs or artists has no related albums/songs. This is suboptimal at best.https://gitlab.gnome.org/GNOME/tracker/-/issues/127TrackerResource may contain circular references to itself -- add `tracker_res...2019-08-27T11:40:58ZSam ThursfieldTrackerResource may contain circular references to itself -- add `tracker_resource_destroy()`A tree of TrackerResource instances may contain reference loops. Simply unreffing the top resource isn't enough to free all of the resources.
So, we need a `tracker_resource_destroy()` function that applications would call to completely...A tree of TrackerResource instances may contain reference loops. Simply unreffing the top resource isn't enough to free all of the resources.
So, we need a `tracker_resource_destroy()` function that applications would call to completely dispose of a tree of resources, even if they contain reference cycles.