Commit b68cef02 authored by Sam Thursfield's avatar Sam Thursfield
Browse files

Rework the tracker-sandbox utility (again)

This fixes #111 and also
simplifies the instructions in the README for running Tracker from
the build tree.

The sandbox utility is now executed from the top directory by running:

    python3 -m utils.trackertestutils

Previously, due to importing stuff from trackertestutils. you needed to
run it from the utils/ directory or set PYTHONPATH appropriately.

Additionally, tracker-miners.git will ship a 'run-uninstalled' script
to provide convenient access to the sandbox script and allow running
Tracker from the build tree.
parent e072f03b
Pipeline #114963 passed with stage
in 3 minutes
......@@ -33,8 +33,8 @@ CI.
# Attaching a debugger to Tracker daemons
Tracker daemons are not started directly. Instead they are started by the D-Bus
daemon by request. When using tracker-sandbox or the functional-tests, it's
difficult to start the daemon manually under `gdb`.
daemon by request. When using the run-uninstalled script or the
functional-tests, it's difficult to start the daemon manually under `gdb`.
Instead, we recommend adding a 10 second timeout at the top of the daemon's
main() function. In Vala code, try this:
......@@ -51,13 +51,12 @@ the 'Pausing' message, run the `gdb attach``command in another terminal within
# Running Tracker daemons under Valgrind
The Tracker daemons are launched using D-Bus autolaunch. When running them from
the source tree using tracker-sandbox or the functional tests, the commandline
is controlled by the D-Bus files stored in `./tests/services`. Just
change the `Exec=` line to add Valgrind, like this:
the source tree using the run-uninstalled script or the functional-tests, the
commandline is controlled by the D-Bus files stored in
`./tests/services`. Just change the `Exec=` line to add Valgrind, like this:
Exec=/usr/bin/valgrind @abs_top_builddir@/src/tracker-store/tracker-store
By default the tracker-sandbox utility and the functional-tests will only
show output from Tracker code. For the functional-tests, set
TRACKER_TESTS_VERBOSE=1 to see output from Valgrind. For tracker-sandbox use
the `--debug-dbus` option.
By default the run-uninstalled script and the functional-tests will only show
output from Tracker code. For the functional-tests, set TRACKER_TESTS_VERBOSE=1
to see output from Valgrind. For tracker-sandbox use the `--debug-dbus` option.
......@@ -84,51 +84,51 @@ At this point you can run the Tracker test suite from the `build` directory:
meson test --print-errorlogs
## Developing with tracker-sandbox
## Using the run-uninstalled script
Tracker normally runs automatically, indexing content in the background so that
search results are available quickly when needed.
When developing and testing Tracker you will normally want it to run in the
foreground instead. The `tracker-sandbox` tool exists to help with this.
foreground instead. You also probably want to run it from a build tree, rather
than installing it somewhere everytime you make a change, and you certainly
should isolates your development version from the real Tracker database in your
home directory.
You can run the tool directly from the tracker.git source tree. Ensure you are
in the top of the tracker source tree and type this to see the --help output:
There is a tool to help with this, which is part of the 'trackertestutils'
Python module. You can run the tool using a helper script generated in the
tracker-miners.git build process named 'run-uninstalled'.
./utils/sandbox/ --help
Check the helper script is set up correctly by running this from your
tracker-miners.git build tree:
You should always pass the `--prefix` option, which should be the same as the
--prefix argument you passed to Meson. You may pass `--index` which to controls
where Tracker's database is kept. You may also want to pass `--debug` to see
detailed log output.
./run-uninstalled --help
The remaining arguments you pass to `tracker-sandbox` are shell commands which
get run inside the sandbox. Use the `--` sentinel to ensure that the
commandline arguments are processed correctly. First, let's see the status of
the Tracker daemons:
If run with no arguments, the script will start an interactive shell. Any
arguments after a `--` sentinel are treated as a command to run in a non-interactive
./utils/sandbox/ --prefix ~/opt/tracker -- tracker daemon status
So, let's see the status of the Tracker daemons. They should be all stopped
right now.
Let's try and index some content...
./run-uninstalled -- tracker daemon status
./utils/sandbox/ --prefix ~/opt/tracker -- tracker index ~/Music
Let's try and index some content. (Subtitute ~/Music for any other location
where you have interesting data). We need to explicitly tell the script to wait
for the miners to finish, or it will exit too soon. (This is a workaround for
[issue #122](
... let's see what files were found ...
./run-uninstalled --wait-for-miner=Files --wait-for-miner=Extract -- tracker index --file ~/Music
./utils/sandbox/ --prefix ~/opt/tracker -- tracker sparql --list-files
Let's see what files were found!
... run a full-text search ...
./run-uninstalled --prefix ~/opt/tracker -- tracker sparql -q 'SELECT ?url { ?u nie:url ?url }
./utils/sandbox/ --prefix ~/opt/tracker -- tracker search "bananas"
Or, you can try a full-text search ...
... or run a SPARQL query on the content:
./run-uninstalled -- tracker search "bananas"
./utils/sandbox/ --prefix ~/opt/tracker -- tracker sparql
--query "SELECT ?url { ?resource a nfo:FileDataObject ; nie:url ?url . }"
If you run `tracker-sandbox` without a command argument, it will open an
interactive shell inside the sandbox. From here you can use debugging tools
such as GDB.
There are many more things you can do with the script.
For more information about developing Tracker, look at and
......@@ -3,8 +3,7 @@ functional_ipc_test_c_args = [
tracker_sandbox = find_program(join_paths(source_root, 'utils', 'sandbox', ''))
sandbox_args = ['--dbus-config', test_dbus_config, '--debug-sandbox', '--index-tmpdir', '--']
sandbox_args = ['-m', 'trackertestutils', '--dbus-config', test_dbus_config, '--debug-sandbox', '--index-tmpdir', '--']
sandbox_env = environment()
......@@ -16,7 +15,7 @@ test_env.set('TRACKER_TEST_DOMAIN_ONTOLOGY_RULE', tracker_uninstalled_domain_rul
insert_or_replace_test = executable('test-insert-or-replace',
'test-insert-or-replace.vala', tracker_sparql_vapi,
dependencies: [tracker_common_dep, tracker_sparql_dep])
test('functional-ipc-insert-or-replace', tracker_sandbox,
test('functional-ipc-insert-or-replace', python,
args: sandbox_args + [insert_or_replace_test],
env: test_env)
......@@ -25,6 +24,6 @@ bus_query_cancellation_test = executable('test-bus-query-cancellation',
c_args: functional_ipc_test_c_args,
dependencies: [tracker_common_dep, tracker_sparql_dep])
test('functional-ipc-bus-query-cancellation', tracker_sandbox,
test('functional-ipc-bus-query-cancellation', python,
args: sandbox_args + [bus_query_cancellation_test],
env: test_env)
......@@ -24,6 +24,7 @@
import argparse
import collections
import configparser
import locale
import logging
......@@ -34,17 +35,19 @@ import signal
import subprocess
import sys
import tempfile
import time
from gi.repository import Gio
from gi.repository import GLib
import trackertestutils.dbusdaemon
from . import dbusdaemon
from . import helpers
from . import mainloop
# Script
script_name = 'tracker-sandbox'
script_version = '1.0'
script_about = "Tracker Sandbox developer tool."
default_prefix = '/usr'
default_index_location = '/tmp/tracker-sandbox'
store_pid = -1
......@@ -83,18 +86,11 @@ enable-writeback=false
log = logging.getLogger('sandbox')
dbuslog = logging.getLogger('dbus')
# Environment / Clean up
def environment_unset(dbus):
log.debug('Cleaning up files ...')
if dbus.get_session_file():
log.debug(' Removing DBus session file')
log.debug('Cleaning up processes ...')
......@@ -106,62 +102,55 @@ def environment_unset(dbus):
os.kill(store_pid, signal.SIGTERM)
def environment_set_and_add_path(env, prefix, suffix):
def environment_set_and_add_path(env, var, prefix, suffix):
new = os.path.join(prefix, suffix)
if env in os.environ:
existing = os.environ[env]
if var in env:
existing = env[var]
full = '%s:%s' % (new, existing)
full = new
os.environ[env] = full
env[var] = full
def create_sandbox(index_location, prefix=None, verbosity=0, dbus_config=None):
assert prefix is None or dbus_config is None
def environment_set(index_location, prefix, verbosity=0, dbus_config=None):
# Environment
index_location = os.path.abspath(index_location)
prefix = os.path.abspath(os.path.expanduser(prefix))
extra_env = {}
# Data
os.environ['XDG_DATA_HOME'] = '%s/data/' % index_location
os.environ['XDG_CONFIG_HOME'] = '%s/config/' % index_location
os.environ['XDG_CACHE_HOME'] = '%s/cache/' % index_location
os.environ['XDG_RUNTIME_DIR'] = '%s/run/' % index_location
extra_env['XDG_DATA_HOME'] = '%s/data/' % index_location
extra_env['XDG_CONFIG_HOME'] = '%s/config/' % index_location
extra_env['XDG_CACHE_HOME'] = '%s/cache/' % index_location
extra_env['XDG_RUNTIME_DIR'] = '%s/run/' % index_location
# Prefix - only set if non-standard
if prefix != default_prefix:
environment_set_and_add_path('PATH', prefix, 'bin')
environment_set_and_add_path('LD_LIBRARY_PATH', prefix, 'lib')
environment_set_and_add_path('XDG_DATA_DIRS', prefix, 'share')
if prefix and prefix != '/usr':
environment_set_and_add_path(extra_env, 'PATH', prefix, 'bin')
environment_set_and_add_path(extra_env, 'LD_LIBRARY_PATH', prefix, 'lib')
environment_set_and_add_path(extra_env, 'XDG_DATA_DIRS', prefix, 'share')
# Preferences
os.environ['TRACKER_USE_CONFIG_FILES'] = 'yes'
extra_env['TRACKER_USE_CONFIG_FILES'] = 'yes'
# if opts.debug:
# os.environ['TRACKER_USE_LOG_FILES'] = 'yes'
extra_env['G_MESSAGES_PREFIXED'] = 'all'
os.environ['G_MESSAGES_PREFIXED'] = 'all'
os.environ['TRACKER_VERBOSITY'] = str(verbosity)
extra_env['TRACKER_VERBOSITY'] = str(verbosity)
log.debug('Using prefix location "%s"' % prefix)
log.debug('Using index location "%s"' % index_location)
# Ensure directory exists
# DBus specific instance
dbus_session_file = os.path.join(
os.environ['XDG_RUNTIME_DIR'], 'dbus-session')
dbus = trackertestutils.dbusdaemon.DBusDaemon(dbus_session_file)
# Important, other subprocesses must use our new bus
os.environ['DBUS_SESSION_BUS_ADDRESS'] = dbus.get_address()
sandbox = helpers.TrackerDBusSandbox(dbus_config, extra_env=extra_env)
# So tests can detect if they are run under sandbox or not.
# Update our own environment, so when we launch a subprocess it has the
# same settings as the Tracker daemons.
os.environ['DBUS_SESSION_BUS_ADDRESS'] = sandbox.daemon.get_address()
os.environ['TRACKER_SANDBOX'] = '1'
return dbus
return sandbox
def config_set(content_locations_recursive=None, content_locations_single=None):
......@@ -230,28 +219,43 @@ def link_to_mime_data():
def argument_parser():
class expand_path(argparse.Action):
"""Expand user- and relative-paths in filenames."""
# From
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, os.path.abspath(os.path.expanduser(values)))
parser = argparse.ArgumentParser(description=script_about)
parser.add_argument('--version', action='store_true',
help="show version information")
parser.add_argument('--debug-dbus', action='store_true',
help="show stdout and stderr messages from every daemon "
"running on the sandbox session bus. By default we "
"only show messages logged by Tracker daemons.")
parser.add_argument('--debug-sandbox', action='store_true',
help="show debugging info from tracker-sandbox")
parser.add_argument('--dbus-config', metavar='FILE',
help="use a custom config file for the private D-Bus daemon")
parser.add_argument('-v', '--verbosity', default='0',
parser.add_argument('--dbus-config', metavar='FILE', action=expand_path,
help="use a custom D-Bus config file to locate the "
"Tracker daemons. This can be used to run Tracker "
"from a build tree of tracker-miners.git, by "
"using the generated file ./tests/test-bus.conf")
parser.add_argument('-p', '--prefix', metavar='DIR', action=expand_path,
help="run Tracker from the given install prefix. You "
"can run the system version of Tracker by "
"specifying --prefix=/usr")
parser.add_argument('-v', '--verbosity', default=None,
choices=['0', '1', '2', '3', 'errors', 'minimal', 'detailed', 'debug'],
help="show debugging info from Tracker processes")
parser.add_argument('-p', '--prefix', metavar='DIR', type=str, default=default_prefix,
help=f"run Tracker from the given install prefix (default={default_prefix})")
parser.add_argument('-i', '--index', metavar='DIR', type=str,
parser.add_argument('-i', '--index', metavar='DIR', action=expand_path,
default=default_index_location, dest='index_location',
help=f"directory to the index (default={default_index_location})")
parser.add_argument('--index-tmpdir', action='store_true',
help="create index in a temporary directory and "
"delete it on exit (useful for automated testing)")
parser.add_argument('--wait-for-miner', type=str, action='append',
help="wait for one or more daemons to start, and "
"return to idle for at least 1 second, before "
"exiting. Usually used with `tracker index` where "
"you should pass --wait-for-miner=Files and "
parser.add_argument('--debug-dbus', action='store_true',
help="show stdout and stderr messages from every daemon "
"running on the sandbox session bus. By default we "
"only show messages logged by Tracker daemons.")
parser.add_argument('--debug-sandbox', action='store_true',
help="show debugging info from tracker-sandbox")
parser.add_argument('command', type=str, nargs='*', help="Command to run inside the shell")
return parser
......@@ -264,43 +268,144 @@ def verbosity_as_int(verbosity):
'detailed': 2,
'debug': 3
return verbosity_map.get(verbosity, int(args.verbosity))
return verbosity_map.get(verbosity, int(verbosity))
def init_logging(debug_sandbox, debug_dbus):
SANDBOX_FORMAT = "sandbox: %(message)s"
DBUS_FORMAT = "|%(message)s"
SANDBOX_FORMAT = "%(name)s: %(message)s"
DBUS_FORMAT = "%(message)s"
if debug_sandbox:
sandbox_log_handler = logging.StreamHandler()
dbus_log_handler = logging.StreamHandler()
if debug_dbus:
root = logging.getLogger()
dbus_stderr = logging.getLogger('trackertestutils.dbusdaemon.stderr')
dbus_stdout = logging.getLogger('trackertestutils.dbusdaemon.stdout')
dbus_handler = logging.StreamHandler(stream=sys.stderr)
# Entry point/start
if __name__ == "__main__":
locale.setlocale(locale.LC_ALL, '')
if debug_dbus:
class MinerStatusWatch():
"""This class provides a way to block waiting for miners to finish.
This is needed because of a deficiency in `tracker index`, see:
def __init__(self, sandbox, miner_name):
self.dbus_name = 'org.freedesktop.Tracker1.Miner.' + miner_name
self.object_path = '/org/freedesktop/Tracker1/Miner/' + miner_name
self._sandbox = sandbox
# Stores a list of (time, status) pairs. This is used to determine
# if the miner has been idle continuously over a time peroid.
self._status_log = collections.deque()
def _log_status(self, time, status):
self._status_log.append((time, status))
if len(self._status_log) > 100:
def setup(self):
log.debug(f"Set up status watch on {self.dbus_name}")
self._proxy = Gio.DBusProxy.new_sync(
Gio.DBusProxyFlags.NONE, None,
self.dbus_name, self.object_path, 'org.freedesktop.Tracker1.Miner',
# FIXME: this doesn't appear to work, so we have to use polling.
#proxy.connect('g-signal', miner_signal_cb)
# This call will raise GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown
# if the miner name is invalid.
status = self._proxy.GetStatus()
self._log_status(time.time(), status)
log.debug(f"{self.dbus_name}: Current status: {status}")
def check_was_idle_for_time_period(self, period_seconds):
now = time.time()
status = self._proxy.GetStatus()
self._log_status(now, status)
log.debug(f"{self.dbus_name}: Current status: {status}")
cursor = len(self._status_log) - 1
previous_delta_from_now = 0
while True:
if cursor < 0 or self._status_log[cursor][1] != 'Idle':
if previous_delta_from_now >= period_seconds:
return True
return False
previous_delta_from_now = (now - self._status_log[cursor][0])
cursor -= 1
def wait_for_miners(watches):
# We wait 1 second after "Idle" status is seen before exiting, because the
# extractor goes to/from Idle frequently.
wait_for_idle_time = 1
while True:
status = [watch.check_was_idle_for_time_period(wait_for_idle_time) for watch in watches.values()]
if all(status):
log.debug(f"Waiting for idle.")
args = argument_parser().parse_args()
if args.version:
print(f"{script_name} {script_version}\n{script_about}\n")
def main():
locale.setlocale(locale.LC_ALL, '')
parser = argument_parser()
args = parser.parse_args()
init_logging(args.debug_sandbox, args.debug_dbus)
shell = os.environ.get('SHELL', '/bin/bash')
verbosity = verbosity_as_int(args.verbosity)
if args.prefix is None and args.dbus_config is None:
print("\nYou must specify either --dbus-config (to run Tracker from "
"a build tree) or --prefix (to run an installed Tracker).")
if args.prefix is not None and args.dbus_config is not None:
raise RuntimeError(
"You cannot specify --dbus-config and --prefix at the same time. "
"Note that running Tracker from the build tree implies "
if args.verbosity is None:
verbosity = verbosity_as_int(os.environ.get('TRACKER_VERBOSITY', 0))
verbosity = verbosity_as_int(args.verbosity)
if 'TRACKER_VERBOSITY' in os.environ:
if verbosity != int(os.environ['TRACKER_VERBOSITY']):
raise RuntimeError("Incompatible values for TRACKER_VERBOSITY "
"from environment and from --verbosity "
if args.command is None and args.wait_for_miner is not None:
raise RuntimeError("--wait-for-miner cannot be used when opening an "
"interactive shell.")
index_location = None
index_tmpdir = None
......@@ -313,22 +418,44 @@ if __name__ == "__main__":
index_location = args.index_location
# Set up environment variables and foo needed to get started.
dbus = environment_set(index_location, args.prefix, verbosity, dbus_config=args.dbus_config)
sandbox = create_sandbox(index_location, args.prefix, verbosity, dbus_config=args.dbus_config)
miner_watches = {}
for miner in (args.wait_for_miner or []):
watch = MinerStatusWatch(sandbox, miner)
miner_watches[miner] = watch
if args.command:
command = [shell, '-c', ' '.join(shlex.quote(c) for c in args.command)]
log.debug("Running: %s", command)
if len(miner_watches) > 0:
print('Starting shell... (type "exit" to finish)')
if args.dbus_config:
print(f"Using Tracker daemons from build tree with D-Bus config {args.dbus_config}")
print(f"Using Tracker daemons from prefix {args.prefix}")
print("Starting interactive Tracker sandbox shell... (type 'exit' to finish)")
if index_tmpdir:
shutil.rmtree(index_tmpdir, ignore_errors=True)
# Entry point/start
if __name__ == "__main__":
except RuntimeError as e:
sys.stderr.write(f"ERROR: {e}\n")
......@@ -35,17 +35,9 @@ class DaemonNotStartedError(Exception):
class DBusDaemon:
"""The private D-Bus instance that provides the sandbox's session bus.
"""The private D-Bus instance that provides the sandbox's session bus."""
We support reading and writing the session information to a file. This
means that if the user runs two sandbox instances on the same data
directory at the same time, they will share the same message bus.
def __init__(self, session_file=None):
self.session_file = session_file
self.existing_session = False
def __init__(self):
self.process = None
self.address = None
......@@ -56,19 +48,6 @@ class DBusDaemon:
self._threads = []
if session_file:
self.address, = self.read_session_file(session_file)
self.existing_session = True
except FileNotFoundError:
log.debug("No existing D-Bus session file was found.")
def get_session_file(self):
"""Returns the path to the session file if we created it, or None."""
if self.existing_session:
return None
return self.session_file
def get_address(self):
if self.address is None:
raise DaemonNotStartedError()
......@@ -79,65 +58,35 @@ class DBusDaemon:
raise DaemonNotStartedError()
return self._gdbus_connection
def read_session_file(session_file):
with open(session_file, 'r') as f:
content =
def start(self, config_file=None, env=None):
dbus_command = ['dbus-daemon', '--print-address=1', '--print-pid=1']
if config_file:
dbus_command += ['--config-file=' + config_file]
dbus_command += ['--session']
log.debug("Running: %s", dbus_command)
self.process = subprocess.Popen(
dbus_command, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
self._previous_sigterm_handler = signal.signal(
signal.SIGTERM, self._sigterm_handler)
address = content.splitlines()[0]
pid = int(content.splitlines()[1])
self.address = self.process.stdout.readline().strip().decode('ascii') = int(self.process.stdout.readline().strip().decode('ascii'))
except ValueError: