Closes #5050, closes #5086; moves the settings migrator someplace sensible, causes the migrator to clean up old paths.