Improve performance for large address books
See also #50
Most of these are related to uselessly incrementing/decrementing reference counts or duplicating memory. There's still some performance bumps left, but these would require some deeper cuts.
Edited by Niels De Graef