Referencing a parent node for each referenced node is overdone.  Instead,
we now reference the parent from build_level and unreference in free_level.
Each level keeps a single reference on its immediate parent.  This both
alleviates the performence problems and should perfectly serve the purpose.