Clarify responsibility of ID mapping cleanup.

* The previous implementation had the destructor also release the mapping.

* That was problematic, because the internals of objects depended on
  the external mapping unnecessarily.

* This also fixes a potential double-free issue discovered by Jan
  Frederik Engels.