smartcard: Don't lose track of token name on card removal

Since landing the p11-kit port we stopped treating removed cards as login cards, and thus neglect to send removal events.

This commit fixes that.