Reapply "users: Don't use another style class for the remove avatar button"

This reverts commit 02792f3c.

With my incredible luck, right after reverting this change, the system wide destructive styling was changed :)