Move access token deletion into auth handler

Also move duplicated deactivation code into the auth handler.

I want to add some hooks when we deactivate an access token, so let's bring it
all in here so that there's somewhere to put it.