Perf: Don't filter events for push

We know the users are joined and we can explicitly check for if they are
ignoring the user, so lets do that.