When I look at the points status, it shows points removed due to user removed.
Ok, I follow that, but could we have a link to the question that we made the effort to answer either just for interest or to be able to re-use the efforts of our answer?
This is something beyond the means of the site and us moderators. It would have to be something which SE would need to implement. I have a feeling the reason why you cannot see it is because a user can be "removed" for one of two reasons.
Also, just because the user was removed, their contributions were not removed and neither were yours if you answered their question. If you look under some questions, I believe it says "user was removed" or it just says "userxxxx" greyed out. While their votes were removed from your Q/A, any contribution they've provided as far as Q/A is not (I'm pretty sure this is the deal; I'm pretty sure Nick or Bob will correct me if I'm wrong).
If you feel this is something you'd like to pursue, please ask this question over on Meta Stack Exchange. Meta is like this Meta, but for all of Stack Exchange (in case you were unfamiliar).
For your edification, you may want to read some of these posts ... one of them might cover it, though I didn't see the information about what you're specifically asking:
Like I said, none of them directly reflect the question you're asking, but there may be something of interest on the subject.