Carter quit on the franchise. His actions in leaving the franchise caused considerable harm both on the court and with the perception of the only non-American franchise. He got by on his natural abilities and never put the work in to truly achieve greatness - whether in Toronto or elsewhere.
I have a big problem with retiring a jersey and ignoring the aforementioned.
I could do a VC bobble head night or a VC fan appreciation night but not much else.
I don't think great popularity, dunk contest championships, and three trips to the playoffs with one series win deserves a jersey retirement.
I am biased because I don't like the guy but I really am trying to be objective in this post.