I will say that if you take Carter's actual remarks, he was not saying that ALL white people who disagree with Obama are racist. He did say that some of the MOST VITRIOLIC actions and remarks are likely rooted in racism. In that respect, I SOMEWHAT agree with him.
I think the vast majority of people who vehemently disagree with Obama are doing so for PARTISAN reasons first. I won't even say it's POLICY first, because much of the stuff they're supposedly disagreeing with Obama on (deficits, spending, etc.) they did not say 'BOO' when Bush was overseeing the largest deficit increases and transfer of wealth in history. I think it's more of a PARTISAN bias than a racial one. Then there are those who actually DO agree with policy.
HOWEVER, I have absolutely NO doubt that some of the disagreement with Obama has a racial edge to it, because of the very PERSONAL nature of some of these people's anger. I haven't been around a long time, but in my years I haven't quite seen it get THIS personal before. Not only that, but the people who have a vested interest in seeing the Obama administration fail (the talking heads, corporations, etc.) have NO problem using racially tinged language to stir up feeling of resentment in some of their constituents. Limbaugh's most recent RACIST dribble was about a bunch of black kids on a school bus who allegedly beat up a white kid. He proclaimed "this is OBAMA'S America!" That, along with the crap that airs daily on FOX "News", is VERY racially tinged.
Now you have people showing up at Presidential events with LOADED guns. And all of a sudden THAT'S OKAY! Who do you think they're trying to scare? What are they trying to prove? What FEAR are they trying to tap into? Would this have even been remotely tolerated with any other President?? What are these people doing waving signs showing MLK's face with the quote "He had a dream", and then Obama's face below it with the quote: "We got a nightmare?" Are you telling me that THAT'S not racial?? Come on! It's more than just racial though. There is a definite XENOPHOBIC aspect to all of this. The whole "birther" movement, the "Muslim" accusations, the growing anti-immigrant sentiment from the right, the talk of secession, people yelling "WE WANT OUR COUNTRY BACK!!" Back from WHAT exactly? Not to mention the uproar of the school address!
There's bound to be a "racial element" to a lot of this stuff. Let's not forget that 96% of African-Americans voted for Obama, but not only that. People forget that in the Deep South, particularly in the states of MS, AL, and LA, 90% or more of white people voted AGAINST Obama. It is also in the Deep South where the highest percentages of people believe that Obama was not born in the U.S.A. To deny that there is some racial element to all of this is being EXTREMELY naive.