Ray J didn’t take too kindly to Fabolous talking smack about him on Twitter, so when he saw him out in Las Vegas last weekend … he clocked him!

Kim K’s sex tape partner re-told the tale on a radio show, explaining, “When I seen him, I was with Floyd [Mayweather] and 50 [Cent] and he tried to say, ‘Don’t touch me.’ And I touched that n***a!”

Ray J says he got kicked out of the Palms and detained by police, but according to cops in Vegas … he was not arrested.