Well, after watching it again, the first notable thing is that the tempo of your groove gets faster around the same time (both of you didn't notice) so that makes your signature fill seemed rushed (thats what I picked up first).
But when you come back to the tempo you correct your self to the faster tempo which is why it seems like its not that noticeable.