My first was a 4-string mutt that I bought. I only kept that one for about 5 months, because I broke the truss rod, by over-tightening it, by then though, my 5-string was already on the way.
I don't think I've had a hard time switching between the different ones, because I don't know the notes on the strings anyway. I play using intervals, so it doesn't matter how many strings I have. It does feel a little constricting, when I play a 4-string though. I feel a little more comfortable on my 5, and my 8 is the most comfortable, because it feels like I won't hit the top or bottom of it, when I'm playing.
I'm slowly learning the notes on the fretboard though. They're easier to see/feel on my 5 string. I get lost on the 4 and 8 strings, because on the 4 I keep thinking the low string is B....I just don't know where I'm at, note-wise, on the 8-string.