For me, it comes from years of playing and listening, listening and playing.
Here's the thing: there are only 12 notes that we use in our modern music, from A to G#. For every note you hear, you've got a one-in-twelve shot of knowing what note it is.
If you can become very familiar those notes, you can hear most anything.
Start with playing simple scales, playing them over and over and over until you memorize them with your eyes closed. Memorize the intervals and how they sound and, very important, try to be able to sing them or hum them.
Once you get the notes in your brain to the point where you can speak them out, you're on your way.
Then expand to more scales and arpeggios and chord voicings and so on, playing them over and over and over until, again, you can play them with your eyes closed and you can pretty much speak them or sing them out.
Again, there are only 12 notes. But it can take a lifetime learning how much you can do with just 12 notes.
