Very good question and a tough one. The way my instructor has told me is that unless you know where the song is going before you play it or have the chord chart for the song it would be pretty difficult to mentally run through all the modes before you find the right one to use. If you have a firm hold on how the Modes work in a piece of music then you will know how to apply them.
I'm still learning the aspects of Scales and Modes myself, so I don't know nearly as much as I should. And they are still difficult for me to learn too. This is a chart I made from the lessons my instructor has taught me. You may already know this stuff, but I will write it so that if someone else has the same questions they can understand too.
First, every key has 7 modes. They are:
I= Ionian
II= Dorian
III= Phyrigian
IV= Lydian
V- Mixolydian
VI=Aeolian
VII= Locrian
The modes run in order from the first note of the key to the 7th note of the key. Example:
In the key of G the Ionian root is G, and every note follows in order.
Ionian = G
Dorian = A
Phrygian = B
Lydian= C
Mixolydian= D
Aeolian= E
Locrian= F#
So when a song is in a 1-4-5 progression the root to each chord change is G-C-D. So in an unaltered chord type the chords would read. GMaj or GMaj7- CMaj or CMaj7 and D7.
Also a very key piece of info. is that a progression does NOT have to go in order from 1-7. The progression can vary at any time and more than just 3 progressions can be used in a particular song. I just used the standard 1-4-5, because it is so basic and so widely used.
And this is how you know which to play.
Ionian Modes are played over M, Maj7 chord tones
Dorian modes are played over m, m7 chord tones
Phrygian Modes are played over m, my chord tones
Lydian Modes are played over M, Maj7 chord tones
Mixolydian Modes are played over Dominant 7 chord tones
Aeolian Modes are played over m, m7 chord tones
Locrian Modes are played over Diminished 7 chord tones
Each Mode has it's own whole note/half note formula or finger pattern, which you are so used to seeing in diagrams. I won't diagram each pattern but that's the way musicians know which mode or scale to use. The language of music is universal and everyone that learns theory has to learn the same things. Piano players, guitarists, bassists, violinists, flute players, you name it. The only musicians that don't learn theory in the same manner are percussionists. They learn more of the rythym studies thoery and what not. I Hope this helps you.
