Some may not agree wit what I'm about to say, but honestly it's up to you SisterCM. If you feel comfortable looking down at your hands, DO IT. I look at my hands all the time to stay focused on what I'm playing. I look up occasionally to watch for signals from the singers or leaders, but most of the time I'm focused on what I'm doing.
It's not because I'm nervous or because I don't know the board, it's just my way of playing. Many times as I'm looking down, i'm scanning the board deciding when I want to switch to a higher or lower inversion, when I want to add in runs or what not. My mind is always thinking ahead about my next move, and my eyes scan the board so I will know where to go to get the sound that's in my head. That's just me though.
If you truly want to not look down at your hands as much, then u must practice not looking down. Blindfolds and playing in the dark work great. After you've practiced, put it into action during service or wherever.
Even people who read music will glance down at their hands occasionally when there's a big jump or when going to another key, etc. The only people who don't look down are the blind people (not trying to be mean).