I feel that! I played for a church 4yrs ago and they gave me the same guilt trip. Even though I played every Sunday morning and evening service, played for the youth choir on their thursday service and at their performances at other churches and ministry events, played at all of the churches conferences, rehearsed twice a week and I had to travel over 25 miles to the Church on each occasion (one way!)....I went on a three week vacation after not having any type of vacation for over 5yrs and when I got back I was told by the ministry leader that I was not dedicated! They had a big conference that was going out on live cable TV and the guy that filled in didn't know how to play most of the songs and wasn't very tight with the rest of the band...so, it was decided it was my fault! If I was ill or needed a break I had no-one to cover me as there was a distinct lack of bassist in the church, so when I needed time off my ministry leader got really snotty about it. There was too much demand on me and it was affecting my personal life as all my spare time to spend with my family and friends was being taken over as I was the only bass player to cover a lot of work. I was also working a day job at that time, so I wouldn't get home much before 10pm most evenings as I would go straight to the church after I finished work. So, I decided to step down and coach some of the younger guys coming through the youth choir.
I was also asked to come back after a few months and I still said no as my heart was with the young guys that I was coaching. As a result they now have 4 young, gifted and very enthusiastic bass players that are rotored every week, so they are never stuck without a solid and experienced bass player.
The only advice I can give is pray about it and have a private meeting with your pastor, ministry leader or even better still, both together! You must be straight up and honest about how you feel. Your hear is in your music and in the way that you help to deliver God's word through your instrument. It takes dedication to do that alone. Your time at home and your time at Church should be balanced otherwise on or the other will suffer...it's the same with an other job. You need to make that very clear. If they value your contribution to the Church they should honour that. If not, I'm afraid your ministry team will always have the same problem with other musicians by being too demanding on their personal time. Remember, you give your time and musicianship to the glory of God! Every note you play no matter how many or how few puts a smile on his face and he will bless you. Pray about it and the answers will become clear as to what you should do. If you need to take a break or get back to it God will speak to you. He knows your heart and he will make the paths for you.
God Bless you.