Well, judging from your side of the story only, I would call him at a time when you know he's alone and express to him your concerns. (It's best to have everything jotted down in writing so you don't become too emotional to remember key points. Also, if he interrupts you, you'll be able to stay focused and come back to your point without getting off track). You should jot down his responses as thoroughly as possible. Before the conversation is done, tell him that you intend to contact his pastor because you've done everything else in your power to keep the situation peaceful and healthy for you and the children. Lastly, contact his pastor (using the same points you discussed in your conversation with the ex).
As far as I'm concerned, cutting off contact with the children in situations where they aren't being harmed should be an absolute LAST resort, only done after consultation with your lawyer, your pastor, and the Lord. That's a very serious and drastic measure to take, and can cause more harm than good.