Mediation and co-parenting

I had mediation yesterday for access with my 19 month old daughter. There is a history of domestic violence so I was going for supervised visits. I initiated mediation as her father would see her every 6-8 weeks and if something didn’t go his way would threaten court. It was a shuttle telephone conference as there is about 1.5 hour drive between us. They spoke to me first but wanted to give me time to talk to my lawyer so spoke to him, about what he wanted (at this stage I hadn’t put forward what we were aiming for). He turned around and said he can’t commit to anything as he doesn’t know when he can come visit (doesn’t work or anything), so to me i find it is as he can’t be bothered making the effort so wants to keep it how it is, so him showing up whenever it suits him (usually at least 6-8 weeks). But the visits he usually has I find are too emotionally traumatic for our daughter, she doesn’t know him so clings to me for the first half an hour then warms up to him, they play and then he leaves again and she is a wreck. In our parenting plan it says visits are to be agreed upon by both parties or through our lawyers. I don’t want the coming and going so I’m thinking when he organises to see her to direct him to his lawyer and say his either all in or all out... I feel so guilty for it though and not sure if it’s the right thing...