How would you react in this situation?

My partner and I purchased our house end of last year. The property was subdivided by the previous owners and a house is being built on the section next door. The section isn't huge and the builders have been using our driveway for access, even having the ladder for their scaffolding on our property. We let a lot of things slide because we understand that it's safer for them that way or we just don't care that they're standing/sitting on the side of our driveway having lunch it's whatever.

Today, I pull into my driveway after work and I see a pile of dirt sitting at the end of the drive. I look at the security camera and I see a digger has been up and down the driveway. I tell my partner and he flicks a message to the foreman letting him know and asking to just ask us in future if they need access through our property and for the dirt to be removed in a timely manner. He says he had no idea about it and he'll look at their security cameras, then comes back calling my partner names and cussing him out. My partner tells him if the dirt isn't removed in a timely manner we will be getting someone in to remove it and billing them for it and asks him to no longer contact him. He responds with another insult and threatened to charge us with the clean up from a leak we had (which was fixed within a week of knowing about it) and for the fence between the property (they've already put up the posts without a written agreement to us so can legally not charge us half) . We have contacted our lawyer and downloaded camera footage.

What do you think?

How would you react in this situation?
Post Opinion