Does a non custodial parent need permission to travel out of state with his kids?

Or is 24 hours enough notice legally. Im an in MN. I have physical custody and joint legal. Never married. We have a decent relationship. But he takes them all the time without telling me. And on a few occasions not brought them back on the correct date because he didn't want to.