For my MP 01 & 04, I forcefully pulled them out.
But you need to do that with caution; the battery box itself won't be much affected. But you need to be aware of the parts around it: the chest doors, the flip boards above and under it, etc. Make sure there's nothing in the way when you pull it.
And after you've got the batteries out just forcefully push it back in. That small red cap over the rivet is actually quite tough.
The trick is to do it quick in one swift pull/push. (And it doesn't require much strength to do it really, so no need to pull it too hard. As long as the direction is vertical and straight it should be fine.)
With my MP 01, because it was my first experiment, after I've reinstalled the battery box it appeared to be a bit loose, and fell out itself. But there's no damage, and I just put it back again. I worked on my MP 04 more experienced and everything went perfectly, the same issue didn't recur.