What could they do? Slap stasis cuffs of them and put them in jail? Put them on trial for war crimes? Banish them by shooting them off on a rocket to the Moon or Mars? Even if they did, they'd just escape and come back, causing more death and destruction when they land.

Earth is the human and Autobot home. The Decepticons are not welcome here, they are invaders, and have previously shown a callous disregard for lives of the native inhabitants of this world at best, and have murderous intentions at worst. And given the revelation of The Fallen and Megatron's plans, NEST's actions were completely justified.

To paraphrase Major Kira from Deep Space Nine, the Decepticons invaded our home and killed our people. It doesn't matter whether the Decepticons were soldiers or concealed scouts, they are all guilty and they are all legitimate targets.