You could risk giving them negative feedback and if they give it in return, try to have it mutally withdrawn - but I'm not sure if the other person has to agree to the withdrawal as well (and they might not if they don't care).