If it's black plastic, and not plastic with a layer of black paint, you could try putting rubbing alcohol (isopropyl alcohol - I got mine from Bunnings) on an ear bud and carefully rubbing.
You could also try scraping it off very carefully with an exacto cutting blade/stanley trimmer. It's what I did when I saw my TLK voyager Optimus had a silver blotch on his face.
Results not guaranteed: your mileage may vary.
Have a go at it with a tooth pick and the isopropyl alcohol, the tip is soft enough to scrape the paint and not damage the plastic and allows more control over the paint removal.