i think it sometimes depends on what alloys you've got.If you've got some thin 5 spoke alloys then the drums are gonna be noticable.
At the end of the day,if you wanna paint them the same colour as your car,feel free to do it,dont let anyone stop you!