Oak Switchplate Covers

We had some crappy remodeling work done to our downstairs bathroom about 3 years ago. We had an exhaust fan installed and had a switch plate put in with two switches (one for the light, one for the fan). Unless it is just my imagination for the past two days I have noticed that when the fan and light are running, when you go to turn the fan off the switch plate seems warm. Also the lights will flicker when turning the fan on and off. Is this a fire hazard? I am seriously starting to fear that my bathroom will catch on fire while I am in the shower.