As you increase the amount of unsprung weight (amount of weight not directly supported by the suspension) - you increase the amount of work done by the suspension. They have to work harder to keep the wheels and tires in contact with the road.
There is also the performance hit in acceleration with heavier/wider wheels. Not a problem if the car has ample power (Dodge Viper, Covette Z06, etc.) But in something with a little less power - the increase in the moment of inertia from the larger, heavier wheel will pull down acceleration numbers. This also goes toward lightwieght alloys - even though they may be very similar to weight of the OEM setup - the distribution of the mass of the wheel will still generate greater moments of inertia.
Granted - there are always exceptions - but pick at random, wheel/tire combinations that are +2 or larger and place them on most cars - their overall acceleration will suffer while gaining better braking and cornering. For some - that lose in acceleration is a small price to pay for better braking/cornering.
How much will it affect the performance - it varies according to the width and weight of the wheels. Usually it is on the order of every increase in static weight by 100lbs, will drop 1/4 times ETs a tenth. For every 10lbs in extra unsprung weight, ETs will drop a 10th.
What this means for most people - probably not much. To lose a half second in the 1/4 or 3-5 MPG is not a deal breaker.