You're both right. Back-pressure and loss of torque with aftermarket exhausts are wildly mis-understood.
I think the problem lies in people's interpretation of the terminology involved here. For a physics standpoint - it is about flow velocity, scavenging, volumetic efficiency and interference.
In a stock exhaust system, the pipes are smaller and the velocity of the gases in the exhaust pipe is high. As the engine pumps the exhaust gases into the pipe at low RPM - a wave or pulse of pressurized gas travels down the pipe at a high velocity. After the exhaust pulse starts to travel down the exhaust pipes, a low pressure area forms behind it. This draws even more exhaust from the cylinder, leaving a partial vacuum in the cylinder as the exhaust valve closes. This is the so called scavenging effect.
The partial vacuum remains in the cylinder as the intake valve opens. The air/fuel charge, in turn, becomes "packed" into the cylinder - creating a denser air/fuel charge than without the partial vacuum. This increase in volumetric efficiency will in turn result in more power and torque.
The problem with larger diameter exhaust pipe is it has lower flow velocity, thus a lower scavenging effect.
The term backpressure - comes from the fact that as a pulse goes down the exhaust pipe, it will cool and slow down. The subsequent exhaust pulse will actually reflect off this slowed pulse and head back up to through the exhaust manifold to the exhaust valve.
What does all this mean - for most people (ie. non-forced induction) - a smaller pipe actually has LESS backpressure (reflected pulse) than a larger one. The idea that a larger pipe will flow more, is true, in the right conditions.
At lower RPMs - A more "restrictive" exhaust (smaller diameter pipe) will generally produce better torque numbers than a "freer flowing" exhaust (larger diameter pipe). This is probably where people get the idea that back-pressure is essential to the development of torque.
At higer RPMs - the opposite will be true. Since the larger diameter pipe has a greater cross-sectional area compared to the smaller pipes - the closely spaced pulses tend to "back-up" more in the smaller pipe.
A caveat: This is all complicated with the different types of aftermarket parts. Some headers have ceramic coatings inside and out - that help with controlling the speed of the pulses (hold in heat). Some use venturi designs to do the same thing (necking down, or different sized pipes). Engine valvetrain technology like variable valve timing and variable camshafts (ie. VTEC, VVT/VVTL, Vanos, etc.) also change the approach to making power.