Never had an accident in the Corolla (knock on wood) - but I had a Camry that was involved in three separate accidents - none of which were my fault - rear ended, t-boned, and both front and rear crumpled after being spun on the highway after a hit in the rear quarter. All required the frame to be straightened - I know what you have heard - if the shop is not competent enough, they will do more damage to it than good.
Still - if the car is put on a body jig and found to be bent - if must be straightened. A car will take a bunch of damage before it becomes apparent during normal operation - even if it road perfectly - will you be willing to risk it? Lots of weird stuff can happen if the frame is bent - not now but mabe six months down the road - could be as small as a squeak that doesn't go away - could be as bad as structural problems or window cracks due to the strain on the body.
We felt the same way - insurance did the repairs the first two times and it wasn't right - the third time we got some money from the insurance but only for a preset amount on what they thought it should cost (may be different from state to state, may be different for different insurance co.). Took it to a race shop to be straightened - about as perfect as it can be now. No squeaks like the first two times around.
Shop around, get references, don't take the first shop or insurance "approved" shops unless they prove themselves competent. Don't let the frame go - a bent frame is a weak one. Might not take well to another accident.
Be safe and Good Luck.