That is hard to say - as there are quite a few owners that I know personally that haven't had any issues with their newer Toyotas. My own 2003 Matrix XRS is running strong with over 115K miles on the clock.
Quality control dropping starting in 2003 - that exact date is pretty hard to believe, as there is no ON/OFF for quality control across the Toyota lineup. Personally, I feel it is not any specific date, but geared on certain generations and models. Some older Toyota products had known issue that were addressed in newer generations of that same model line. In those cases, the newer models were superior to their older counterparts, quality wise. In the same sense, in other product lines, some of the older models were superior in quality compared to their newer replacements.
I'd say it depends on your budget, what you want from the car, and how it will be driven. Just test drive them back to back and see which catches your fancy. In my case - I couldn't get comfortable in the 2003+ Corollas, no matter how I adjusted the wheel and seat. Just felt like not enough seat travel, even though the car was significantly larger than my 2002. Matrix was a different story - its like driving a truck - one that was big on the inside and small on the outside.