Correct - need a new o-ring gasket for the valvecover. There are cases where the original gasket is still in good shape and you could reused it - though for $20 or so, well worth getting a fresh gasket.
You can use any petroleum solvent of choice and rag on the valvecover itself, but don't try and wipe down anything on the engine itself - that is just asking for loosening up something and potentially causing you problems down the road. If the deposits are heavy enough where you want to try and clean up the engine - then you'll need to run some engine cleaning treatment first.
NOTE: I prefer change the oil immediately after opening the valvecover - just in case some debris dropped onto the valvetrain. As such, I tend to time this when I get close to an oil change interval.
Valvecover has a baffle that can get pretty nasty, even on a well maintained engine. I personally use carb cleaners, brake cleaners, whatever I have at hand. Most evaporate pretty quickly, so I don't need to wipe anything down. For stubborn spots - I hit it with a nylon parts cleaning brush, spray and soak, repeat. If you have a solvent tank, just dump in the valvecover and let it do its thing.
The OCV filter is right behind a bolt, directly underneath the OCV solenoid (has a electrical connection running to it) - both by the alternator. Once you remove the bolt infront of the filter - some oil will dribble out. The filter assembly itself is a metal screen in a plastic carrier. As most solvents will eat plastic - not a whole lot you can do other than minimizing solvent contact time. If you do it quickly enough, shouldn't be an issue. I've cleaned that filter with throttlebody cleaner and even brake cleaner - just spray to remove deposits, then use compressed air to blow the solvent off. If you don't have access to compressed air - air drying should be fine, just soak up as much solvent with a papertowel first to speed the drying. There is a metal washer on that bolt - check the condition of it, should be good for one reuse - if in doubt, buy a spare washer before you start.
Haynes manual has a decent description of how to check the valves and what the tolerances are. If I remember correctly, it looks identical to the factory service manual (direction and photos).
Just in case, here are the tolerances from the factory service manual (note measurements when engine is cold):
Intake 0.15 - 0.25 mm (0.006 - 0.010 in.)
Exhaust 0.25 - 0.35 mm (0.010 - 0.014 in.)
As for the OCV filter, you should have only found one part number. The DIY guide refers to the 2ZZ-GE engine, which as two OCV solenoids and filters (one for VVTi and one for "lift" portion of VVTLi). The 1ZZ-FE engine just has the VVTi OCV.
