I thought of that Adrian.
then I thought what if each websites wants a different robot.txt.
maybe put in a property that lets you select from a common or individual.
=========================
BJ Freeman
http://bjfreeman.elance.comStrategic Power Office with Supplier Automation <
http://www.businessesnetwork.com/automation/viewforum.php?f=93>
Specialtymarket.com <
http://www.specialtymarket.com/>
Systems Integrator-- Glad to Assist
Chat Y! messenger: bjfr33man
Linkedin
<
http://www.linkedin.com/profile?viewProfile=&key=1237480&locale=en_US&trk=tab_pro>
Adrian Crum sent the following on 4/28/2010 2:39 PM: