We have the need to use robots.txt files to hide some areas of our site from robots etc.
Because of the way matrix handles 'files' I beleive we are going to experience some difficulty with these.
Can anyone guide us on this?
What you need is the current development version. In this version, you can set a flag on a file asset so it never uses the __data URL - it only ever uses the Matrix URL.
This feature is not in the current stable release. In this release, your pretty much stuck besides changing the URL in the DB by hand.
Another alternative would be to create your robots.txt file as normal and upload it to your server then use an Alias line similar to the 4th alias line that Matrix uses in your apache configuration to point requests to /robots.txt to the right file.
I use Rodney's method very successfully for both the robots.txt and favicon.ico files on non-Development versions of Matrix.
Thanks for answers gentlemen.
As you guys are administering our server I assume this means providing you with the robots.txt file and letting you upload and modify the apache configuration?
Yup. You can log a support job for that. 
any idea when this feature;
[quote]What you need is the current development version. In this version, you can set a flag on a file asset so it never uses the __data URL - it only ever uses the Matrix URL.[/quote]
will be available in stable release?
It'll be available when v3.6 is released. I'm not sure on a timeframe for that, though.
v3.6 still has a fair way to go. A beta release will be available May/June. Wouldnt expect a stable release for at least 2 months after that.
3.6 has a lot of new stuff in it so it is taking longer to finish and release.