Is it possible to hide a matrix folder from search engine crawl?

Squiz Matrix General Discussion
1

Hi,

Can a folder of files be hidden so the site search engine doesn’t crawl them, but can still be accessible through a direct link (i.e, a link in an email)?

Or can Matrix only able to do this on an individual file using asset metadata (page.robots)?

Thanks

2

Hi Manjeet

I would think that in terms of the folder itself, that would be dependent on the design applied. If it has the standard design applied then the folder would appear as a page and you could use the metadata field to prevent indexing.

However my understanding is that once a document is published it is offloaded to __data and no longer has the folder URL.

I have set up robot.txt file using a search page to list any asset that has robots set if that would be of any use?

3

Thanks Matthew,
I actually don’t want to update robots.txt for each folder as it will b lie a document library but shouldn’t be indexed by search engines.

I will probably go with metadata field to prevent indexing but might need to keep relative url and not use __data