Robots.txt

Squiz Matrix General Discussion
1

Given a matrix installation with multiple sites, how can I implement a different robots.txt file for each site? Some sites I don't want indexed, other sites I do, and for the last lot of sites I only want to allow partial indexing. Being able to use a different robots.txt file for each site will be more convenient than using meta tags with customizations for the partial indexing case.


Ideally I'd want some way to manage this from within Matrix. But I can't see any controls to do it.

2

More generally my problem is that for certain file types I can't assign a webpath to. Prime examples are favicon.ico and robots.txt. For these files the url needs to be www.yoursite.com/favicon.ico and www.yoursite.com/robots.txt. If Matrix allowed me to do this I could manage both files as assets of the sites.


However, I guess in this case Matrix should provide a setting for favicon.ico and a configuration page for robots.txt per site asset.



Right now I'm working around the problem with aliases per virtual host. But this means I am now managing content outside the content management system. :wink:



Is there a better way to do this?

3

I think you would create a text file assett at the site root, and upload the robots.txt file.


It will be pathed as /robots.txt





Richard

4

Not once it goes live (unless you're on a development version where you can choose to use the full matrix paths).

5

Not once it goes live (unless you're on a development version where you can choose to use the full matrix paths).


Ah, yes, I've been talking about matrix alot but haven't said which version I'm using. :wink: I'm using version 3.4.1. So this functionality will be available in 3.6?

6

Not once it goes live (unless you're on a development version where you can choose to use the full matrix paths).


Woops…



R

7

If you're using 3.4, set up a Redirect Page to point to a robots.txt file stored somewhere else in the site.

    Site
     |-- robots.txt (Redirect Page)
     |
     `-- Robots (Folder)
          |
          `-- robots.txt (File)


Make both live, and the Redirect Page will have the path: http://my.system.url/robots.tx
8

I don't have the CMS package. But if I did, does your work around return the correct mime type?