How does the sitemap and robot.txt go together? Why is robot.txt empty when I get a correct tree from sitemap.axd?page=1?
It doesn’t seems that my customers pages gets indexed when only sitemap.axd is returning data.
Litium version: 8.17.5
Your channel has a setting for whether or not sitemap should be generated and if set to true a link to the sitemap should be included in the robot.txt. but it is possible that a sitemap has been generated before and there for exists even if set to false.
you robot.txt is empty if you don’t add robot.txt text to your domain setting.
I have checked the settings on the channel for using sitemap.
Found the “Robots” field on Domain names. Is that what you mean with your last statement? Should I add “robots.txt” in the textarea there?
if sitemap is set to true on channel then that should automatically be visible in your robots.txt even if you leve the textbox on the channel empty.
#www.yousitedomain.se
Sitemap: https://www.yousitedomain.se/sitemap.axd
if you want something other then the sitemap i the robots.txt then you need to add it to the textbox in the channel settings but it is not required,
in test it is good practice to add disallow all and in production to exclude /Litium/
Test
User-agent: *
Disallow: /
Prod
User-agent: *
Disallow: /Litium/
Ok! Doesn’t seem better than that I wasn’t so clear in my description of the error. Robots.txt contains the url to the sitemap (Sitemap: https://www.mysitedomain.se/sitemap.axd) so the configuration looks right. But then it’s probably that I don’t understand how the crawlers use these to index the urls because the customer says that their urls haven’t been indexed.
Their wish is that the sitemap is in xml format (instead of .axd) and robot.txt links to that. Is that possible and a reasonable way to do it? Perhaps wrong forum but since I already started this thread I check.
The sitemap is in xml format both the content and the content-type, only the filename that ends with .axd and that have no impact.
If you running in Litium Serverless Cloud, ensure that you have set the environment to production, otherwise the server will send out headers that will prevent indexing.
I have digged a little deeper and found that the customers pages actually is indexed and also that the format Litium implemented is standard ASP.NET HTTP handler which should be enough for them. Not sure why they request us to change. I have checked all the things you mentioned and it looks correct. Thanks for you explanations!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.