Bots can't follow label cloud?

[QuirkyJessi]

A while back, I changed my normal archives into a "label cloud" using this set of code: New Blogger Tag Cloud / Label Cloud

Now when I check Google's webmaster tools, though, it says I have 51 "URLs restricted by robots.txt".....and the links are all the label tag links.

Any idea why the bots can't crawl those pages? I don't see a no-follow snippet in there right off. I'm tempted to just switch back to the regular format without even waiting for an answer, but if you know why or how to change it so that bots can crawl, it'd be great.

[rjleaman]

You have archives (post files stored by 'date') and your tag cloud (effectively, search results sorted by 'label'), so two different animals. You're on blogspot, so you don't have access to your robots.txt file, so I don't see much you can do to make the tag cloud get spider-crawled as well as the archives. But here's the thing - are you sure you want to have it crawled? Because it would just count as duplicate content. I just did a bit of SEO-tweaking on one of my Wordpress blogs to stop some of my duplicate pages from getting indexed (categories/tags/archives/author are all different ways of getting to the same content) and saw an almost-immediate increase in traffic from Google. Since Blogger is owned by Google, it's possible that they've put that restriction on there on purpose, I'm thinking...

[QuirkyJessi]

Yeah, I checked one of my other blogs after I posted this and its labels aren't being crawled either....and I have it back with its original settings, so it's not just the cloud. I knew they used to be indexed, though, because I used to get quite a bit of traffic on my label pages....and now I'm not obviously. I'm guessing maybe they did put in restrictions because the others aren't either, but it still seemed a bit weird.

[keym4k3r]

In robots.txt you have the line Disallow : /search
and that means the spiders will not crawl it.
This is mostly to avoid duplicated content issues, as the search results just repeat the posts, there were already indexed.

[anshul555]

yups It doesnt index tag cloud and labels are alos not crawled by it sometimes