I’m almost in shock: Technorati admitted on their blog that they sometimes don’t index sites properly:
The good people at Strumpette contacted us [… ] told us that we hadn’t been processing updates from their blog for a few months, even though they’d complained about it. We looked into this more closely, and it turned out they were quite right.
This is the first time I’ve ever seen an admission from Technorati that they have issues spidering. Kevin Marks explained the problem:
If the feed is not full content, we correlate posts between the feed and the HTML of the main page, to see if there is more there (many sites have brief summary feeds, and full posts on the main page; some have the opposite).
However, until yesterday, this code had a problem with sites that appear with and without ‘www’ in front of them. For example Strumpette’s permalinks start with ‘http://www.strumpette.com’ in the feed, but we were indexing the HTML from ‘http://strumpette.com’, seeing relative permalinks from the page, and missing the correlation between the two. This bug is now fixed, and we handle this case better, so Strumpette and others will see better indexing of their posts.
Why, then does Technorati processing seem so random? This blog has had periods of a week to two weeks where Technorati stops indexing, and then spontaneously starts up again. In fact, many people have had similar experiences. Just read some of the comments on these posts:
- Technorati, Technorati, Technorati
- Bloggers care enough about Technorati to want it to be better
- Will Technorati ever come back up?
- Sound of one blog clapping
- Technorati having issues
- Is Technorati broken?
- Somebody, anybody, please buy Technorati
- Technorati and Google: What’s taking so long?
Time and again, after I post on Technorati, pinging, searching, and problems, bloggers come forward and say: yup, that’s happened to me to. Frequently, it’s still happening to them. And it’s happening right now on this blog.
I publish full feeds, and WordPress supports both Atom and RSS – Kevin mentioned in his post that Atom was best. Here’s what I’d like to ask Kevin Marks (or anyone at Technorati):
Why are there always problems with Technorati’s pinging?
. . .
. . .
Posts discussing these issues:
[tags] technorati, problems, ping, kevin marks, strumpette, john koetsier [/tags]