Yahoo! Pipe to pull Technorati API data for multiple blogs

The problem with the perl-based approach is that it’s a little inaccessible to people who aren’t comfortable using a command line environment. So I really wanted to make something that more people would feel comfortable using, and perhaps play around with.

So, with some help and kind words from Bob Briski, one of whose pipes I’d stumbled across and bookmarked during my research for this project, I decided to finish off the pipe and publish it so that others could use it, or (better still) improve upon it.

What does it do?

The pipe first pulls a list of blog URLs from a Google spreadsheet, then checks Technorati to get the “inbound blogs” count for each blog, then outputs the blog title, link and count to an RSS feed.

Technorati Authority (a useful metric when ranking a list of blogs) is calculated as “the unique blogs linking to a blog over the past six months” — in a way it’s a measure of how many readers may be influenced by a blogger at one remove.

Before you start

You’ll need:

1. An API key from Technorati
2. A Yahoo! account
3. A Google account

I’ve done all the rest, so you could simply clone my pipe, which you’ll find here.

Publishing a Google spreadsheet as a CSV source

In my first pass at this challenge, I used a simple text list of URLs stored on this server. That’s pretty easy if you have (a) a server, (b) a text editor, and (c) some way — like ftp or ssh — of getting the two to talk to each other. But lots of people don’t have servers these days, and still more probably have no idea what I mean by (c), so I decided to follow a new approach before writing this article. Everyone, I thought, must have access to Google Docs.

So here’s my test list as published on Google:

Now all I have to do is Share > Publish as a web page, then click More publishing options and select ‘CSV’ or ‘TXT’ (1) as the publishing option to create a simple text list.

publishing google spreadsheet as a csv

You can see the CSV file here. You need to copy (or make a note of) the link (2) because we use it as the seed for our pipe.

One of the nice things about using Google Docs is that it’s easy to edit, view and share the list. For whatever reason, you may not want to share your list of blogs so if you’d like the original version with the server-based file which is more ‘private’, it’s still available.

Setting up the Pipe

The first thing to do is to build a URL for each request I’m going to send to the API. Technorati takes requests in the following format:
http://api.technorati.com/bloginfo?key=[apikey]&url=[blog url]

setting up the pipe

The pipe takes two variables that will need to be customized for your own version. One, rather obviously, is the link URL for the Google CSV that you copied or made a note of above (you did make a note of it, didn’t you?). The other is your Technorati API key for which you’ll probably need to sign up. It’s free, but is limited to 500 calls to Technorati’s database per day.

I’ve plugged the link into the Fetch CSV module at top left, and the API key into the Private String module. I use this module so that I can protect my API key — no-one else should be able to see it now.

The URL Builder module placed within a Loop module does this. I’ve hooked the API Key from the Private String into the parameter named ‘key’, and the output of the Fetch CSV module into the ‘url’ parameter.

Getting the data from Technorati

getting the XML by calling technorati's API

Now it’s simply a matter of looping through the requests that I built during the set-up stage above, and sending each one to Technorati so that I can fetch the data. Technorati responds with an XML file that looks like this:

[code lang="xml"]





    
        http://www.mediaczar.com/blog
             
                Mediaczar
                http://mediaczar.com/blog
                
                http://feeds.feedburner.com/mediaczar/posts
                14
                21
                2009-01-06 12:45:23 GMT
                422833
                
                    
                        mediaczar
                        Mat Morrison
                        Mat Morrison is a digital marketing and
                        communications strategist with over a decade's
                        experience in online advertising, eCRM, and social
                        media.
                        http://technorati.com/people/technorati/mediaczar
http://static.technorati.com/progimages/photo.jpg?
                        uid=139758
                    
                
             
                14
                21
    


[/code]

I can choose which bit of the XML file I want to receive by setting the ‘path to item list’ parameter in the Fetch Data module. You’ll see that I’ve set it to ‘document.result’ — compare that to the XML example above and I think you’ll see how this works: I’ve removed the wrapper information from the file and gone straight to the meat.

Renaming the fields to suit RSS output

renaming the fields to suit RSS output

Checking the output of the Pipe

Now I’ve saved the pipe, I can run it to see if it’s working.

checking the output of the pipe

Importing the RSS feed back into the Google Spreadsheet

Google Spreadsheets has some useful functions that you don’t find in Excel; and which are more geared towards the web. The only one that I’ve really played with is the importFeed function which — as you might expect — imports RSS feeds into your spreadsheet. So here, we paste the RSS link from the pipe into the sheet…

google spreadsheet showing output from importFeed function

Thanks to IBM’s Many Eyes service it’s relatively simple to create complicated visualizations that my current version of Excel can’t handle. For example, this “matrix chart” that I built using Excel’s bubble chart function is clearly unacceptable. I can’t easily link statements or values to the X and Y axes, and there’s lots of overlapping that seems (after many attempts) to be impossible to fix.

Matrix chart built using Excel

This chart depicts the statements that generated most “consensus” (either agreement or disagreement.) While this might be interesting to some people, I’m actually looking for areas where there’s least consensus — the whole point of the blogger typology is to identify those variables that will, so to speak, separate sheep from goats and apples from oranges. To continue with this spurious analogy, I’m looking less for statements like “I am a ruminant” or “I am a fruit” and more for statements like “I enjoy eating laundry” and “I am a citrus.”

So here’s a quick look at those statements:

You should immediately be able to see that these responses are much less polarised than in the previous chart. That’s the sort of thing that we’re looking for.

Caveat

Please bear in mind that this current sample is self-selecting, and sourced online via Twitter, this blog, LinkedIn and WOM referrals. What I’m working on here are the techniques I’m going to be using, rather than the actual analysis. I’m really quite into the prototype-and-fix approach to planning (more like “get it up and fix it later” — it means I spend less time beard-stroking and more time in front of a martini.) So the current results aren’t very robust, but they will help us focus our future research.

Where can I download the latest data?

Here, as a Google Docs spreadsheet. If you do anything interesting with the data, do please let me know, and I’ll link to it from here. Feel free to share this spreadsheet as http://icanhaz.com/blogger_data

I’ve published the first dump of survey and “blog metrics” data from the blogger questionnaire as a spreadsheet on Google Docs. Many, many thanks to all of you who volunteered your information.

Please feel free to use this as you see fit for your own projects. I’ve anonymised this data (just because it’s best practice, not because I think any blogger would be mortally offended by having the world know what inspires them to blog!)

I’m slowly ploughing through the data and doing the quant analysis, but I thought I’d share a few bits and pieces first.

I’m not really in a position to offer anything other than very top-line results right now. Remember that the purpose of this exercise is to create a broad and commonsensical “blogger typology” that will help us with our planning and engagement programmes.

All I have here are “descriptive” statistics; that is, data about the data themselves. You can jump straight to the results should you so wish. The next bit is just a rumination of what we can measure (I’ve left out the “why” for now, but would be happy to discuss.)

Blog metrics background

First of all, a quick aside about how we chose what we’re going to measure. This has been a source of great interest to us at Porter Novelli, and I think will continue to be so over the coming year.

There are any mumber of blog metrics out there to choose from. I’m rather interested in Stowe Boyd’s Conversation Index, which he defines as:

The ratio between posts and comments+trackbacks (posts/comments+trackbacks)

A healthy blog, he suggests, has a Conversation Index less than 1.0 — that is, there’s something more than simple broadcasting going on. To me there seem to be only two downsides to this metric:

1. a poorly-moderated blog with do follow links in the comments section will often have a lower conversation index than the norm.
2. it’s hard to generate Conversation Index data automatically for a large number of blogs

This says nothing about using the Conversation Index as a performance indicator for one’s own blogs and posts of course (indeed, it’s to be recommended.) It would probably work well for Twitter accounts, too – we’ll try to take a look at this when we’ve finally completed the twitter eavesdropper.

But taking Stowe Boyd’s post as inspiration, here’s what I’d like to be able to collect and store from the blogs I’m looking at. These are, I think, the relevant data points that are common to nearly all blogs, and from which a more complex and meaningful set of metrics (like Boyd’s Conversation Index, Technorati’s Authority, or indeed Google’s Page Rank, AideRSS’s PostRank, and even Porter Novelli’s own network analysis metrics) can be constructed.

At this point it’s worth inserting a note of reality. As mentioned above, collecting this data is difficult. Since every blog is different, every blog’s schema (the way that its information is structured) differs. XML feeds like RSS and Atom go a long way to fixing this (hence the focus on “last 10 posts” for some of the metrics listed above) but even then it’s hard to automate.

So — right now I’ve been using the mechanical turk approach to getting the data. I mostly use the excellent and reliable Get A Freelancer, but given that I’m part of a global network I’m also looking at Amazon’s service (I don’t understand why this isn’t available outside the United States.) I like the idea of paying per-transaction; it offers more flexibility.

The big problem (as with any data entry project) is accuracy, and I’m still working on the processes for that (although I do have a couple of ideas.) I’d be interested to hear from anyone who has some experience in this area so that we can share ideas.

Early results

Our standard set of blog metrics are based on well-known CRM RFM metrics: recency, frequency, and tenure. Obviously we can’t get “spend” data (the “M” in RFM), but we can substitute it with Technorati’s “Authority” data. This seems (and is) pretty arbitrary — but it’s relatively easy to automate as demonstrated in yesterday’s post on using perl to access Technorati’s API.

Let’s look at the data we’ve gathered.

Recency

We calculate “recency” as the number of days since the latest post (in Excel we subtract “date of last post” from “date of retrieval”.) When it comes to recency, the lower the score the better: it implies an active blogger.

The first box plot is heavily skewed by one outlier who hasn’t posted for over three years.

Box plot describing posting recency of 62 bloggers who completed questionnaire

So in the following box-plot we’ve removed that outlier; none of the other numbers really changes of course, but we can display a better picture.

Box plot describing posting recency of 61 bloggers (one outlier removed)

Frequency

To calculate frequency we look at the number of posts in the last complete month. In some studies, frequency of posting has shown a high correlation with Technorati Authority (of which more later.) I expect to see a wide range of frequency “behaviours” when we come to do the segmentation; people who are mostly link blogging will, I think, show higher frequency than people who are opinion blogging or announcement blogging (these terms are borrowed from current typologies, and have nothing to do with the final product!) It’s hard to say therefore that “higher frequency is better” but again, it’s an indicator of active and engaged blogging. Or possibly, of course, of a spam blog.

From the public relations engagement point of view, a blog with higher frequency will have a higher probability of carrying our story and (in all likelihood) a larger readership.

Box plot describing frequency of posting of 62 bloggers who completed questionnaire

The “complete month” we looked at for this exercise was December; numbers are likely to have been affected by the holidays (and affected differently for different segments.)

Active tenure

We define ‘active tenure’ as the number of days between the first and the last post. When taken together with recency and frequency, this is a good measure of commitment: someone may have high recency and frequency scores, but if their tenure is low, then we have no guarantee that this behaviour will continue into the future. Lots of us start blogs in a fit of enthusiasm only to find that work begins to get in the way, and that we have less time to post than we did! Occasionally we’ll find a blog that has relatively high active tenure and low recency; we’d read that as a bad sign.

Box plot describing active tenure of 62 bloggers who completed questionnaire

Authority

Technorati’s “Authority” is the only metric that we don’t calculate ourselves. Indeed, as per yesterday’s post, I’ve just succeeded in automating this bit of the data gathering process.

Technorati has this to say:

Technorati Authority is the number of blogs linking to a website in the last six months. The higher the number, the more Technorati Authority the blog has … The best way to increase your Technorati Authority is to write things that are interesting to other bloggers so they’ll link to you.

Right now, Technorati’s Authority scores range from 0 to 28,378 (HuffPo, as retrieved on Sunday January 4, 2009.) Our respondents (as seen below) range from 0 to 356, with a mid range from 5 to 41. I don’t think that this really gets us where we want to be going (although it does guide our hand a little better when it comes to looking at the next rounds of data gathering.)

Box plot describing Technorati Authority of 62 bloggers who completed questionnaire

A word about the box plots

We use box plots as a simple way to represent and compare data. Simply plotting the average (arithmetic mean) of the data disguises more than it reveals; a few outliers (as we have seen) can drastically alter the picture and give an “unrepresentative” picture.

Each box plot shows the following information:

1. Maximum (the highest value observed)
2. Minimum (the lowest value observed)
3. The Interquartile Range or IQR (where the middle 50% of the values fall: this is a more robust statistic than the full range, and is depicted as the “box” from which the box plot gets its name)
4. The Median (the number separating the higher 50% of the sample from the lower 50%, the median gives you an idea of which way the data skew.)

Where can I get the latest data

Here, as a Google Docs spreadsheet. If you do anything interesting with the data, do please let me know, and I’ll link to it from here. Feel free to share this spreadsheet as http://icanhaz.com/blogger_data

How were the respondents recruited?

To date, everyone has been self-selecting. I’ve used four main channels to promote the questionnaire, this blog, Twitter, a LinkedIn Q&A, and word-of-mouth. If you’d like to take the questionnaire yourself, please do — it’s never too late. Furthermore, I’d be most grateful if you’d pass the link along (please post it on your blog, or on twitter, or send it via email as http://icanhaz.com/blogger_questions).

Here (courtesy of Google spreadsheets) is how the current set of respondents got to the questionnaire (WOM isn’t tracked here, just the seed link.)

Sometimes (for instance when I’m doing the research for the blogger typology) you need to get a whole load of Technorati data for a whole load of blogs.

This research can (of course) be done by hand. And (of course) for a long list of blogs this would take a great deal of time. Handily, Technorati provides developers with an API that lets you automate those queries. An API (for those of you who don’t know) is an Application Programming Interface – a toolkit provided by a service or application (in this case by Technorati) that lets other computer applications ask it questions and use the answers for their own purposes. It may be helpful to think of APIs as being like the knobs on top of a Lego brick that let you stick other Lego on to it without in any way changing the nature of the brick itself. On the other hand it may not be so helpful after all.

After much struggling with a Yahoo! Pipe to query the Technorati API for a list of blogs, I was forced to abandon my attempt. I would have liked to have shared that Pipe with the world (if you’re good with Yahoo! Pipes, do please take a look at it and see if you can help me!) [Tuesday January 6, 2009: Thanks to help and encouragement from Bob Briski, this now looks like it's on its way to working!]

Instead, I’ve written a perl script to do this. Perl isn’t as easy for people to use for themselves as Pipes, but if you are comfortable with a command prompt, then you’re half way there.

What this script does is take a list of blog urls, and for each item in the list queries Technorati for the following information:

1. Blog title
2. Inbound blogs (the number of unique external blogs linking to the blog over the past six months, this is also known as “Technorati Authority”)
3. Inbound links (the total number of links into the site)
4. Technorati Rank (a sort of overall score)

The script

[code lang="perl"]#!/usr/bin/perl
# use modules
use LWP::Simple;
use XML::Simple;
# set up variables
open(INFILE, $ARGV[0]) or die "Can't open list of blogs to read: $!";
$apikey='enter your Technorati API key here';
# create object
$xml = new XML::Simple;
# read each line, and make the Technorati API call
while () {
chomp;
&callTechnoratiAPI;
}
sub callTechnoratiAPI {
$url = 'http://api.technorati.com/bloginfo?format=xml&key='.$apikey.'&url='.$_;
# get XML file from Technorati
$content = get $url;
die "Can't get $url" unless defined $content;
# read XML file
$data = $xml->XMLin($content);
# access XML data and print TSV to screen
# (you can fiddle with this as much
# or as little as you like)
print ""$data->{document}->{result}->{weblog}->{name}"t";
print "$data->{document}->{result}->{url}t";
print "$data->{document}->{result}->{weblog}->{inboundblogs}t";
print "$data->{document}->{result}->{weblog}->{inboundlinks}t";
print "$data->{document}->{result}->{weblog}->{rank}n";
}[/code]

How to use it

I can’t give you any real advice on how to run perl on your system. If you want to play around with it, Macs come with perl already installed, Windows users should download and install the free ActivePerl. But you’ll need to install the perl bundle XML::Simple, and I don’t know where to begin telling you how to do that if you don’t already know how perl and CPAN work. You see why I wanted to use Yahoo! Pipes?

If all of that doesn’t bother you, you’ll also need to sign up for a Technorati account (if you’re into this sort of thing, you should already have an account), and get your free API key. This key will let you make 500 queries in a 24-hour period, so you’ll need to plan how you use it.

The script as it’s listed above outputs tab-separated values to screen like this:
matm% ./parse_technorati.pl bloglist.txt
"Chris Gilmour's Diary Vol. 14" http://www.illandancient.blogspot.com 6 10 861604
"The Red Rocket: Technology, PR and social media marketing" www.theredrocket.co.uk 15 29 397843
"Going Underground's Blog" http://london-underground.blogspot.com 254 467 13332

The blog’s title and url are followed in order by the inbound blogs (authority) count, the inbound links count, and the Technorati rank.

I use tab-separated values because that makes it simple to cut-and-paste directly into Excel or Google Spreadsheets for further analysis.

Known bugs

Right now, the script occasionally throws out something like this:

matm% ./parse_technorati.pl bloglist.txt
"Lytham Villa" http://lythamvilla.blogspot.com/ HASH(0x8ff7a0) HASH(0x8ff7f4) 4978471
"KickTime || A Driftless Regional Webspace" http://kicktime.org HASH(0x908e0c) HASH(0x908db8) 1951828

I’ll work on this, but if anyone can point me in the right direction, I’ll be most grateful.

(NB: If you have both a blog and a short attention span, please skip the article, and go straight to this short survey. Many thanks!)

What is a blogger? Everyone seems to think they know, and yet the longer I work in this area, the more I realize I know nothing. And the less I know, the more suspicious I become of marketers who use vague terms like “conversation” (which has – after all – become little more than a Latinization of the ghastly “dialogue”.) I can just about understand what Technorati means when they talk about

The ecosystem of interconnected communities of bloggers and readers at the convergence of journalism and conversation.

(State of the Blogosphere 2008)

…but there are an awful lot of long words that could turn out to hide an awful lot. And that’s the carefully thought-out distillation of a bunch of experts. Most of us, most of the time fall back on lazy or confusing language. We talk about “social media” and never stop to think that — depending on who’s doing the talking (and what they have to sell) — what is meant by that apparently innocuous phrase shifts wildly from speaker to speaker.

I think that we do the world (and ourselves) as much of a disservice by lumping together a bunch of web sites based on the fact that they share a similar technology as we do if we can only lump all fiction, non-fiction, reference, text-books, guidebooks and manuals together as “books.” We need a better classification.

I’ve spent a lot of time thinking about this sort of thing, and now I think we’re ready to do something about it. But while my ambitions are actually rather modest, I need a lot of data to get started. This is where you come in. I really need your help.

(Seriously, if you’ve read this far, and aren’t sure if you want to read the rest of the article, please don’t. But please, if you’re going to leave, take this short survey. Goodbye, thank you, and see you soon I hope!)

OK. Here’s a quick distillation of what I’ve been thinking about that will help give us some context.

All blogs are not the same

This seems abundantly obvious. But I’m not talking about topic here. What else differs? Let’s look at what we know:

Audience reach differs

I am fond of drawing charts like this:

We know that a few large blogs together reach the great majority of the audience. The great majority of blogs, on the other hand, will reach no more than a few dozen unique users, let alone unduplicated audience, in their lifetime.

And yet we don’t have the language to distinguish between the various areas on the spectrum. Sure, we’ve borrowed the term “A-list” from the entertainment industry to describe the really big guys. But is that going to be enough?

Target audiences differ

Different bloggers write for different audiences. They may write for their friends and family, their peers, their colleagues, or for “people who are into the same sort of thing.” Some write for themselves, for people they know, or people with whom they have developed a writer/reader relationship, while others write for one-off strangers who are typing search terms into Google.

Or they may write to satisfy or attract customers and clients. Or to communicate with journalists and analysts. It seems perverse to lump “corporate bloggers” in with the rest of the blogosphere.

Blogger motivation differs

For the past three years, the Bivings Group has published research on The Use of the Internet by America’s Largest Newspapers. You can see (from the chart above – well, if you click and blow it up, you can see) that nearly all of them have featured “reporter blogs” for a couple of years. This highlights a point that’s worth making: at one end of the blogger audience spectrum are a bunch of people who are paid to blog. Whether they draw a salary, or a traffic-related bonus their motivation seems pretty clear. As does that of the blogger who joins an ad programme, or an affiliate programme, or who seeks sponsorship.

Corporate bloggers are also driven by material needs; whether they’re trying to promote their product, service or organization, there’s a financial impetus behind it all. It may be harder to measure, but it’s pretty clear why they’re in the game.

Other bloggers, on the other hand may be motivated by deeper Maslow needs; to express themselves, to explore and share ideas (me), to make a name for themselves on a wider stage (probably me, and most of you who are reading this.) Or they may want to record their experiences and share news with friends and relatives (travel bloggers, for example, and some mommy bloggers.)

Or perhaps their blog simply gives them a better platform than email to share the amusing/interesting/shocking/useful stuff that they find on the web, a sort of digital scrapbook.

Why I think we need a typology

It seems to me that — as marketers and communicators and public relations people and social media experts — we need to distinguish between different types of blogger quickly and easily, and that it would help to have some kind of shared language with which we can do this.

It will help when formulating plans and approaches. For example, and off the top of my head, right now one can probably approach journalist bloggers as though they were journalists, while it’s probably not worth approaching someone who’s mainly a CEO blogger. I look forward to being shot down on this, but would first refer readers to my esteemed colleague’s notes on how to approach a blogger.

And it will help bring new people into the game. Currently clients and colleagues can be split easily into two groups, those who get it, and those who don’t¹. We need a way to talk to the second group so that they can get it. A simpler, logical categorization is one of the things we need to help us do that.

How many types are we looking for?

Obviously it’s possible to have as many types of blogger as there are bloggers; the interesting thing is to try and keep this typology as terse as possible. Katy & Co.’s valiant attempt to categorize different types of blog makes great linkbait, but we should avoid the unnecessary proliferation of terms. Seven (or ideally four) types of blog would make for something memorable and easy to use.

What we’re going to be doing

At the moment, this is an unfunded project, but we’re not going to let that get in our way. We’re not looking to create an industry-standard after all, just something that is useful and usable by us and whoever wants to share it.

We’re going to collect a little qual data from as many bloggers as would like to take part. We’ve created a short test survey (five questions, with a total of 33 statements at this stage) which we would like you to answer. We’ll also use our own tools to collect a little quant data from your blog (typically we look at “recency, frequency, tenure and authority” figures.) We’ll make the (suitably anonymised) data available to whoever wants it on this blog.

Once we’ve begun to code and analyse the data to generate what look like statistically robust clusters of behaviour, we’d like to start interviewing a few of you to find out more about what makes you tick, and what works for your audience.

From that, I hope we can develop some kind of a simple-yet-strong typology that we can share with you. Nothing too complicated, nothing too clever. We’ll also refine our survey, remove lots of questions that don’t turn out to be particularly significant or good indicators of anything, add a carefully-tuned question here and there. Then – with any luck – we’ll go out and do the same exercise again, but this time with more respondents and more languages. Again, we’ll try to share data sets, and will certainly share findings.

How you can help

Please help us.

Answer my survey

If you have a blog of any kind, please answer this short survey. It should take less than five minutes to complete.

Share the survey link

Please share this link via twitter, email, or your blog

http://icanhaz.com/blogger_survey
For us to get started on this, we’ll need lots of data; a few hundred respondents at least. We’ll be doing all sorts of things to get people to answer the survey, but the more you can help push it along the better.

Share your thoughts

Leave a comment below, write a follow-up post on your blog, or drop me a line. I’ll be dropping in and out of this subject for the foreseeable future, and I’ll try to write digests of what we’ve been discussing.

Thank you for reading all the way to the end. I hope that you’ve now got a better idea of what I’m trying to do (and why). Furthermore, I hope that you approve, and that you’ll try to help us out.

Note 1. Of course, this is a gross over-simplification. There’s at least one more group, “those who think they get it.” How can any of us be sure that we’re not in this group?