Geek alert: if the title of this post isn’t a dead giveaway I should tell you — unless you’re interested in APIs and badly-put-together bits of code — this probably isn’t for you.

I’ve recently found myself using a service provided by Damon Clinkscale called DoesFollow. All it does is answer the simple question “does twitter user A follow twitter user B?” Apart from a frill which lets you reverse the order of your question (“does twitter user B follow twitter user A?”) that’s all it does. You can even interrogate it from the address bar like this: http://doesfollow.com/barackobama/mediaczar

While I was thinking about how useful a service this is, I was suddenly struck by a moment of clarity. A lot of the research I’ve been doing could be simplified by something like this.

Quite often I want to find out whether MPs or congressmen or PR people follow each other on Twitter.

The way that I’ve been doing this until now is

1. make a list of the people who I’m interested in researching
2. for each person on that list, grab the list of all the Twitter people whom they follow
3. process the list so that only relationships between the people on the list show up

If all I’m doing is checking to see who follows whom, then this is a horribly wasteful way of doing things. The Twitter API limits the number of calls one can make on it — so this wastage leads to things taking much longer.

If only I could cycle all the names I want to check through something like DoesFollow!

Well – it turns out that I can. And in theory it’s not much harder than using DoesFollow. The Twitter API (which is what DoesFollow uses, after all) has a method called friendship/exists. All we have to do is send Twitter the following request:

http://twitter.com/friendships/exists.xml?user_a=barackobama&user_b=mediaczar

and it will come back with the answer:

<friends>true</friends>
or
<friends>false</friends>

Kludge-y perl code

(This fabulous picture courtesy of There, I Fixed It)

So I tried to do this using Yahoo! Pipes, but there are too many nested loops. You need to do something like this:


get list of names

}


There’s no easy way to get Pipes to do this, as far as I can see (I’ll keep trying, but if someone else can help, I’d be v. grateful.)

So I’ve pulled together a badly-written perl script to do the work for me.

The script

[code lang="perl"]
#!/usr/bin/perl
# checks the Twitter API to find the friendships between a list of usernames
# this should really use the NEW API call that would let us halve the number
# of API calls
# author: Mat Morrison
# date: Friday July 10, 2009
use warnings;
use LWP::Simple;
# set up variables
# we're just using a whitespace delimited list for the moment
my @usernames = qw(kerrymg mediaczar timhoang titusbicknell);
# let's build the matrix with a hash of hashes...
# to begin with, we'll include diagonal values -
# that is -- we'll check to see whether @mediaczar follows @mediaczar
foreach $user_a(@usernames) {
foreach $user_b(@usernames) {
# we should put in a conditional clause that will check for the diagonal values
# and not bother checking whether someone is a friend of themselves...
$url = 'http://twitter.com/friendships/exists.xml?user_a='
.$user_a
.'&user_b='
.$user_b;
# get XML file from Twitter -- it's an astonishingly simple XML file that reads
# true
# or
# false
# so we don't need to do much with it...
$follows = get $url;
die 'Can\'t get $url' unless defined $follows;
# strip the tags - I'm using a generic "HTML stripping" regex
$follows =~ s/<(.|\n)+?>//g;
# we should probably convert "true" values to 1 and "false" values to zero or blank
# now let's push data into the matrix
$matrix{$user_a}{$user_b} = $follows
}
}
# spit out the data as a tab-delimited table
# print the top line first
for $user_b ( keys %matrix ) {
print "\t$user_b";
}
# now print the values
# they're all neatly arranged in the matrix so we
# can just print them out sequentially
for $user_a ( keys %matrix ) {
print "\n$source";
for $follows ( keys %{ $matrix{$user_a} } ) {
print "\t$matrix{$user_a}{$follows} ";
}
}
print "\n";
[/code]

Where next?

Most of my thinking is included above in the code comments. An obvious mistake I’m making is checking to see whether, say, @mediaczar follows @mediaczar. That wastes n API calls per search. But a more serious mistake is not to be using the new friendships/show method. Because it tells you whether user A follows user B and whether user B follows user A at the same time, it would save me lots of API calls. How many lots? Well take a look at this.

This is what I’m doing at the moment — checking each and every cell in the matrix:

This is what I’d be doing if I removed the diagonals:

And this is what I’d be doing if I used the newer API call:

I had to look up the formula for working this out without colouring in little boxes. With a little tweaking (to prevent the diagonals from creeping back in), here it is:

((n-1)^2)+n-1)/2

So — for a list of congress people (159 on twitter as at Tuesday July 14, 2009) that’d be ((156-1)^2-1+156)/2 = 12,090 API calls. Which is still a lot and will require some careful throttling, but (literally) not half as many as the 156^2 = 24,336 API calls that I’d need to run it as the script currently stands.

So – back to the drawing board for a while. I really can’t work out a programmatic way of doing this. Hmph.

TwitterCounter, the service that tells you how many people followed a given Twitter user on a given date (among other things) has an API – so I thought I’d take a look at it to see whether I could create a quick automated table of rankings.

Here’s the simplest way to query the API:

[code]

http://twittercounter.com/api/?username=mediaczar&output=xml

[/code]

Just cut and paste that into the address bar of your browser for example. Fairly simple. Change the username and you’ll get the data for a different user. Here’s what you get back from the API — an XML file with lots of rich meaty data:

<?xml version="1.0" encoding="UTF-8"?><twittercounter> <user_id>8206</user_id> <user_name>mediaczar</user_name> <followers_current>658</followers_current> <date_updated>2009-01-24</date_updated> <url>http://mediaczar.com</url> <avatar>69490155/q506317361_7156_normal.jpg</avatar> <follow_days>215</follow_days> <started_followers>169</started_followers> <growth_since>489</growth_since> <average_growth>2</average_growth> <tomorrow>660</tomorrow> <next_month>718</next_month> <followers_yesterday>652</followers_yesterday> <rank>16681</rank> <followers_2w_ago>169</followers_2w_ago> <growth_since_2w>489</growth_since_2w> <average_growth_2w>35</average_growth_2w> <tomorrow_2w>693</tomorrow_2w> <next_month_2w>1708</next_month_2w> <followersperdate> <date2009-01-10>480</date2009-01-10> <date2009-01-11>480</date2009-01-11> <date2009-01-12>481</date2009-01-12> <date2009-01-13>486</date2009-01-13> <date2009-01-14>498</date2009-01-14> <date2009-01-15>498</date2009-01-15> <date2009-01-16>514</date2009-01-16> <date2009-01-17>534</date2009-01-17> <date2009-01-20>594</date2009-01-20> <date2009-01-21>594</date2009-01-21> <date2009-01-22>616</date2009-01-22> <date2009-01-23>652</date2009-01-23> <date2009-01-24>658</date2009-01-24> </followersperdate> </twittercounter>

First attempt: Google Spreadsheet’s importXML function

As part of this project, I was particularly interested to explore Google Spreadsheet’s useful importXML function. Google lets you pull data out of an XML document anywhere on the web and put it into a spreadsheet cell.

There’s a pretty average bit of support documentation from Google on this function and a peculiarly hard-to-understand description of the XPATH reference that you can read if you want to, but all you really need to know is that you can address any item in the document using a double slash like so: //item_name.

So if there’s an item called followers_yesterday in the document above and you want to access it

[code lang="xml"]652
[/code]

I can access it like this

[code]=ImportXML("http://twittercounter.com/api/?output=xml&username=mediaczar","//followers_yesterday")[/code]

So with not much work, I can create a Google spreadsheet that will do this for a list of twitter usernames

Wahey! That’s great! But what’s this?

I’ve got more than 200 people on the list I want to rank. Foiled by this apparent petty-mindedness on Google’s part, I decided to try again, this time using Yahoo! Pipes.

Second Attempt: Yahoo! Pipes

First of all I make a simple pipe to build the URL and make the API call for a single user (I’ve learned to follow this modular approach from looking at pipes built by the Open University’s Tony Hirst.)

Now all I need to do is plug this into another pipe. I’m pulling the data from a Google spreadsheet list of Twitter usernames published as a text file. Whenever I update the spreadsheet, of course, the text file is updated dynamically.

The first FetchCSV module pulls that text file in and passes the data to the next module as “username.” The Loop module loops through each username on the list and passes it through the TwitterCounter API Call module I just built. All the XML data from the call will be passed on as “item.data” Now all I need to do is select the bits I want and format the data for output. Should be simple.

The most important thing to know about pipes is that we’re usually working with RSS (there are ways around this.) That limits us to only a few fieldnames; generally title, link and description.

Here I’m using the Rename module to set

username

as the title, and

item.data.followers_yesterday

as the description. Let’s save the pipe and test it out.

Success! Now I can take the RSS feed for this and (using Googles importRSS function) drop it back into another spreadsheet.

Only right now, I can’t work out how to to do that for more than twenty items. So I’ve actually got fewer results than I had before.

I’ll continue plugging away at this and post any solution I come to here – but in the meantime if you’re reading this and know how to fix my issue with Google Spreadsheets, please help. There’s some hope held out by this article on Sphinn which I’ll struggle along with.

In the meantime, I can output the results as a CSV so that’s OK.

Yahoo! Pipe for automating Marshall Kirkpatrick's Social Media CheatSheet process

any way to ramp up your knowledge of these fields, fast, other than the “Google and wander” method?

He then outlines an almost perfect example of how to use social media to do this.

You should read his article before reading any further. It’s short and punchy and won’t take much time.

Read it? Good. Now you may have noticed in the comments section that the first commenter doubts that you can:

find one baker or candlestick maker that will go through all of that.

So I thought I’d see if I can automate the process. The short answer is that I can and I can’t. I can’t yet automate one or two really important bits and pieces, notably:

1. ranking delicious bookmarks by popularity, not recency
2. human editorial selection of bookmarks

Perhaps someone could help me with this.
But otherwise, I’ve published this Yahoo! Pipe, Automating Marshall Kirkpatrick’s Social Media Cheatsheet Process which automates 90% of the process, and may make it easier for the bakers and candlestickmakers.

All comments and — more importantly — suggestions and improvements gratefully received.

Monday, 12 Jan 2009 00:27: I’ve just added a bit to the pipe to list posts in descending order according to PostRank. Don’t know if this is useful

Yahoo! Pipe to pull Technorati API data for multiple blogs

The problem with the perl-based approach is that it’s a little inaccessible to people who aren’t comfortable using a command line environment. So I really wanted to make something that more people would feel comfortable using, and perhaps play around with.

So, with some help and kind words from Bob Briski, one of whose pipes I’d stumbled across and bookmarked during my research for this project, I decided to finish off the pipe and publish it so that others could use it, or (better still) improve upon it.

What does it do?

The pipe first pulls a list of blog URLs from a Google spreadsheet, then checks Technorati to get the “inbound blogs” count for each blog, then outputs the blog title, link and count to an RSS feed.

Technorati Authority (a useful metric when ranking a list of blogs) is calculated as “the unique blogs linking to a blog over the past six months” — in a way it’s a measure of how many readers may be influenced by a blogger at one remove.

Before you start

You’ll need:

1. An API key from Technorati
2. A Yahoo! account
3. A Google account

I’ve done all the rest, so you could simply clone my pipe, which you’ll find here.

Publishing a Google spreadsheet as a CSV source

In my first pass at this challenge, I used a simple text list of URLs stored on this server. That’s pretty easy if you have (a) a server, (b) a text editor, and (c) some way — like ftp or ssh — of getting the two to talk to each other. But lots of people don’t have servers these days, and still more probably have no idea what I mean by (c), so I decided to follow a new approach before writing this article. Everyone, I thought, must have access to Google Docs.

So here’s my test list as published on Google:

Now all I have to do is Share > Publish as a web page, then click More publishing options and select ‘CSV’ or ‘TXT’ (1) as the publishing option to create a simple text list.

publishing google spreadsheet as a csv

You can see the CSV file here. You need to copy (or make a note of) the link (2) because we use it as the seed for our pipe.

One of the nice things about using Google Docs is that it’s easy to edit, view and share the list. For whatever reason, you may not want to share your list of blogs so if you’d like the original version with the server-based file which is more ‘private’, it’s still available.

Setting up the Pipe

The first thing to do is to build a URL for each request I’m going to send to the API. Technorati takes requests in the following format:
http://api.technorati.com/bloginfo?key=[apikey]&url=[blog url]

setting up the pipe

The pipe takes two variables that will need to be customized for your own version. One, rather obviously, is the link URL for the Google CSV that you copied or made a note of above (you did make a note of it, didn’t you?). The other is your Technorati API key for which you’ll probably need to sign up. It’s free, but is limited to 500 calls to Technorati’s database per day.

I’ve plugged the link into the Fetch CSV module at top left, and the API key into the Private String module. I use this module so that I can protect my API key — no-one else should be able to see it now.

The URL Builder module placed within a Loop module does this. I’ve hooked the API Key from the Private String into the parameter named ‘key’, and the output of the Fetch CSV module into the ‘url’ parameter.

Getting the data from Technorati

getting the XML by calling technorati's API

Now it’s simply a matter of looping through the requests that I built during the set-up stage above, and sending each one to Technorati so that I can fetch the data. Technorati responds with an XML file that looks like this:

[code lang="xml"]





    
        http://www.mediaczar.com/blog
             
                Mediaczar
                http://mediaczar.com/blog
                
                http://feeds.feedburner.com/mediaczar/posts
                14
                21
                2009-01-06 12:45:23 GMT
                422833
                
                    
                        mediaczar
                        Mat Morrison
                        Mat Morrison is a digital marketing and
                        communications strategist with over a decade's
                        experience in online advertising, eCRM, and social
                        media.
                        http://technorati.com/people/technorati/mediaczar
http://static.technorati.com/progimages/photo.jpg?
                        uid=139758
                    
                
             
                14
                21
    


[/code]

I can choose which bit of the XML file I want to receive by setting the ‘path to item list’ parameter in the Fetch Data module. You’ll see that I’ve set it to ‘document.result’ — compare that to the XML example above and I think you’ll see how this works: I’ve removed the wrapper information from the file and gone straight to the meat.

Renaming the fields to suit RSS output

renaming the fields to suit RSS output

Checking the output of the Pipe

Now I’ve saved the pipe, I can run it to see if it’s working.

checking the output of the pipe

Importing the RSS feed back into the Google Spreadsheet

Google Spreadsheets has some useful functions that you don’t find in Excel; and which are more geared towards the web. The only one that I’ve really played with is the importFeed function which — as you might expect — imports RSS feeds into your spreadsheet. So here, we paste the RSS link from the pipe into the sheet…

google spreadsheet showing output from importFeed function

Sometimes (for instance when I’m doing the research for the blogger typology) you need to get a whole load of Technorati data for a whole load of blogs.

This research can (of course) be done by hand. And (of course) for a long list of blogs this would take a great deal of time. Handily, Technorati provides developers with an API that lets you automate those queries. An API (for those of you who don’t know) is an Application Programming Interface – a toolkit provided by a service or application (in this case by Technorati) that lets other computer applications ask it questions and use the answers for their own purposes. It may be helpful to think of APIs as being like the knobs on top of a Lego brick that let you stick other Lego on to it without in any way changing the nature of the brick itself. On the other hand it may not be so helpful after all.

After much struggling with a Yahoo! Pipe to query the Technorati API for a list of blogs, I was forced to abandon my attempt. I would have liked to have shared that Pipe with the world (if you’re good with Yahoo! Pipes, do please take a look at it and see if you can help me!) [Tuesday January 6, 2009: Thanks to help and encouragement from Bob Briski, this now looks like it's on its way to working!]

Instead, I’ve written a perl script to do this. Perl isn’t as easy for people to use for themselves as Pipes, but if you are comfortable with a command prompt, then you’re half way there.

What this script does is take a list of blog urls, and for each item in the list queries Technorati for the following information:

1. Blog title
2. Inbound blogs (the number of unique external blogs linking to the blog over the past six months, this is also known as “Technorati Authority”)
3. Inbound links (the total number of links into the site)
4. Technorati Rank (a sort of overall score)

The script

[code lang="perl"]#!/usr/bin/perl
# use modules
use LWP::Simple;
use XML::Simple;
# set up variables
open(INFILE, $ARGV[0]) or die "Can't open list of blogs to read: $!";
$apikey='enter your Technorati API key here';
# create object
$xml = new XML::Simple;
# read each line, and make the Technorati API call
while () {
chomp;
&callTechnoratiAPI;
}
sub callTechnoratiAPI {
$url = 'http://api.technorati.com/bloginfo?format=xml&key='.$apikey.'&url='.$_;
# get XML file from Technorati
$content = get $url;
die "Can't get $url" unless defined $content;
# read XML file
$data = $xml->XMLin($content);
# access XML data and print TSV to screen
# (you can fiddle with this as much
# or as little as you like)
print ""$data->{document}->{result}->{weblog}->{name}"t";
print "$data->{document}->{result}->{url}t";
print "$data->{document}->{result}->{weblog}->{inboundblogs}t";
print "$data->{document}->{result}->{weblog}->{inboundlinks}t";
print "$data->{document}->{result}->{weblog}->{rank}n";
}[/code]

How to use it

I can’t give you any real advice on how to run perl on your system. If you want to play around with it, Macs come with perl already installed, Windows users should download and install the free ActivePerl. But you’ll need to install the perl bundle XML::Simple, and I don’t know where to begin telling you how to do that if you don’t already know how perl and CPAN work. You see why I wanted to use Yahoo! Pipes?

If all of that doesn’t bother you, you’ll also need to sign up for a Technorati account (if you’re into this sort of thing, you should already have an account), and get your free API key. This key will let you make 500 queries in a 24-hour period, so you’ll need to plan how you use it.

The script as it’s listed above outputs tab-separated values to screen like this:
matm% ./parse_technorati.pl bloglist.txt
"Chris Gilmour's Diary Vol. 14" http://www.illandancient.blogspot.com 6 10 861604
"The Red Rocket: Technology, PR and social media marketing" www.theredrocket.co.uk 15 29 397843
"Going Underground's Blog" http://london-underground.blogspot.com 254 467 13332

The blog’s title and url are followed in order by the inbound blogs (authority) count, the inbound links count, and the Technorati rank.

I use tab-separated values because that makes it simple to cut-and-paste directly into Excel or Google Spreadsheets for further analysis.

Known bugs

Right now, the script occasionally throws out something like this:

matm% ./parse_technorati.pl bloglist.txt
"Lytham Villa" http://lythamvilla.blogspot.com/ HASH(0x8ff7a0) HASH(0x8ff7f4) 4978471
"KickTime || A Driftless Regional Webspace" http://kicktime.org HASH(0x908e0c) HASH(0x908db8) 1951828

I’ll work on this, but if anyone can point me in the right direction, I’ll be most grateful.

Since writing this post, three simpler, better ways of solving the problem have been submitted in the comments section. Feel free to read this post, but look to the comments for the solution!

If you already know about conditional formatting and navigated here via Google, please jump straight to the hack. If not, I hope the following introduction is useful. You might also like to check out the WikiHow introduction to conditional formatting in Excel. This post is actually concerned with an interesting hack that lets you reference the value of a cell itself when setting up formula-based conditional formatting rule.

Conditional Formatting

Excel’s conditional formatting feature is a boon to heavy spreadsheet users like me. It is a flexible and powerful tool that (among other things) lets me highlight data according to a set of rules so that I can easily spot the interesting bits in what would otherwise be an almost impossibly dense and meaningless cloud of numbers. Here’s an example; a table of the correlations between 32 different statements (taken from some ongoing work looking at a simple blogger typology.)

I’m sure you’d agree that your eyes would go squiggly if you looked for interesting data points in that mess (and that’s the processed table!) Instead, I use Excel’s conditional formatting to look through it for me. Here’s the same table with interesting points coloured red, orange, and yellow (in decreasing order of interest.)

Useful, eh? However, there are some very tight restrictions. Among them are:

1. The rule must evaluate to TRUE or FALSE. Either the cell is more than x, for example, or it isn’t.
2. Like wishes, you can only apply three conditional-formatting rules to any cell.
3. Once Excel has evaluated a condition as TRUE it stops processing any further rules. It evaluates Condition 1, and if that’s FALSE will move on to Condition 2 and if that’s TRUE will never evaluate Condition 3 (however interesting Condition 3 may be.)

The Problem

Here’s a much reduced version of the table of correlations example I used above. I still wouldn’t want to look through a table like this very often, but it’s easier to make my point when you can read the numbers.

I did this by applying the following conditional formatting rules.

For the sake of this analysis, any correlation of 0.5 or over is seen to be “possibly interesting”, 0.6 or over “interesting”, and 0.7 or over “really quite interesting indeed.” I should probably come up with a better scale before I share this stuff in future.) But as it happens, correlations can go two ways, positive (“people who agree with this statement tend to agree with that statement”) or negative (“people who agree with this statement tend to disagree with that statement”.) So I also want to highlight any correlation lower than -0.5, -0.6, or -0.7. You can see from the following screen grab that this isn’t working.

But I only have three rules. What am I to do?

As it happens, Excel lets one specify a formula as a rule rather than simply a cell value. So, for example, I could check to see if:

=OR(A1<= 0.5, A1>=0.5)

But I’d have to code the conditions for each separate cell in the table. I am notoriously lazy and therefore unlikely to do something like this, particularly for the 32×32 matrix in the first example.

What I need to do is refer to ‘this cell’ in the formula. Now in normal Excel use this would lead to recursion, a circular reference, and therefore an error. For this logical reason, Excel’s developers never saw the need to let a cell refer to itself. But that’s exactly what I need to do!

The Excel Conditional Formatting Hack

After much faffing, I finally worked out a way around the problem.

=ABS(INDIRECT("R"&ROW()&"C"&COLUMN(),FALSE))>=0.5

Which you will see works very nicely.

Let’s look at how it works (I’ve put the bit that matters in bold face.)

=ABS(INDIRECT("R"&ROW()&"C"&COLUMN(),FALSE))>=0.5

That probably doesn’t help all that much. On the assumption that for many of you it’s the first time you’ve seen these functions, let’s pull it apart a bit.

INDIRECT is a way of linking to a cell using a “reference text” – that is, a text string that looks like a cell reference (“A1″ for example, or “R1C1″ which is the same thing – pointing to Row 1 Column 1)

We actually employ it like this:
INDIRECT(ref_text,a1)
Where ref_text is the “reference text” and a1 is a TRUE or FALSE value that tells Excel whether to expect an A1-style reference (TRUE or omitted) or an R1C1-style reference (FALSE)

ROW() tells Excel to insert the value for the current Row, and…
COLUMN() tells Excel to insert the value for the current Column, so:

INDIRECT("R"&ROW()&"C"&COLUMN(),FALSE) gives us the R1C1-style reference to the current cell. This cell! Hooray!

These didn’t work…

Incidentally, I also tried these which I think should work but don’t. I include them for completeness’s sake.

=ADDRESS(ROW(),COLUMN())

=INDIRECT(ADDRESS(ROW(),COLUMN()),FALSE)>=0.5

=INDIRECT(TEXT(ADDRESS(ROW(),COLUMN())),FALSE)>=0.5