There’s a school of thought that says that what’s important in social media is to attempt to create debate, not consensus.

Peter Hay from PR Week and I appear to have been rather successful in this. This morning, PR Week published an article, Twitter has suddenly exploded. Almost immediately, Twitter (or at least our particular neighbourhood of Twitter) suddenly exploded.

One or two people were rather scathing: suggesting that the stats demonstrated that Peter and I didn’t understand the “essence of Twitter” or that they were “obviously flawed”, or that we had “redefined shallow”.

Indeed (horror of horrors) some people even went so far to suggest that Porter Novelli had ginned up the results to put us at the top. In fact, in PR week’s list, we came second. But no doubt this was a Machiavellian ploy — it’s details like those, Pooh Bah would say, that “give artistic verisimilitude to an otherwise bald and unconvincing narrative.”

I joke, but I can completely understand people’s strong feelings about this; PR Week was torn between a desire to cover our approach (and give credit where appropriate) and a need to keep the article readable and relevant to the greater proportion of their readers.

I’d like to share our methodology with you all so that you can repeat our experiments, should you so wish. After that, I’ll talk about the methodology that we were originally going to follow,

Tomorrow (once it’s had a chance to blow over), I’ll post some quick thoughts on the whole storm-in-a-Tweetcup thing.

Methodology

We used Michael Litman’s (@litmanlive) list of UK Media Tweeple. This was based on original work by Stephen Davies (@stedavies) but has been wikified so that agencies can (should they so choose) keep their information up to date.

Lots of people on the list were pretty borderline — there are in-house teams and vendors there, as well as agencies with a significantly broader remit than simply “PR”. I am a relative newcomer to the world of PR, and was more than happy to let PR Week define who is PR and who isn’t, but we erred on the generous side. We are Social, for example, made the cut to be on the research list.

Had we had the time, I’d have sent a note out over Twitter asking everyone to update their entries. Time, however was not on our side, and I didn’t even get around to hinting at what I was doing until the evening of the 23rd.

By then though it was already clear that I had a large job on my hands; there were almost 350 people on the list. On the whole, the UK PR community should be proud of how quickly it has reacted to the whole “Twitter thing”.

I took the list, published it as a Google Spreadsheet and — using a Yahoo! Pipe that I adapted for the purpose, queried the Twitter API for the summary data on each account on that list.

Twitter gives you all sorts of interesting information, but what we were grabbed were the following:

• Date joined Twitter
• Number of Friends
• Number of Followers, and
• Number of Updates.

That allowed us to create this spreadsheet, from which the stats mentioned in the PR Week article were taken.

Again, Porter Novelli took no part in the editorial decisions (although they seem pretty straightforward.) You will recall that Peter and Gemma were writing for a general readership, not for the Twitterverse!

Methodology we’d like to have used

Those of you who’ve read my blog before will know that my real interest in Twitter is more complex than the previous methodology would suggest. When Peter and I first discussed the exercise on Monday we had been hoping to do something more along the lines of the network analysis that we’ve been fiddling with at Porter Novelli.

Here are some points to bear in mind.

First of all, not all followers are created equal. If I have only ten followers, but they each have a thousand followers, that may mean I have more opportunity-to-influence than if I had a hundred followers with only ten followers each.

More to the point, the fewer people those ten people follow themselves, the more influence I wield within their networks (if I am one of only ten people they follow between them, I will have greater share-of-voice than if I am merely one of ten thousand.)

Secondly, the followers whom I don’t share with the rest of the network count for more than those who follow several (or many) of my peers. The more “exclusive” my follower-base, the greater my control over on the flow of information within the overall network, and the greater my value to the network.

I’ve been doing some work looking at unduplicated reach among twitter networks. For example, looking at Porter Novelli’s own global Twitter footprint, it was interesting to see how many of our contacts were duplicated.

So what Peter and I really wanted to do was to use some of these techniques on the PR Week data set. For those of you with a mathematical (or social network analytical) bent, we were going to run some eigenvector shizzle on the whole bizzle. Oh — and look at unduplicated reach for the various companies on the list.

What went wrong?

It was always an ambitious project. The 344 people who were under analysis had a fairly daunting 95K followers between them. The Twitter API lets you make 100 requests an hour, and each request returns data on up to 100 followers. Even if we were to assume that everyone had followers in nice tidy multiples of 100 (they don’t) then it would have taken 9.5 hours to download the data using one Twitter login.

The trick of course, is to use more than one login. Tim Hoang (@timhoang) and I quickly registered 50 temporary accounts to power the API requests. Twitter’s terms have historically been quite relaxed about this sort of thing, and we’ve always been very careful to try and stay within the spirit of those terms.

But.

Twitter has been hit lately by a bunch of bad things (like spam bots and pyramid schemes), and they’re tightening up their defenses. This past weekend, they’ve tightened up a lot, and things that used to be fine just aren’t.

We managed to collect information on only around 60K followers out of the 95K. This was too large a margin of error to correct (although we made several attempts to do so).

So — we had to abandon our grand plans, and revert to the simple counts approach (as detailed above.) This won’t stop us trying to improve our processes, but we’ll need to talk to Twitter about that.

Some thoughts

Kate Hartley from Carrot Communications (who sits with me on the PRCA’s Digital Working Group) joked that it’s strange how PR people create research-for-news-stories for their own clients on a daily basis, but are miffed when their own techniques are used against them. At one level, I agree with her — I think that some people are probably disappointed that they aren’t the ones with their names on the research.

But there’s more to worry about than that. Here are my thoughts.

1. For God’s sake get over yourselves! We’re talking about Twitter here, not the economy. Worry about something important, why don’t you? I still can’t get over the fact that — when a pilot managed land an airplane on a river, the story we all tell each other is “how it broke on Twitter.” What — the story’s not about a man who magically landed a f*cking plane on a f*cking river? Are we really getting this right?
2. How influential you are on Twitter is not a real thing. It doesn’t really matter how many Twitter friends you have (although I’ve now got heaps, thank you very much!) Context is everything. My boss, who runs Porter Novelli’s EMEA network and sits on our Executive Committee is on Twitter. She is more influential than I, and will continue to be, no matter how many Twitter followers I accrue.

   Twitter is just one channel through which exercise your influence. Don’t give up on your blogs, your Facebook pages, your Amazon reviews, or your Last.fm playlists or your IM friend lists, for God’s sake. But remember, it’s who you are, and your relationships that matter; your “context”, and not your “counts.”

3. The really interesting question isn’t “who are the Twitterati” or twitter influencers. I’m interested in the Twitter thing mainly because I want to see how well it reflects real life. After today, I’d probably say that it doesn’t very well, wouldn’t you?

Be warned — I may just follow this research up with some research on “how many phone numbers PR people have on their mobile phones.”

TwitterCounter, the service that tells you how many people followed a given Twitter user on a given date (among other things) has an API – so I thought I’d take a look at it to see whether I could create a quick automated table of rankings.

Here’s the simplest way to query the API:

[code]

http://twittercounter.com/api/?username=mediaczar&output=xml

[/code]

Just cut and paste that into the address bar of your browser for example. Fairly simple. Change the username and you’ll get the data for a different user. Here’s what you get back from the API — an XML file with lots of rich meaty data:

<?xml version="1.0" encoding="UTF-8"?><twittercounter> <user_id>8206</user_id> <user_name>mediaczar</user_name> <followers_current>658</followers_current> <date_updated>2009-01-24</date_updated> <url>http://mediaczar.com</url> <avatar>69490155/q506317361_7156_normal.jpg</avatar> <follow_days>215</follow_days> <started_followers>169</started_followers> <growth_since>489</growth_since> <average_growth>2</average_growth> <tomorrow>660</tomorrow> <next_month>718</next_month> <followers_yesterday>652</followers_yesterday> <rank>16681</rank> <followers_2w_ago>169</followers_2w_ago> <growth_since_2w>489</growth_since_2w> <average_growth_2w>35</average_growth_2w> <tomorrow_2w>693</tomorrow_2w> <next_month_2w>1708</next_month_2w> <followersperdate> <date2009-01-10>480</date2009-01-10> <date2009-01-11>480</date2009-01-11> <date2009-01-12>481</date2009-01-12> <date2009-01-13>486</date2009-01-13> <date2009-01-14>498</date2009-01-14> <date2009-01-15>498</date2009-01-15> <date2009-01-16>514</date2009-01-16> <date2009-01-17>534</date2009-01-17> <date2009-01-20>594</date2009-01-20> <date2009-01-21>594</date2009-01-21> <date2009-01-22>616</date2009-01-22> <date2009-01-23>652</date2009-01-23> <date2009-01-24>658</date2009-01-24> </followersperdate> </twittercounter>

First attempt: Google Spreadsheet’s importXML function

As part of this project, I was particularly interested to explore Google Spreadsheet’s useful importXML function. Google lets you pull data out of an XML document anywhere on the web and put it into a spreadsheet cell.

There’s a pretty average bit of support documentation from Google on this function and a peculiarly hard-to-understand description of the XPATH reference that you can read if you want to, but all you really need to know is that you can address any item in the document using a double slash like so: //item_name.

So if there’s an item called followers_yesterday in the document above and you want to access it

[code lang="xml"]652
[/code]

I can access it like this

[code]=ImportXML("http://twittercounter.com/api/?output=xml&username=mediaczar","//followers_yesterday")[/code]

So with not much work, I can create a Google spreadsheet that will do this for a list of twitter usernames

Wahey! That’s great! But what’s this?

I’ve got more than 200 people on the list I want to rank. Foiled by this apparent petty-mindedness on Google’s part, I decided to try again, this time using Yahoo! Pipes.

Second Attempt: Yahoo! Pipes

First of all I make a simple pipe to build the URL and make the API call for a single user (I’ve learned to follow this modular approach from looking at pipes built by the Open University’s Tony Hirst.)

Now all I need to do is plug this into another pipe. I’m pulling the data from a Google spreadsheet list of Twitter usernames published as a text file. Whenever I update the spreadsheet, of course, the text file is updated dynamically.

The first FetchCSV module pulls that text file in and passes the data to the next module as “username.” The Loop module loops through each username on the list and passes it through the TwitterCounter API Call module I just built. All the XML data from the call will be passed on as “item.data” Now all I need to do is select the bits I want and format the data for output. Should be simple.

The most important thing to know about pipes is that we’re usually working with RSS (there are ways around this.) That limits us to only a few fieldnames; generally title, link and description.

Here I’m using the Rename module to set

username

as the title, and

item.data.followers_yesterday

as the description. Let’s save the pipe and test it out.

Success! Now I can take the RSS feed for this and (using Googles importRSS function) drop it back into another spreadsheet.

Only right now, I can’t work out how to to do that for more than twenty items. So I’ve actually got fewer results than I had before.

I’ll continue plugging away at this and post any solution I come to here – but in the meantime if you’re reading this and know how to fix my issue with Google Spreadsheets, please help. There’s some hope held out by this article on Sphinn which I’ll struggle along with.

In the meantime, I can output the results as a CSV so that’s OK.

Yahoo! Pipe for automating Marshall Kirkpatrick's Social Media CheatSheet process

any way to ramp up your knowledge of these fields, fast, other than the “Google and wander” method?

He then outlines an almost perfect example of how to use social media to do this.

You should read his article before reading any further. It’s short and punchy and won’t take much time.

Read it? Good. Now you may have noticed in the comments section that the first commenter doubts that you can:

find one baker or candlestick maker that will go through all of that.

So I thought I’d see if I can automate the process. The short answer is that I can and I can’t. I can’t yet automate one or two really important bits and pieces, notably:

1. ranking delicious bookmarks by popularity, not recency
2. human editorial selection of bookmarks

Perhaps someone could help me with this.
But otherwise, I’ve published this Yahoo! Pipe, Automating Marshall Kirkpatrick’s Social Media Cheatsheet Process which automates 90% of the process, and may make it easier for the bakers and candlestickmakers.

All comments and — more importantly — suggestions and improvements gratefully received.

Monday, 12 Jan 2009 00:27: I’ve just added a bit to the pipe to list posts in descending order according to PostRank. Don’t know if this is useful

Yahoo! Pipe to pull Technorati API data for multiple blogs

The problem with the perl-based approach is that it’s a little inaccessible to people who aren’t comfortable using a command line environment. So I really wanted to make something that more people would feel comfortable using, and perhaps play around with.

So, with some help and kind words from Bob Briski, one of whose pipes I’d stumbled across and bookmarked during my research for this project, I decided to finish off the pipe and publish it so that others could use it, or (better still) improve upon it.

What does it do?

The pipe first pulls a list of blog URLs from a Google spreadsheet, then checks Technorati to get the “inbound blogs” count for each blog, then outputs the blog title, link and count to an RSS feed.

Technorati Authority (a useful metric when ranking a list of blogs) is calculated as “the unique blogs linking to a blog over the past six months” — in a way it’s a measure of how many readers may be influenced by a blogger at one remove.

Before you start

You’ll need:

1. An API key from Technorati
2. A Yahoo! account
3. A Google account

I’ve done all the rest, so you could simply clone my pipe, which you’ll find here.

Publishing a Google spreadsheet as a CSV source

In my first pass at this challenge, I used a simple text list of URLs stored on this server. That’s pretty easy if you have (a) a server, (b) a text editor, and (c) some way — like ftp or ssh — of getting the two to talk to each other. But lots of people don’t have servers these days, and still more probably have no idea what I mean by (c), so I decided to follow a new approach before writing this article. Everyone, I thought, must have access to Google Docs.

So here’s my test list as published on Google:

Now all I have to do is Share > Publish as a web page, then click More publishing options and select ‘CSV’ or ‘TXT’ (1) as the publishing option to create a simple text list.

publishing google spreadsheet as a csv

You can see the CSV file here. You need to copy (or make a note of) the link (2) because we use it as the seed for our pipe.

One of the nice things about using Google Docs is that it’s easy to edit, view and share the list. For whatever reason, you may not want to share your list of blogs so if you’d like the original version with the server-based file which is more ‘private’, it’s still available.

Setting up the Pipe

The first thing to do is to build a URL for each request I’m going to send to the API. Technorati takes requests in the following format:
http://api.technorati.com/bloginfo?key=[apikey]&url=[blog url]

setting up the pipe

The pipe takes two variables that will need to be customized for your own version. One, rather obviously, is the link URL for the Google CSV that you copied or made a note of above (you did make a note of it, didn’t you?). The other is your Technorati API key for which you’ll probably need to sign up. It’s free, but is limited to 500 calls to Technorati’s database per day.

I’ve plugged the link into the Fetch CSV module at top left, and the API key into the Private String module. I use this module so that I can protect my API key — no-one else should be able to see it now.

The URL Builder module placed within a Loop module does this. I’ve hooked the API Key from the Private String into the parameter named ‘key’, and the output of the Fetch CSV module into the ‘url’ parameter.

Getting the data from Technorati

getting the XML by calling technorati's API

Now it’s simply a matter of looping through the requests that I built during the set-up stage above, and sending each one to Technorati so that I can fetch the data. Technorati responds with an XML file that looks like this:

[code lang="xml"]





    
        http://www.mediaczar.com/blog
             
                Mediaczar
                http://mediaczar.com/blog
                
                http://feeds.feedburner.com/mediaczar/posts
                14
                21
                2009-01-06 12:45:23 GMT
                422833
                
                    
                        mediaczar
                        Mat Morrison
                        Mat Morrison is a digital marketing and
                        communications strategist with over a decade's
                        experience in online advertising, eCRM, and social
                        media.
                        http://technorati.com/people/technorati/mediaczar
http://static.technorati.com/progimages/photo.jpg?
                        uid=139758
                    
                
             
                14
                21
    


[/code]

I can choose which bit of the XML file I want to receive by setting the ‘path to item list’ parameter in the Fetch Data module. You’ll see that I’ve set it to ‘document.result’ — compare that to the XML example above and I think you’ll see how this works: I’ve removed the wrapper information from the file and gone straight to the meat.

Renaming the fields to suit RSS output

renaming the fields to suit RSS output

Checking the output of the Pipe

Now I’ve saved the pipe, I can run it to see if it’s working.

checking the output of the pipe

Importing the RSS feed back into the Google Spreadsheet

Google Spreadsheets has some useful functions that you don’t find in Excel; and which are more geared towards the web. The only one that I’ve really played with is the importFeed function which — as you might expect — imports RSS feeds into your spreadsheet. So here, we paste the RSS link from the pipe into the sheet…

google spreadsheet showing output from importFeed function