Geek alert: if the title of this post isn’t a dead giveaway I should tell you — unless you’re interested in APIs and badly-put-together bits of code — this probably isn’t for you.

I’ve recently found myself using a service provided by Damon Clinkscale called DoesFollow. All it does is answer the simple question “does twitter user A follow twitter user B?” Apart from a frill which lets you reverse the order of your question (“does twitter user B follow twitter user A?”) that’s all it does. You can even interrogate it from the address bar like this: http://doesfollow.com/barackobama/mediaczar

While I was thinking about how useful a service this is, I was suddenly struck by a moment of clarity. A lot of the research I’ve been doing could be simplified by something like this.

Quite often I want to find out whether MPs or congressmen or PR people follow each other on Twitter.

The way that I’ve been doing this until now is

1. make a list of the people who I’m interested in researching
2. for each person on that list, grab the list of all the Twitter people whom they follow
3. process the list so that only relationships between the people on the list show up

If all I’m doing is checking to see who follows whom, then this is a horribly wasteful way of doing things. The Twitter API limits the number of calls one can make on it — so this wastage leads to things taking much longer.

If only I could cycle all the names I want to check through something like DoesFollow!

Well – it turns out that I can. And in theory it’s not much harder than using DoesFollow. The Twitter API (which is what DoesFollow uses, after all) has a method called friendship/exists. All we have to do is send Twitter the following request:

http://twitter.com/friendships/exists.xml?user_a=barackobama&user_b=mediaczar

and it will come back with the answer:

<friends>true</friends>
or
<friends>false</friends>

Kludge-y perl code

(This fabulous picture courtesy of There, I Fixed It)

So I tried to do this using Yahoo! Pipes, but there are too many nested loops. You need to do something like this:


get list of names

}


There’s no easy way to get Pipes to do this, as far as I can see (I’ll keep trying, but if someone else can help, I’d be v. grateful.)

So I’ve pulled together a badly-written perl script to do the work for me.

The script

[code lang="perl"]
#!/usr/bin/perl
# checks the Twitter API to find the friendships between a list of usernames
# this should really use the NEW API call that would let us halve the number
# of API calls
# author: Mat Morrison
# date: Friday July 10, 2009
use warnings;
use LWP::Simple;
# set up variables
# we're just using a whitespace delimited list for the moment
my @usernames = qw(kerrymg mediaczar timhoang titusbicknell);
# let's build the matrix with a hash of hashes...
# to begin with, we'll include diagonal values -
# that is -- we'll check to see whether @mediaczar follows @mediaczar
foreach $user_a(@usernames) {
foreach $user_b(@usernames) {
# we should put in a conditional clause that will check for the diagonal values
# and not bother checking whether someone is a friend of themselves...
$url = 'http://twitter.com/friendships/exists.xml?user_a='
.$user_a
.'&user_b='
.$user_b;
# get XML file from Twitter -- it's an astonishingly simple XML file that reads
# true
# or
# false
# so we don't need to do much with it...
$follows = get $url;
die 'Can\'t get $url' unless defined $follows;
# strip the tags - I'm using a generic "HTML stripping" regex
$follows =~ s/<(.|\n)+?>//g;
# we should probably convert "true" values to 1 and "false" values to zero or blank
# now let's push data into the matrix
$matrix{$user_a}{$user_b} = $follows
}
}
# spit out the data as a tab-delimited table
# print the top line first
for $user_b ( keys %matrix ) {
print "\t$user_b";
}
# now print the values
# they're all neatly arranged in the matrix so we
# can just print them out sequentially
for $user_a ( keys %matrix ) {
print "\n$source";
for $follows ( keys %{ $matrix{$user_a} } ) {
print "\t$matrix{$user_a}{$follows} ";
}
}
print "\n";
[/code]

Where next?

Most of my thinking is included above in the code comments. An obvious mistake I’m making is checking to see whether, say, @mediaczar follows @mediaczar. That wastes n API calls per search. But a more serious mistake is not to be using the new friendships/show method. Because it tells you whether user A follows user B and whether user B follows user A at the same time, it would save me lots of API calls. How many lots? Well take a look at this.

This is what I’m doing at the moment — checking each and every cell in the matrix:

This is what I’d be doing if I removed the diagonals:

And this is what I’d be doing if I used the newer API call:

I had to look up the formula for working this out without colouring in little boxes. With a little tweaking (to prevent the diagonals from creeping back in), here it is:

((n-1)^2)+n-1)/2

So — for a list of congress people (159 on twitter as at Tuesday July 14, 2009) that’d be ((156-1)^2-1+156)/2 = 12,090 API calls. Which is still a lot and will require some careful throttling, but (literally) not half as many as the 156^2 = 24,336 API calls that I’d need to run it as the script currently stands.

So – back to the drawing board for a while. I really can’t work out a programmatic way of doing this. Hmph.

Here’s an interesting ethical question: is it OK to ask employees to share company and client news through their personal social networks?

Here’s a hypothetical example. An agency has just launched a new ad campaign and posted the TV spot on YouTube. Is it OK to send an all-hands email asking people to share the link on Twitter and Facebook?

Let’s take it a little further. Is it OK to ask them to sign into YouTube using their personal accounts, and rate the video? It seems harmless enough, doesn’t it? You’re not telling them how they should rate it, after all.

But what if you asked them to leave comments? Any normal agency or client side social media policy will tell them that they have to disclose their relationship with the makers of the video. And you wouldn’t really want a whole bunch of comments that start “Hi, I work for the agency that made this ad and I think it’s really great,” would you? What makes the two things different?

Digging for victory

OK. Another hypothetical example. The social news site Digg is a huge source of traffic for most news websites. The Telegraph, for example, gets around 75K visits a day from the service. That’s an awful lot of traffic.

The thing is, with Digg, you really want to make the front page if you want the big traffic. Around 10K stories are submitted by users to Digg every day, and only 150 or so make it to the front page. So we’d need to be smart.

Digg is less open to being gamed than it used to be, but let’s say that a smart agency could still deconstruct Digg’s algorithm sufficiently that it can use its network of staff to improve the chances of a story (or review) making it to the front page that shows their client in a positive light.

All they’d need to do (say) is send an all-hands email that mobilised your staff to digg a particular story at the right time of day. Would this be legitimate? Digg clearly thinks not, but are they the best judge?

Ballot stuffing

Let’s say a client gets nominated for a Webby or one of the other user-voted awards out there. It’s common and acceptable practice for web services to use any means at their disposal to beg for votes.

So is it OK to send an all-hands email asking your staff to register and vote? Is it OK to ask them to use their Facebook and Twitter accounts to ask their friends and followers to vote?

Here’s what I think

I think that this is an ethical minefield, but I’ve got a couple of clear points of view that are up for discussion. First off, and from a purely business perspective:

If it’s valuable then clients should pay us to do it. If it’s not valuable we shouldn’t do it.

Ignore the ethics. When was the last time you sent an email that said, “Please share this with your friends and use billing code xxx.xx when you record it on your time sheet”? Of course, it only takes a few seconds to relay a message (fewer if you simply copy and paste the message from the all-hands email to your social media presences). What the hell – it’s all just part of the team spirit, isn’t it?

But if this service is of any value at all to the client, then we’ll see the demands on our time beginning to increase. Soon we’ll find ourselves doing several a day. Larger agencies with more staff will offer a more valuable service to clients than boutique agencies (“We have a thousand trained staff with Twitter and Facebook accounts primed to promote your campaign!”)

We could even work out some kind of ratcheted scale that said that people with more Twitter followers could bill at a higher rate; or we might start looking at unduplicated reach.

This might seem fanciful, but stories like the uSocial’s offer to game Digg’s front page for around $200 tell us that the thriving black hat market for this sort of thing is getting a little greyer.

If clients value this, then they should pay us. If we give it away as a “value added” service, then are we sure that we’ve communicated this properly so that the client understands the value we added?

How can we be certain we aren’t just undermining our digital value proposition?

Now, here’s the tricky bit where I bring ethics back into it. If we’re paying our staff to relay messages to their networks on behalf of our clients, what makes this different from spam?

Employees personal networks are their personal property

I’m one of those people who believes that the boundaries are blurring between our personal, public and professional lives. My colleague Chris Nee has posted about the presentation of self in social media, expressing a lot of how I feel rather better than I could myself, although that hasn’t stopped me from covering this elsewhere — notably in a discussion about our social media policy where I said:

I think that a good PR person is someone who manages their relationships well; who can tread the fine line between doing good work for their clients without abusing or exploiting their relationships. Who recognizes the value of their personal network, and their personal brand.

Do we have a right to ask our employees to use their personal networks on behalf of our business? I think the answer is – of course we do. We pay experienced staff more partly because of the social capital that they’ve managed to accrue in their address books. When you leave your job to go to a new agency, you’ll take your contacts with you along with all the shared experience, the favours you’ve done, and the favours you owe.

So if part of the reason we’re paying them more is because of that network, then we clearly value their network.

So why on earth would we encourage our staff to spam their personal networks?

I trust my peers. I don’t trust spam monkeys

Part of the reason for the success of social networking tools like Facebook and Twitter is that I can choose who I follow and who I don’t. I can restrict conversations to a bunch of people I trust and respect. Sure, I’ve got lots of Twitter followers with names like “Sophie1982″ and “EdelmanHR” but that just means that they hear my inconsequential blatherings, not that I have to hear theirs!

And I follow a lot of my colleagues’ Twitter streams. I’m pleased to say that these are — on the whole — full of meaty chunky content and devoid of spam.

But if we increase the spam content, what will happen? Here’s what (in the absence of evidence) I believe: their follow rates, retweet rates and mention rates will all begin to drop off. From your experience, what do you think?

Where does this leave us?

I’ve put a somewhat one-sided argument here. I’m sure I’m open to all sorts of rebuttals like, “surely, if the content is interesting/useful/entertaining then our staff will only be adding value to their networks?” and “but surely it’s up to them whether they want to pass it on to their friends?” I’ll wait for these to roll in before I start trying to address them.

But here’s what I think. I think that we’re trying to teach our colleagues to learn from managing their personal communities in order better to manage our clients’ communities. Anything that teaches them to prostitute their networks is a retrograde move.

Am I wrong?

There’s a school of thought that says that what’s important in social media is to attempt to create debate, not consensus.

Peter Hay from PR Week and I appear to have been rather successful in this. This morning, PR Week published an article, Twitter has suddenly exploded. Almost immediately, Twitter (or at least our particular neighbourhood of Twitter) suddenly exploded.

One or two people were rather scathing: suggesting that the stats demonstrated that Peter and I didn’t understand the “essence of Twitter” or that they were “obviously flawed”, or that we had “redefined shallow”.

Indeed (horror of horrors) some people even went so far to suggest that Porter Novelli had ginned up the results to put us at the top. In fact, in PR week’s list, we came second. But no doubt this was a Machiavellian ploy — it’s details like those, Pooh Bah would say, that “give artistic verisimilitude to an otherwise bald and unconvincing narrative.”

I joke, but I can completely understand people’s strong feelings about this; PR Week was torn between a desire to cover our approach (and give credit where appropriate) and a need to keep the article readable and relevant to the greater proportion of their readers.

I’d like to share our methodology with you all so that you can repeat our experiments, should you so wish. After that, I’ll talk about the methodology that we were originally going to follow,

Tomorrow (once it’s had a chance to blow over), I’ll post some quick thoughts on the whole storm-in-a-Tweetcup thing.

Methodology

We used Michael Litman’s (@litmanlive) list of UK Media Tweeple. This was based on original work by Stephen Davies (@stedavies) but has been wikified so that agencies can (should they so choose) keep their information up to date.

Lots of people on the list were pretty borderline — there are in-house teams and vendors there, as well as agencies with a significantly broader remit than simply “PR”. I am a relative newcomer to the world of PR, and was more than happy to let PR Week define who is PR and who isn’t, but we erred on the generous side. We are Social, for example, made the cut to be on the research list.

Had we had the time, I’d have sent a note out over Twitter asking everyone to update their entries. Time, however was not on our side, and I didn’t even get around to hinting at what I was doing until the evening of the 23rd.

By then though it was already clear that I had a large job on my hands; there were almost 350 people on the list. On the whole, the UK PR community should be proud of how quickly it has reacted to the whole “Twitter thing”.

I took the list, published it as a Google Spreadsheet and — using a Yahoo! Pipe that I adapted for the purpose, queried the Twitter API for the summary data on each account on that list.

Twitter gives you all sorts of interesting information, but what we were grabbed were the following:

• Date joined Twitter
• Number of Friends
• Number of Followers, and
• Number of Updates.

That allowed us to create this spreadsheet, from which the stats mentioned in the PR Week article were taken.

Again, Porter Novelli took no part in the editorial decisions (although they seem pretty straightforward.) You will recall that Peter and Gemma were writing for a general readership, not for the Twitterverse!

Methodology we’d like to have used

Those of you who’ve read my blog before will know that my real interest in Twitter is more complex than the previous methodology would suggest. When Peter and I first discussed the exercise on Monday we had been hoping to do something more along the lines of the network analysis that we’ve been fiddling with at Porter Novelli.

Here are some points to bear in mind.

First of all, not all followers are created equal. If I have only ten followers, but they each have a thousand followers, that may mean I have more opportunity-to-influence than if I had a hundred followers with only ten followers each.

More to the point, the fewer people those ten people follow themselves, the more influence I wield within their networks (if I am one of only ten people they follow between them, I will have greater share-of-voice than if I am merely one of ten thousand.)

Secondly, the followers whom I don’t share with the rest of the network count for more than those who follow several (or many) of my peers. The more “exclusive” my follower-base, the greater my control over on the flow of information within the overall network, and the greater my value to the network.

I’ve been doing some work looking at unduplicated reach among twitter networks. For example, looking at Porter Novelli’s own global Twitter footprint, it was interesting to see how many of our contacts were duplicated.

So what Peter and I really wanted to do was to use some of these techniques on the PR Week data set. For those of you with a mathematical (or social network analytical) bent, we were going to run some eigenvector shizzle on the whole bizzle. Oh — and look at unduplicated reach for the various companies on the list.

What went wrong?

It was always an ambitious project. The 344 people who were under analysis had a fairly daunting 95K followers between them. The Twitter API lets you make 100 requests an hour, and each request returns data on up to 100 followers. Even if we were to assume that everyone had followers in nice tidy multiples of 100 (they don’t) then it would have taken 9.5 hours to download the data using one Twitter login.

The trick of course, is to use more than one login. Tim Hoang (@timhoang) and I quickly registered 50 temporary accounts to power the API requests. Twitter’s terms have historically been quite relaxed about this sort of thing, and we’ve always been very careful to try and stay within the spirit of those terms.

But.

Twitter has been hit lately by a bunch of bad things (like spam bots and pyramid schemes), and they’re tightening up their defenses. This past weekend, they’ve tightened up a lot, and things that used to be fine just aren’t.

We managed to collect information on only around 60K followers out of the 95K. This was too large a margin of error to correct (although we made several attempts to do so).

So — we had to abandon our grand plans, and revert to the simple counts approach (as detailed above.) This won’t stop us trying to improve our processes, but we’ll need to talk to Twitter about that.

Some thoughts

Kate Hartley from Carrot Communications (who sits with me on the PRCA’s Digital Working Group) joked that it’s strange how PR people create research-for-news-stories for their own clients on a daily basis, but are miffed when their own techniques are used against them. At one level, I agree with her — I think that some people are probably disappointed that they aren’t the ones with their names on the research.

But there’s more to worry about than that. Here are my thoughts.

1. For God’s sake get over yourselves! We’re talking about Twitter here, not the economy. Worry about something important, why don’t you? I still can’t get over the fact that — when a pilot managed land an airplane on a river, the story we all tell each other is “how it broke on Twitter.” What — the story’s not about a man who magically landed a f*cking plane on a f*cking river? Are we really getting this right?
2. How influential you are on Twitter is not a real thing. It doesn’t really matter how many Twitter friends you have (although I’ve now got heaps, thank you very much!) Context is everything. My boss, who runs Porter Novelli’s EMEA network and sits on our Executive Committee is on Twitter. She is more influential than I, and will continue to be, no matter how many Twitter followers I accrue.

   Twitter is just one channel through which exercise your influence. Don’t give up on your blogs, your Facebook pages, your Amazon reviews, or your Last.fm playlists or your IM friend lists, for God’s sake. But remember, it’s who you are, and your relationships that matter; your “context”, and not your “counts.”

3. The really interesting question isn’t “who are the Twitterati” or twitter influencers. I’m interested in the Twitter thing mainly because I want to see how well it reflects real life. After today, I’d probably say that it doesn’t very well, wouldn’t you?

Be warned — I may just follow this research up with some research on “how many phone numbers PR people have on their mobile phones.”

Twixdagen does for Swedish politics what Tweetminster does for British. Hampus Brynolf (@hampusbrynolf) just sent me a link to this map he’s pulled together for their blog:

You’ll need to click through to his blog post to experience and interact with the map properly.

Hampus says that he used aiSee to generate an SVG file which could then be opened in Illustrator to “search and replace” on shapes, colors and lines (which explains the good-looking graph.)

Using nothing more than their public twitter relationships, is it possible to predict whether a US Congressperson is a Republican or a Democrat? The answer seems to be a guarded “yes” — our tools predict correctly 40/46 times (or around 87% of the cases.)

This post follows on from a post earlier today in which I asked, “can we calculate party affiliation?” The data set in the earlier post was gathered from the 16 members of the UK parliament who are on Twitter and the relationships between them.

Tweetcongress maintains a list of US congresspeople on Twitter. Today (February 13, 2009) there are 76 congresspeople on the service, but when I collected my data set of “who follows who” on February 3, 2009 there were only 65. Of these 65, fully 19 (29%) lived a life of noble isolation with regards the network — none of their peers linked to them, and they in turn linked to none of their peers. Removing these Miss Havishams from the data set leaves me with 46 twittering congresspeople who form a network.

Now as both social network analysis and Aesop would have it, “a man is known by the company he keeps.” What I mean by this is that given the partisan nature of politics, we should expect that Democrats will link to other Democrat twitterers more often than they link to Republican twitterers and vice versa. So that’s what NetDraw[1] , the software I’m using for most of this stuff, looks for, or more accurately:

To identify factions, NetDraw software iteratively searches for a distribution of nodes among a selected number of factions to minimise the number of connections between factions and to maximize the number of connections within factions.

Whatever. So I let NetDraw loose on the data, and here’s what it did.

I coloured the nodes red for Republican and blue for Democrats[2], labeled the nodes by party (for the sake of clarity, and for the hard-of-thinking, that’s “R” for Republican and “D” for Democrat) then counted all the nodes where label said one thing but colour another. There were six of these nodes; so NetDraw got the answer right 40⁄46 of the time (just about 87%.) This is less than the astonishing 93.75% accuracy we got with the Westminster twittering members of parliament in the previous post. Nevertheless I think we can safely say that it’s not a particularly integrated (or bipartisan) network if we can predict party affiliation with quite such success.

Here’s exactly the same map with the errant sheep re-labeled with their proper names so it’ll be easier to refer to them (if it helps, you can click on the image to view or download a larger version.)

You’ll see, I hope, that NetDraw has made a pretty good fist of the job. Where it has gone wrong on the whole is where the data clearly suggests something else. So Rep. Jared Polis for instance follows (and is followed by) no Democrat peers. Rep. Nancy Pelosi (D) and Sen. Richard Durbin (D) follow each other, but since Pelosi is followed by several Republicans and none of her other Democrat peers you can see why the algorithm has made the incorrect guess that the two of them are Republicans. Long-serving member Neil Abercrombie, as discussed in a previous post on US Congress Twitter folk, forms a bit of a bridge between the two parties, so despite his membership of the Congressional Progressive Caucus and liberal voting record, from the Twitter network point of view, his affiliation is somewhat ambiguous.

Sen. McCain follows none of his peers, and appears to inherit his incorrect attribution from Sen. Susan Collins. For the life of me, I can’t work out what makes it think that Sen. Susan Collins is a Democrat. She really isn’t, you know.

Note 1: NetDraw is a free program written by Steve Borgatti from the University of Kentucky. If you’re interested in playing around with this stuff, you’ll need to get yourself a copy.

Note 2: Actually, that’s not true. Despite a friend sharing the simple mnemonic that “‘Republicans’ and ‘red’ begin with the same letter,” I just can’t get it out of my English head that the Republicans should be blue and the Democrats red. As a result I waste precious minutes re-colouring these maps in Illustrator. It is worth pointing out that I also have problems with “left” and “right” on occasion — preferring instead the binary opposition “left” and “No! no! The other left, for God’s sake!”

This is a follow-up post to Why doesn’t the Tory MP have Twitter friends? — a report on some early research into the interrelationships between the few Westminster MPs who are on Twitter.

According to Tweetminster, the number of UK MPs on Twitter has doubled since this time last month. Where there were eight Twittering MPs, there are now sixteen. Here’s the map that shows who follows whom (the labels may be too small to read — if you want to see a larger image, click on the map.

I’ve coloured each node to show party affiliation; for those of you who are unfamiliar with British politics, Labour (our left-of-centre party) shows up in red, Conservatives (our right-of-centre party) in blue, and Liberal Democrats (what it says on the tin) in yellow.

The size of each node represents the individual’s “betweenness centrality” — a network analysis term that helps us place a value on individuals within a network. To give you a sense of what it means, the higher the betweenness centrality of an individual, the greater the impact when you take them out of the network. For those of you who work in large companies, it may be worth noting that senior management’s personal assistants generally have very high betweenness — something that is mostly remarked upon when they go on holiday (simultaneous translation: “take a vacation”.)

So far so good. By now, regular readers will probably be kissing their teeth and thinking “so what?” I’ve done a lot of these Twitter maps in the past and the novelty must be wearing off on you by now.

So here’s the thing. There are a few network analysis techniques that let one identify cliques and factions. What we’ve got here is a small set where we already know what people’s affiliations should be. How interesting, I thought, would it be to see how well the calculated result fits the real world data? Here’s what I found:

To my fairly untrained eye, the fit seems remarkably good. Only one MP out of sixteen is incorrectly placed (Derek Wyatt should be tightly affiliated to the “Labour” faction, but shows up affiliated to the “Lib Dem” faction.) That’s 1⁄16 wrong (or 93.75% accurate.)

The usual caveats

The method I used relies on you telling it how many factions there are. I knew there were three, so that’s what I told it. But what if I’d said two, or four? The software I’m using tells you how well-fitted the result is to the data, so there’s some feedback at least. But I’m going to need to play with this a lot more before I’m fully confident about putting it in front of a client.

I’m not a statistician, but even I know there’s something fishy about my claims to 93.75% accuracy.

A couple of days ago I did a little more analysis on Republican and Democratic Congresspeople on Twitter. Towards the end of the post, I realized that the unduplicated reach pareto chart that I’d built would only make sense if the US were a one-party state (or to be fair, if both parties had a single issue that they were united in wanting to promote.)

So — wanting to make this a little more representative — I went back and produced two charts; one showing Republican unduplicated reach (which follows a typical 80:20 distribution)…

And one showing the Democratic unduplicated reach (which clearly doesn’t)…

What the charts hint at but don’t show is that — while around 61% of “Republican” Twitter followers follow more than one Republican congressperson, this is true of only 29% of “Democratic” Twitter followers; that is — there’s much more duplication among the right than there is among the left.

I thought I might be able to draw some conclusions from the difference between the two charts, or at least suggest some theories to explain the differences. But I think my thoughts were best put by one of my correspondents, Kathryn Corrick who simply pointed out that the “left can never agree” (she also suggests that the political right in the UK blogosphere is more “clubbable” and co-operative than the political left – which rather fits the US maps I’ve been able either to draw or to get my hands on.)

Three weeks ago (and at the prompting of my colleague Eddie Garrett who heads up Porter Novelli DC’s digital team) I mapped out the interconnections between US Congress Tweeters. We’d been working on a Twitter crawler and it seemed like a good opportunity to test things out on a new data set.

This is a follow-up post. Once again it was prompted by a third party: Christie Findlay at Politics Magazine asked whether it would be OK to print a copy of one of the maps in their March edition. I’ve heard that three weeks are a long time in politics, so I thought I’d better run the crawl again just in case. Also I’ve got a new crawler that uses the proper Twitter API (I can see some of your eyes glazing over you know. Just skip ahead when that happens.) I’d tried it out on the Porter Novelli data set, but welcomed a chance to try it on something more meaty.

So yesterday morning before work I ran the crawl. I use the excellent Tweet Congress as my source of information about which congress people are on Twitter.

Here’s the data I get from them republished as a Google Spreadsheet.

I use Twitter itself as the source of information about who follows whom and who — in turn — is followed by whom. Then my computer does a little analysis on the data I get so that I can put it through our mapping software. Here’s the map. It’s a little too small to see, but if you click on it, it’ll take you to a huge version on Flickr.

It’s probably worth going through some of the features again briefly, just in case you haven’t seen my other maps. Actually, I don’t think I’ve ever really explained them properly.

The arrows show the direction of the relationship. If Senator Susan Collins follows Senator John McCain then there’s an arrow that points from Sen. Collins to Sen. McCain. If Representative Jason Chaffetz (Rep. Utah) and Representative Jared Polis (Dem. Colorado) follow each other then there’s a double-headed arrow between Reps. Chaffetz and Polis.

If they follow none of their peers and none of their peers follows them, then they’re relegated to the bench. In this map the bench is the list of unconnected circles in the upper left corner. For those of you who are comparing this to the last map, one of the interesting things to note is that Senator Hillary Clinton has left the ranks of the Democrat congressperson Twitter folk.

I’m not sure why Rep. Thomas Price (R, GA) and Rep. Steven Driehaus (D, OH) have Twitter handles listed. They aren’t on Twitter as far as anyone can tell. Perhaps it’s a tyop?

So far, so obvious I suspect. And it should be obvious to any US readers, the red circles are the Republicans and the blue circles are the Democrats. This is mildly contrary and confusing to my UK audience who are used to the left-of-centre Democrat analogue Labour Party being red and the right of centre Republican analogue Tory party being blue. It’s like going to France and forgetting that the tap with C on it gives you hot water. For my US readers, you probably know a tap better as a faucet. It’s rather exhausting having to do all this translation.

You’ll notice that the circles are different sizes. If you’re very observant, you’ll probably notice that the large circles have more arrows pointing at them, and that the smallest circles have no arrows pointing at them. For this map I’ve sized the circles by a factor called “indegree” which pretty much translates as “the number of arrows pointing at you.” Indegree is used in network analysis as an indicator of popularity or authority (depending on the circumstances.)

First caveat

This can be pretty spurious when you’re looking at what is (after all) only one communication channel for this network. We have to imagine that congresswomen and men (although it’s mostly men at present) will have opportunities tp communicate face-to-face, on the telephone, and via email, memos and mail and the like. So I have no idea how well this fits the reality. I’d have thought (for example) that Sen. McCain was pretty authoritative — but the Twitter map shows him to be peripheral.

This leads us to our second caveat.

Second caveat

For most of this group, the primary audience is not their peers. It is (rather obviously) their voting public. As we’ll come to see some of them have established some pretty hefty audiences (although none of them is really in celebrity territory yet.) So that probably makes this network even less representative.

Still it’s an interesting exercise. Let’s carry on.

I don’t want to labour this too much. Those of you who regularly read this blog must be getting sick of hearing about betweenness centrality. For those of you who don’t know, then here’s a brief explanation: betweenness is a measure of how important the person is in terms of flow of information. It’s calculated by working out who is on most shortest paths through the network.

If Twitter were the only way that information flowed in Congress, then Representative John Culberson is (still) the single most important person there. Taking him out of the loop would mean that the flow of information would slow down considerably, and that some people would never get the message at all.

Take both him and Representative Neil Abercrombie out of the picture and it’s like something out of West Side Story.

Sharks and Jets

Because every social media expert other than me has written a term paper on “How Obama used social media to win the election”, and because the general social media buzz tends to be slightly Democrat biased we’ve become accustomed to the idea that the Democrats do social media well. That may be true. But all I can say is — looking at this map — you’d never know. Republicans outnumber them 1.6 to 1 on Twitter (once we discount the spectral accounts of Driehaus and Price.)

I’d like to say that the Democrats had made up some ground recently, but they just haven’t. A little more analysis shows the growth in Twitter adoption by two parties. Democrats made a good start, but the Republicans really grew their numbers during June and July last year.

update: I’ve fixed yesterday’s chart. I got the whole Red vs Blue thing the wrong way around again. Thx to EG for setting me straight.

However the new Democrat joiners aren’t really linking to each other. Why is this? If you’re hoping to attract followers, one good strategy is to make the most of your network: their followers are more likely to be interested in what you have to say. More interested, that is than an equivalent control group chosen at random from the Twitterverse. So not joining networks just makes things harder.

Partly of course, we can attribute it to naivety. Most new people coming into blogging, Twitter and the rest see creating content as the hard work. Most people who’ve been doing this for a while know that it’s the promotion and moderation that sucks up your time.

Perhaps we might also attribute this to the point raised in Caveat 2 above — that they’re mostly interested in their electoral constituencies. However, if this were the case then I think we’d expect to find factions (or “cliques” or “clusters”) between congresspeople of the same colour who represent the same state. Nothing obvious there, though (and the data set’s quite small). So let’s stick with the “Twitter newbies” theory for the moment.

How many people follow the Congressmen?

So far we’ve only looked at peer relationships: all the charts and calculations we’ve done so far are based on a much-reduced data set. Experience shows that it’s very difficult to map “all followers” (we end up with something that looks like a hedgehog that’s had a wild party in a cranberry field.) But we can still count followers. Here’s a quick list — you can see that there are one or two superstar congresspeople, but that the rest really aren’t doing even as well as your average journalist.

OK — now let’s assume that these aren’t unduplicated audiences; that there must be several people interested in US politics who follow more than one, many, or all the names on the list. What is that duplication like?

Luckily I’ve just written a little analysis script to look at exactly this sort of question, so I can tell you with confidence that (while the absolute numbers above might lead you to think that the congress Twitterers reach an audience of around 56K) if they all pull together, they actually reach an unduplicated audience of 24K. Of course, now I look at it, I realize that they are unlikely to pull together in this way, and that I should have separated the lists into Republicans and Democrats before I went at this. This is somewhat frustrating — but I can cover it in a new post, I suppose.

Here, then, is the somewhat useless Pareto chart showing the unduplicated reach as though the US were a one-party state.

I’ll come back to this, but if you have questions, you might like to look at this Q&A I wrote over the weekend.

A recent post about some Pareto analysis of the Porter Novelli Twitter sample , “Porter Novelli Twitter folk – the 80/20 rule”, stirred up a little bit of interest on Twitter — and made me think again about what I’m doing and why. Partly because those conversations were off-blog (and I’d like to capture the answers I gave somewhere more permanent) and partly because I’ve now had time to think of better answers I thought I’d set them down here.

First, a little background. This Q&A is the sixth post in an impromptu series about the Twitter people where I work (Porter Novelli, the international public relations agency.) By now you might think that I’d be tired of this stuff, but you’d have another think coming. Here’s a quick list to bring you up to date.

1. Map of Porter Novelli people on Twitter on 17th Jan 2008
2. Map of Porter Novelli people on Twitter on 20th Jan 2008
3. Introducing the Porter Novelli magic Twitter friend maker (beta)
4. Porter Novelli Twitter folk ranked by number of followers
5. Porter Novelli Twitter folk – the 80/20 rule

Looking at this, you might also think I clearly had nothing better to do than analyze Porter Novelli people and their Twittering ways. In fact, as an experimental data set, I couldn’t really ask for anything much better. It’s sufficiently large (more than 200 people), international (I’ve counted more than 10 countries — and I’m sure there are more), and I have some real-world access to all of the people in the sample, which means I can compare my findings with some hard data.

That said, the experiment is more about learning about how we can analyze Twitter networks — about discovering how representative they are as a word-of-mouth (WOM) channel for example, and what they can tell us about other kinds of social network, or about finding new ways to analyze such data sets — than it is about answering any specific questions. So I’ve not got any carefully mapped-out research plan. Instead I follow paths that strike me as interesting, or possible, or that are suggested to me by friends and readers.

Question 1

I’ve posted this exchange as a slideshow so you can sort of follow along. Twitter updates are rather telegraphic (it’s the 140 character limit) so that’s another reason to take the time to answer the question more fully here.

Caroline (@lemondrizzle) is a Porter Novelli colleague based in London. She wanted to know what purpose of de-duping the follower list serves. The first answer is that it gives us a clearer idea of the sample set of Twitterers’ reach. Here’s what the unduped list looks like:

As I said in an earlier post, there must be many, many duplicated connections here.

When I de-duped the network, it looked more like this:

If we count people more than once it gives us an inaccurate picture of the potential impact of the network. Looking at the Alice/Bob/Carol example (I changed “Adam” to “Alice” after Caroline accused me of sexism) you’ll see there are two ways to count the followers.

If we include the duplicate entries, then the three have an apparent combined reach of 18 people. Once we de-dupe, they have a combined reach of 10. The same goes for the Porter Novelli Twitter network (only more so): once we dedupe that, it decreases from an apparent combined reach of around 21K users to a combined reach of around 6K people.

So were we able to mobilize everyone on our Twitter list in another campaign like Porter Novelli CMO Marian Salzman’s #pnobama exercise from last month, we’d reach around 6K people.

Of course, it’ll be impossible to mobilize everyone. By ordering the sample from most to fewest followers and de-duping, we could now identify our key targets for the activity and instead focus our attention on them.

Costs and Benefits

Furthermore we have a good understanding of the costs and benefits of this strategy.

Assume that we are dealing with a new Twitter network and not our own internal network. Assume further there is a “unit cost” of engaging with each new Twitter person in the network of one whojamaflip (where a whojamaflip might be a financial cost, or the amount of time and effort it takes to read their blog and previous tweets, work out who you know in common, prepare and deliver your pitch etc.)

Knowing that the top 20% or so of the group (at a cost of 40 whojamaflips) will reach 80% or so of all the followers — around 4.8K opportunities-to-see (OTS) at a cost-per-thousand (CPM) of 8⅓ whojamaflips — tells us a lot about how and where we should focus our efforts, and about the incremental benefit of reaching the remainder of the group — 1.2K OTS at a whopping and unjustifiable CPM of 133⅓ whojamaflips.

Do we really reach 4.8K people?

Of course not. For one thing, not every follower is “real”, and for another we can’t guarantee a view. One of my followers, for example, is Barack Obama. If I thought he were hanging on my every tweet, I’d be too scared to commit words to screen.

Instead we should see followers as opportunities-to-see (or “OTS”) rather than views or eyeballs.

Are numbers all that matter?

No, of course not. I tend to work on a simple model of “influence” that looks at 3 factors:

1. Popularity/Reach — the number of people whom an individual influencer can affect – generally a ’size of audience’ figure
2. Authority — ‘people who talk about subject x are most likely to mention these influencers’
3. Proximity — probability of being present at or involved or consulted in a given decision.

But even these won’t give you the whole picture, there are lots of other factors (things like “trust”, say) which don’t completely fit into this model. And — as people like Victoria have pointed out on previous posts — I make no judgment on things like how compelling a message or product we’re selling.

I think it’s a matter of perspective. Just like there are things that make an effective virus (things like a short incubation period, long infectious period and resistance to treatment, say) there must be things that make an effective message. Lots of people I know are (or have been) working on this. Indeed, I maintain a keen interest myself — I just don’t have much to share.

What do you look at, then?

Instead, I find myself looking at the channels, the vectors of transmission. To continue the virus metaphor; however virulent a virus may be, it makes little difference if (like smallpox) it is stuck in a few laboratory freezers and can’t get out.

What’s that about clumpiness and bridge nodes, then?

When we map networks of people we generally find that they are “clumpy” — what I mean is we can see many tightly knit groups of people who are linked to the other clusters by a few thin threads.

Take your own Facebook friends as an example. There’s a strong probability (depending on factors like your age group) that there will be around three or four clusters; most probably colleagues at your present job, university friends, and old school friends. Let’s say you share some news with these close-knit groups via Facebook. What incentive do they have to share the information on? There’s a very strong likelihood that they share many of the same friends and colleagues with you — the natural assumption must be that everyone who matters is already aware of the news you’ve shared. The last thing they want to do is tell everyone what you’ve already told them. Where’s the network utility in that?

What this means is that news is actually most likely to be repeated by people on the very edge of that group; people who are (from their position) less likely to think of that group as the only audience. These are the people who share stories; what — along with lots of other people in this area — we’re currently calling “bridge nodes.” Caroline doesn’t like me calling them bridge nodes, though, so we may need to address this. Otherwise she’ll steal my pencils.

Question 2

@Amber1230 asks a different kind of question, and one that’s a bit harder to answer.

As it happens, second-degree networks can (in fact) be factored into our calculations. I’ve done quite a lot of stuff with second-degree networks in the past. Take a look, for example, at this map I made of my Facebook connections and their connections back in 2007.

Sure, the maths becomes a little more complicated. If you really want to know, we start talking about things like “eigenvector centrality” rather than “degree centrality”. Eigenvector centrality says (more or less) “it’s not who you know, it’s who they know.” A short paper by Steve Borgatti called Centrality and AIDS has this description of the distinction:

In the context of HIV transmission, we can see that all else being equal, a person A with one sex partner has a better chance of escaping the infection than a person B with many. But if that one sex partner that A has is having sex with most of the network, A’s chances of getting infected are nearly as good as if she were having sex with all those others herself.

So far, I think this is exactly what @Amber1230 is saying. The problem is one of tense; I’m trying to predict stuff, not describe what has already happened. We know that everyone in someone’s first-degree network will have an opportunity to see the initial message. But the only ways that someone in the second-degree network might see it is if they are already in the first-degree network, or if one of the first-degree people retweets it (or does something off-network.) The retweets issue is actually rather difficult to solve (although some headway is being made by people like Dan Zarella and Joost de Valk among others.)

So for the moment, I’m trying to work with what I know — or what I can know.

Last weekend I posted a chart of Porter Novelli Twitter folk and their followers. If you read it, you’ll recall that I was dissatisfied by what it implied about the collective reach of Porter Novelli twitterers.
Well, thanks to a long-ish train journey to Bolton and back, I was able to fudge a little perl script together to look through the data to find and remove everything other than the first instance of a follower. Let’s make that a little clearer. Let’s say that we’re looking at three Twitter people, Alice, Bob, and Carol. The first thing to do is to see who follows them:

Now we need to rank them in order of “who has the most followers” (also known as “popularity” as it happens). Here I’ve done that from left to right. Bob has the most followers and Carol the fewest.

And finally we go through from left to right removing all followers who have already shown up on someone else’s list.

Bob, being at the top of the list gets to keep all his followers which may seem unfair. But it’s not unfair if the question we’re trying to answer is “how do I reach as many people as possible by speaking to as few people as possible?” That is, I’m looking for reach (marketing people often express themselves in terms of “reach” — or the number of people who are exposed to a message — and “frequency” — or the number of times the average person is exposed to that message.)

Looking at the example above, we can see that Alice really delivers an incremental benefit of two new people, and Carol only reaches one new person. That gives us a much better idea of how valuable the most popular person (Bob) really is.

Applying this to the Porter Novelli data set

Clearly it would be extraordinarily boring to perform the process described above for the 205 people in the Porter Novelli data set that I want to analyse. But the analysis script that I wrote (with plenty of help from the perl monks) goes through exactly these steps. It’s a pretty straightforward job, ranking and deduping. Here’s what we get.

This makes much more sense than the last run. According to the Pareto principle, roughly 80% of the effects should come from 20% of the causes. Here we see that 20% of the Porter Novelli Twitter users (marked in black) account for slightly more than 80% of the reach (marked in red.) It’s pretty much a text-book example. Things are as they should be, I suppose.

More to the point, we can now assign appropriate value to coverage at the head of the graph. This is of great value when thinking about our media planning and engagement

By the way — if you’d like a copy of either the Twitter follower API query engine (it’s a well-behaved command-line thing that was developed by the excellent Joachim Larsen) or the slightly shonky perl script that I wrote on the train, you have only to ask: I’ll be pleased to share. Send me a tweet at @mediaczar and I’ll send you the scripts.