Characteristic Tweets

By: Jeff Clark    Date: Fri, 01 May 2009

There are huge numbers of Twitter status messages being created every day. I've been tracking tweets containing the word 'obama' for more than 250 days now and on average there are more than 10,000 tweets/day. There is so much data that it can be overwhelming to try and extract useful information. The nature of the twitter platform means that any useful information for a particular topic is highly fragmented. There is also a large amount of redundant information especially since so many tweets are actually 'retweets'.

Can we construct something approaching a narrative from all the bits ? Can we eliminate much of the redundancy ? I've started to tackle this problem with the following approach:

  1. Gather a collection of tweets for a topic of interest
  2. Eliminate non-English tweets
  3. Partition the tweets into separate bunches by date and time
  4. Analyze the word frequency in the bunches and determine, for each bunch, what the characteristic words are. These are the words that occur relatively more frequently in that bunch compared to the complete set
  5. Use the word relative frequency for each bunch to find a 'characteristic tweet' for each bunch. Roughly speaking, this is the tweet in that bunch which contains the highest proportion of words that are characteristic of the bunch

As an example I have analyzed a sample of tweets taken from Obama's first 100 days in office. The table below shows the characteristic tweet for each day. I used every 25th tweet containing 'obama' in the time period and discarded non-English tweets. This left me approximately 75,000 tweets for the analysis. It seems to work fairly well. You can read through them and get a pretty good summary of the various Obama-related events that have recently occurred.

Days 1-50:

DateCharacteristic Tweet
Jan 20, 2009charlesta: watching Barack Obama's inauguration on TV
Jan 21, 2009francis_gt: watching obama's inauguration speech
Jan 22, 2009GeorgeReese: Obama retakes the oath of office tonight :)
Jan 23, 2009Hops11: Obama overturned global gag rule! YES!
Jan 24, 2009PoliticsFix: Obama reverses abortion-funds policy - http://is.gd/h1VQ - WFIE-TV
Jan 25, 2009odanielpavon: Some global adversaries ready to give Obama chance (AP): AP - In his inaugural address, President Barack Obama signaled conciliation t..
Jan 26, 2009dustytrice: Breaking: Obama will direct EPA to move swiftly to grant 14 states the right to set strict auto emission standards on Mon (via @Populista)
Jan 27, 2009nyycarl07: @ricksanchezcnn Obama's Al-Arabiya interview/Mitchell Mideast visit...mending fences with the Arabic world..meaningful dialog..long overdue!
Jan 28, 2009YahooNews: Obama open to compromise on $825B stimulus bill (AP)
Jan 29, 2009keramurphy: Obama signed the Lilly Ledbetter Equal Pay Bill. Love it.
Jan 30, 2009binikadwa: Even Obama's rooting for the Steelers
Jan 31, 2009bigkumadog: Obama's half brother arrested on charge of drug possession: NAIROBI, Kenya - George Obama, the half brother of U.. http://tinyurl.com/dzazy8
Feb 01, 2009wbaustin: Obama Takes Jab at Chief of Staff at Alfalfa Club Dinner: President pokes fun at his volatile chief of staff Rah.. http://tinyurl.com/cbhkrd
Feb 02, 2009caerickson: Rooney just thanked Obama for supporting the Steelers!
Feb 03, 2009Headline_News: Daschle withdraws as HHS nominee: Former Sen. Tom Daschle has asked President Obama to withdraw his nomination f.. http://tinyurl.com/d66eaj
Feb 04, 2009idigg: Obama To Cap Executive Pay At $500K For Bailout Recipients http://tinyurl.com/abqpq2
Feb 05, 2009gregspradlin: Reading about Fairey and AP......AP alleges copyright infringement of Obama image .. http://tinyurl.com/czxvat
Feb 06, 2009nelking: @joshcagan Headline: "Senate Struggles on Stimulus in Nighttime Session" Related news: Obama adds Dr. Ruth to Economic Advisory Board
Feb 07, 2009latimesnational: Artist of famed Obama poster arrested in Boston: Police in Boston say the artist famous for his "Hope" posters o.. http://bit.ly/FPN6
Feb 08, 2009inaug: #Inauguration Lompoc man has front row seat at Obama inauguration - Lompoc Record: Lompoc man has f.. http://tin.. http://tinyurl.com/bm ...
Feb 09, 2009ElkhartTruth: Obama: "We've got the best workers right here in Elkhart." #obamaelkhart
Feb 10, 2009jclayiv: watching the obama press conference
Feb 11, 2009fwstylewatch: breaking... michelle obama's march vogue cover finally unveiled!
Feb 12, 2009Love_The_Oscars: Obama praises Lincoln's legacy at Ford's Theatre
Feb 13, 2009Politisite: Republican Senator Judd Gregg withdraws as Obama's Commerce Pick over conflict on stimulus #tcot
Feb 14, 2009NewsOnTwitter: MSNBC - Obama: Stimulus bill is 'major milestone': President Barack Obama, savoring his first major victo.. http://tinyurl.com/cvv6gc
Feb 15, 2009lemonhed77: news update Air Force One is one 'spiffy ride,' Obama says: It's longer than a hockey rink, has two f.. http://tinyurl.com/b8wky4
Feb 16, 2009imacsweb: Obama decides on task force to oversee auto industry reform rather than appoint "car czar" http://tinyurl.com/cv66z3
Feb 17, 2009keyc: Pres. Obama Signs Stimulus Bill in Denver | http://keyc.tv
Feb 18, 2009timesnews: Obama to unveil mortgage foreclosure plan http://www.timesoftheinternet.com/47845.html
Feb 19, 2009caniba: Obama goes to Ottawa, ON, Canada and what do the Internets call it? #Obamawa -- I don't say this enough but... I love you Internets.
Feb 20, 2009ThomasGalvin: thinks its funny that Obama is lecturing mayors to "spend wisely"
Feb 21, 2009roadkillrefugee: Obama's Weekly Video Address: Quickest & Broadest Tax Cut EVAH! http://tinyurl.com/dxdg7b
Feb 22, 2009IvorKellock: Obama aims to halve deficit by 2013 http://ff.im/-1aRkZ
Feb 23, 2009AccordionGuy: Sasha Obama Keeps Seeing Creepy Bush Twins While Riding Tricycle Through White House: http://is.gd/kyi1
Feb 24, 2009sumbonet: NewsOnTwitter: BBC NEWS - Japan PM visits Obama White House: Japan's Prime Minister Taro Aso will be the first... http://ff.im/-1bNY1
Feb 25, 2009amyz5: For those who missed my post speech commentweet last night: Obama is to Jindal as Dylan is to the Jonas Brothers. #nsotu
Feb 26, 2009neilkelty: Disappointed in President Obama's budget.
Feb 27, 2009profchandler: RT: @NewsHour: At 11:45 Obama will address Marines at Camp Lejeune.expected to announce withdrawal of U.S. combat forces from Iraq Aug 2010
Feb 28, 2009headlinenews: AP: Obama moved toward commanders in Iraq decision: WASHINGTON (AP) -- President Barack Obama leaned heavily .. http://tinyurl.com/chavyl
Mar 01, 2009ReddingNews: Data on Obama's Helicopter Breached Via P2P?: Tiversa, headquartered in Cranberry Township, Pa., reportedly disc.. http://tinyurl.com/cd28gf
Mar 02, 2009thebodybreaks: Obama nominates Gov. Sebelius for health post: Kansas Gov. Kathleen Sebelius, President Obama's nominee to head .. http://tinyurl.com/d989ha
Mar 03, 2009atifunaldi: Sources: Obama to shelve species rule
Mar 04, 2009TechGlance: Obama taps Julius Genachowski to head the FCC http://tr.im/h10T
Mar 05, 2009leeharveydent: Watching CNN: Obama's Rx for health care reform.
Mar 06, 2009news_by_robots: Obama to Lift Ban on Funding for Embryonic Stem Cell Research @Washington_Post
Mar 07, 2009caketeagirl: Pleased about Obama's decision to reverse Bush's limits on stem cell research
Mar 08, 2009ftantillo: "The Rock" Obama on SNL = awesome
Mar 09, 2009Atticus_James: yay obama and stem cell research!
Mar 10, 2009HootieMcBoob: Go Obama on the stem cell research! WOOOT! :D

Days 51-100:

Mar 11, 2009NewsHour: President Obama Signs $410B Spending Bill Despite Earmarks http://twurl.nl/r0xrti
Mar 12, 2009Pseudonymdc: FBI just raided the office of Obama's nominee for Chief Tech Officer.
Mar 13, 2009theRQ: Obama dropping term "Enemy Combatant" for his Guantanamo buds. Perhaps going to use "Comrade"?
Mar 14, 2009newstalk1430: Obama: Food safety a 'hazard,' picks FDA chief: WASHINGTON (AP) -- The nation's food safety system is a "hazard .. http://tinyurl.com/bdhzc3
Mar 15, 2009blukentucky: RT @ladyreporter Obama to Unveil Proposals to Help Small Businesses http://ff.im/-1wAND
Mar 16, 2009UNCLEDAVIDFAMIL: Obama will try to block executive bonuses at AIG
Mar 17, 2009carolevansmd: @googlenews Did Obama name Rooney ambassador to Ireland yet?
Mar 18, 2009justin: Obama certainly has some *conservative* picks in his NCAA bracket. yuk yuk.
Mar 19, 2009cbobest: I can't belive President Obama Ncaa Bracket Smh Prez!!!!
Mar 20, 2009rsspberry: Barack Obama makes 'special Olympics' gaffe on Jay Leno's The Tonight Show - http://rsspberry.com/view/35969/
Mar 21, 2009mazmazcool: Loved the obama 'special olympics' joke
Mar 22, 2009StephanieTaylor: Obama says he would not accept Geithner's resignation. Really? Am I the only one pissed?
Mar 23, 2009Smalltalkwitht: RT @txsbelle: RT @infidelsarecool: 60 Minutes Reporter Asks Giggly Obama: "Are You Punch Drunk? http://twurl.nl/p6p1xz #tcot #sgp
Mar 24, 2009foxnewspolitics: Obama Tries to Temper Furor Over AIG Bonuses: President to use prime-time news conference to... http://tinyurl.com/cjbo4x
Mar 25, 2009bkmonroe: CNN on the diversity of questions at tonight's Obama press conference (including Ebony's question)http://tinyurl.com/c3d5g5
Mar 26, 2009chuckbaakel: Obama's hosting an online town hall. 1/4 of the top voted questions by the people are about marijuana. http://bit.ly/oKgE6
Mar 27, 2009DanteDeRycke: Obama announces new strategy for Afghanistan, Pakistan.
Mar 28, 2009WorkingClassFag: Obama monitors Midwest flooding, pledges govt help. http://tw2.us/Vy
Mar 29, 2009ABQTweets: Widening Afghan mission, Obama narrows goals - AFP: FOXNewsWidening Afghan mission, Obama narrows goalsAFPWASHIN.. http://tinyurl.com/d8 ...
Mar 30, 2009wwlp: Obama says Auto industry must do more: President Barack Obama says General Motors Corp., Chrysler LLC and all th.. http://snipurl.com/etpzu
Mar 31, 2009mottv: 20090331T17:47:19 (BBCNews) President Obama is due to arrive in London this evening ahead of the G20 summit, his first trip outside North A
Apr 01, 2009MikeSoron: Obama Depressed, Distant Since 'Battlestar Galactica' Series Finale - http://is.gd/pY1f
Apr 02, 2009abroadjz: Obama gave the Queen an iPod
Apr 03, 2009TheOval: Obama in France for NATO summit; budget moves ahead : Good morning. President Obama is now in Strasbourg, France.. http://tinyurl.com/dcmpur
Apr 04, 2009liquidmarket: Listening to Obama live on nato summit
Apr 05, 2009ObamaNews: President Obama in Prague: "A world without nuclear weapons" http://twurl.nl/j87l4c
Apr 06, 2009pdrouin: RT Turkish security services have arrested a Syrian man who was planning to assassinate President Obama during his trip in Turkey. Vrai?
Apr 07, 2009charlesadler: President Obama Lands in Baghdad on Surprise Visit to Iraq
Apr 08, 2009tweetybud: The White House has hired actor Kal Penn (Kumar) as a liaison between President Barack Obama's administration and Hollywood.
Apr 09, 2009KBinSC: LYING PRES _ White House: Obama didn't bow to Saudi -'He grasped his hand with 2 hands, and he's taller than King Abdullah'
Apr 10, 2009NewserOpinion: No Honorary Degree for Obama: ASU http://ow.ly/2xBs
Apr 11, 2009femsol: RT @ryking: #p2 #tcot Wingnuts in a tizzy because Obama ordered a pizza: http://tr.im/iDlv mLOLOLOL
Apr 12, 2009hartfordcourant: Check out this cute pix of the Obama's new puppy, Bo - a Portuguese Water Dog. http://sn.im/fr959
Apr 13, 2009stockingvixen: RT @BreakingNews: AP: Obama will allow unlimited travel and money transfers by Cuban Americans to family in Cuba.
Apr 14, 2009mrwashy: Obama opens crack in U.S. embargo against Cuba http://viigo.im/iHZ
Apr 15, 2009mrshmrsh: "White House says Obama 'unaware' of tea parties..." and that is how he is running the country *UNAWARE*. #teaparty
Apr 16, 2009zachflauaus: Really? Obama is pushing for a high speed rail? I'm so excited! Not.
Apr 17, 2009yatinsactivity: Obama Unveils High-Speed Rail Plan http://ff.im/-2bwPG
Apr 18, 2009kenellis: Obama and Venezuela's Hugo Chavez enthusiastically - each other at the Summit of the Americas as Chavez says: 'I'd like to be your friend.
Apr 19, 2009intunit: Return to Soft Power? Americas 34nation summit ends on hopeful note 'I think President Obama is an intelligent man' Chavez http://ow.ly/3eO1
Apr 20, 2009bostonherald: Official: Obama wants agency spending cut by $100M: WASHINGTON - President Barack Obama convenes his first forma.. http://tinyurl.com/c6sv6a
Apr 21, 2009RepublicanGOP: RT @USAMIA @AwakenToTruth @allahpundit: Cheney calls Obama's bluff: Release the CIA memos proving that torture works http://is.gd/tyH2
Apr 22, 2009rtsradio: Obama heads to Iowa for Earth Day http://ow.ly/3wTN
Apr 23, 2009ObamaNewsOn: Obama to prod credit card firms on fee practices - Reuters http://ad.vu/3kbx
Apr 24, 2009Milagro_BeBe: Obama in a Florida GATORS jersey, now that is hott!!!!
Apr 25, 2009ChannelMassive: RT Day 95: President Obama furiously asks Cabinet who unlocked Sakura on his Street Fighter IV save file.
Apr 26, 2009foxnewspolitics: White House: Swine Flu Briefing Not a Deflection From Obama Golf Outing: White House spokesman... http://feeds.foxnews.com/~r/foxnews/po ...
Apr 27, 2009timdoctor: Has Obama appointed a Swine Flu Czar, yet?
Apr 28, 2009SmartAsshat: "They lied. Obama WAS on Air Force One over New York. The proof: He sprayed swine flu on all the schools! NObama!"- #tcot twats
Apr 29, 2009webnex: Obama On Flickr: The First 100 Days and More http://ff.im/2rsL2

 


TED Shaped Word Cloud
Blog
Chrysler Tweet Summary