Category: DailyRiczWest

Capturing your FlipBoard Magazine posts

FlipboardIconAs I’ve written about previously, although I’m not directly on Twitter at the moment, I am still curating information using FlipBoard Magazines and currently “publishing it” weekly as the Weekly RiczWest, which is basically a static page on WordPress with some links to my FlipBoard magazines that is available as a menu item at the top and is also tweeted using IFTTT (If This Then That).

DailyRiczWest-7oct13This is not really a satisfactory solution as I’d actually like to produce a “custom” magazine which contains only the content of the past week, which is the same idea I had with the Daily RiczWest. For all my gripes against Twitter, at least they make their content available via IFTTT (it wasn’t for a while, but now it’s back). Unfortunately, FlipBoard seems to be rather a closed system. This has obviously offended the “hacker” (and I mean that in a good way, not the common debased usage) so I’m looking to open it up and continue work on the core concept of taking a set of “highlighted posts” (by Flipboard Magazines, Twitter Favourites, …) and put them in to a custom stream and newspaper.

The first step is to free the information from FlipBoard to a neutral format that can be worked with. As usual, “the web” can help rescue us. FlipBoard have (marginally) “opened up” their magazines by giving them a URL which is updated and viewable by others on a browser. This gives us just enough room to actually capture the information, but how?

There are essentially three problems here:

  1. Sense changes to the web page
  2. Capture those changes
  3. Save them somewhere

3 will be solved by our old friend DropBox along with IFTTT for the high level orchestration, but it really could be anything.

For 1, before I found out I could only get my magazines “externally” from FlipBoard I was hoping they would have them exposed as RSS – unfortunately not :-( It’s a good idea though, so this lead to me searching for some way to convert web pages to RSS feeds. Luckily, there are a number of possible solutions, but I’ve chosen page2rss which is very easy to use (i.e. just give it the URL and it “does” the rest).

For 2, you simply set up an IFTTT Recipe that is triggered by your RSS feed and saves the ‘changes’ in the Web Page from 1

FlipBoard to DropBox 1

FlipBoard to DropBox 2

Once you have this set up you’ll get way too much information in your file as it seems to be a JSON Object – I’ve given a stripped down example below

"id":"flipboard-Kp8ig9DcRLGi9DYCWVwKXg:a:1234567-1386361164",
"hashCode":3819861521,
"sortValue":31117306186617,
"type":"post",
"title":"Z Motherboards For Mini-ITX Builds - Reviews - Tom’s Hardware",
"excerptText":"Pre­vi­ous<p>The Mini-ITX Mar­ket Is Small, But Growing...<p>
ASRock ZE-ITX<p>ZE-ITX Software<p>ZE-ITX Firmware<p>Asus ZI-Deluxe<p>
ZI-Deluxe Software<p>ZI-Deluxe Firmware<p>EVGA Z Stinger<p>Z Stinger Software<p>Z …",
"sourceURL":"http://www.tomshardware.com/reviews/z-motherboard-roundup,3656.html#xtor=RSS-999",
"hideOnCover":true,
"inlineImage":{…},
"language":"en",
"authorDisplayName":"Thomas Soderstrom",
"referredByItems":[
{
"type":"status",
"authorDisplayName":"RiczWest",
"authorUsername":"RiczWest",
"userid":"1234567",
"authorImage":{
"smallURL":"http://cdn.flipboard.com/uploads/avatar/7b432b8649b94792ac273a75720224789510b1fc.jpg"
},
"text":"",
"urls":["http://www.tomshardware.com/reviews/z-motherboard-roundup,3656.html#xtor=RSS-999"],
"service":"flipboard",
"remoteServiceItemID":"Kp8ig9DcRLGi9DYCWVwKXg:a:1234567",
"dateCreated":1386361164,
"ingestionTime":1386373813,
"sectionLinks":[…],
"original":{…},
"id":"flipboard-Kp8ig9DcRLGi9DYCWVwKXg:a:1234567-1386361164"
}
…
],

as in the next post I’ll be looking at parsing this for enough information to produce a WeeklyRiczWest newspaper. I’m hoping that we may even be able to use some of FlipBoards summary extraction features…

DailyRiczWest v0.9

Ralph MemorialIf you’ve been reading this blog for a while, you’ll recognise the title as ages ago I did a post DailyRiczWest – behind the scenes v0.1. That was quite a while and a bit has changed so I thought I’d do a quick update for people.

This program will take the concept of a personal paper like paper.li and spotify to it’s ultimate conclusion (well, for me at least ;). Instead of some “automated filter” picking things for you, the DRW works off any twitter items that I’ve favourited from any application (e.g. Flipboard, Tweetbot etc…). One advantage of this is that I can just tweet key items and pop everything else in a digest form. Here’s a link to an example of what was produced for yesterday.

DailyRiczWest-7oct13How does it work? Easy – it’s based upon an If This Then That recipe called Favourites to Dropbox for DailyRiczWest which just adds information on anything that you favourite to a file ifttt/favourites.txt in your Dropbox (assuming you have a “Public” folder).

There’s then a perl program and some templates which munge this in to an HTML digest of the previous days favourites. Eventually, I’d like to pull the content to the links summarized in to the digest but that’s a ways off. Being an agile kinda guy though, I’m doing this socially and releasing early and often.

Eventually I’d like to make this an easy to set up and stand alone project, but that’s a long ways off (as this is just a play project for me) but if you can wrap your head around how this works and set it up then let me know. If you have some problems, then feel free to ask…

We’re All going on a Twitter Holiday

Twitter Holiday

So, the time has come where I’ve finally been “overloaded” by Twitter on many levels.

This is not a bad thing, it’s just a thing… So as a result, surprise, surprise: I’ll be going on a Twitter Holiday (not really with Sir Cliff Richard ;) for a number of reasons:

  • A real Holiday has actually “emerged” in my life – I’ve finished a contract and:
    • An Aussie Mate will be visiting us for a while
    • We’re going to Seville in Spain for a while with our US relations (one of whom speaks Spanish because she works at a community library in US)
    • I’m catching up with an Aussie Uni Buddy and his new “girlfriend” (sounds strange as we’re both getting near 50 ;) in France
  • DailyRiczWest is dead! As I recently tweeted, I was going to do a lot more with that, but now that Twitter killed ifttt.com, I can’t really be bothered, for the moment… Anyone who knows or has worked with me, should know that I can be a stubborn, sneaky and clever bastard, capable of working around the most entrenched and stubborn systems. Needless to say, Twitter have handed me a challenge which I thank them for – Namaste’ :-)
  • I’m getting tired of the “cognitive constraints” of 140 chars! But as a result of those chars, I want to start  blogging more – around the 1,400 char point – concise, yet clearly explained
  • I love the twitter debates, but I want to get personal! Again, this is more a reference to the restrictions of twitter – I’m quite happy to email people. I could Skype them, or even maybe meet them F2F! Not Tweeting will inspire me to progress some relationships…
  • I’ve just finished up “handover” for another contract and in that way, this feels like a natural extension of that, as I’m kind of doing a handover of my experience on Twitter
  • We’re (ie my wife and I) are going through an incredible period of change – moving from the most amazingly toxic street, where we lived (foolishly) for two years to a most amazingly positive one in a totally new county! This takes time to integrate…

Finally, and probably most importantly, I’m just over-filled with Twitter Information and People!

I mean this in a totally positive way. I have formed relationships which will be developed and have been exposed to so many very smart and aware people, that I’ve yet to catch up with. And therein lies the problem – I can’t catch up with them by reading their tweets! I’ll meet some of them and read much material by them or recommended by them…

Another way to put it is that I need some reflection time from the firehose. During the past (really year or so as although I “joined” in 2009, I didn’t do much for a year or so…) I have gathered just so much information!

  • 4,444 tweets, probably a lot of which are links to articles
  • A lot (~1k?) of favourites, which I used when Twitter weren’t API Fascists to dump to text files
  • Other information related to tweets, which I have traced and tweeted

Probably, more importantly, I have formed what I feel are real connections with people – sharing my highs & lows. Having arguments, disagreements and agreements (mostly :) whom I’d like to acknowledge and encourage you to check out and maybe follow:

  • Jennifer Sertl @JenniferSertl – We just connect! I’m looking forward to meeting her some day (probably at CultureCon London 2013 :) Oh – what does she do? Strategy, Leadership and the Soul: Resilience, Responsiveness and Reflection for a Global Economy – we obviously need people like her!
  • Bob Marshall @flowchainsensei – sometimes we think the same, sometimes different. I like challenging him and having a good ‘ol discussion / debate / duke out (:) – hopefully catch up F2F @ XTC…
  • My Anonymous Friend who would probably have to kill you if you knew who he was ;-) We have so many good discussions that are not on Twitter, but should be, even if in censored form or relayed via me… You obviously know who you are – and who everyone else is ;-)
  • Alex Ley @AlexEvade – worked with him for a bit – you should check out his latest project : http://www.nerdability.com
  • tom_gall @tom_gall – Originally followed because he had a NeXT logo as his avatar – must admit I’m a bit dissapointed that it’s still not that, but he’s definitely worth following…
  • Tony DaSilva @Bulldozer0 – always great stuff – we like Control Theory ;-)
  • Andrea Chiou @andreachiou – puppy person and GREAT agilist
  • Jim Benson @ourfounder – do I really need to introduce him? no…
  • Scott Amber @scottwambler – do I really need to introduce him? no…
  • Grady Booch @Grady_Booch – do I really need to introduce him? no…
  • Ron Jeffries @RonJeffries – Never really been able to figure out who this guy is ;-) …
  • Peter Vander Auwera @petervan – I call him “The Banker”. But I mean that in a positive way – many good tweets
  • Tom Graves @tetradian – strangely enough, I have never really “twalked” with him. Probably because we think quite closely on the issues he tweets about
  • Ben Wilcock @benbravo73 – a “SOA Thinker” – Never met, but I found out about him when I worked at Glue / Reply. Good tweets
  • Ruth Malan @ruthmalan – now that Tom Gall has forsaken his NeXT Avatar, Ruth now has the best One!- we’re pretty aligned…
  • Phil Nash @phil_nash – met @ XTC – does interesting stuff
  • Dave Snowden @snowded– do I really need to introduce him? no, yes? I’ve only recently “really discovered” him, and he makes so much sense!
  • Byron Conway @aricochet – Recoveering Scientist – tweets some very broad and amazing stuff
  • Dan Mezick @DanMezick – was introduced recently – the man behind #CultureCon2012 and hopefully the one also behind #CultureCon 2013 which will be in London!
  • John Wenger @JohnQShift – hmm… hard to describe – just follow him! :-)
  • Bernd Nurnberger @CoCreatr – recently found out about him via #CultureCon2012 and an all-round interesting guy
  • Paul Klipp @agileactivist – Heard of Kanbanery? he’s that, and a lot more, including the ACE Conference, which I’ll be going to next year
  • Chris Ford @ctford – Saw him at a London Clojurians meeting. Music, Clojure – what’s not to like about this guy?
  • Ari-Pekka Skarp @apskarp – A mysterious Finn. But then again, which Finn’s aren’t mysterious? ;-)
  • Dave O’Hara @David_A_Ohara – hey – I’ve worked with Dave and he’s brilliant – you’ll be lucky if he starts blogging one day about what he really does – apart from check in to locations and listen to music ;-)
  • Richard Veryard @richardveryard – met at BCS event, works for CBDI and has some interesting EA stuff to say
  • Mark Pesce @mpesce – do I really need to introduce him? no? yes? Apparently has some sort of Secret Project – I suppose we’ll find out about it some time…
  • Ross Dawson @rossdawson – futurist and maker of many cool diagrams…
  • Sleepy Fox @sleepyfox – met at XTC – again, interesting
  • Sit At The Table @SitAtTheTable – so go sit at the table with her! ;-)
  • Rasperry Pi @Raspberry_Pi – do I really need to introduce them? NO! Yes, they are the one and ONLY – follow them if you don’t already…
  • Marcus du Sautoy @MarcusduSautoy – do I really need to introduce him? Yes? OK, if you’re outside of the UK, this guy is a maths legend…

Finally, to repeat my BlogRoll, here’s some blogs you may want to follow (that are not already mentioned above) :

So does that mean I’m disconnecting myself from the world? Hell NO! I’ll be going to XTC’s when I can. Probably also London Clojurians if timing is OK, ACE next year and any other things that are bought to my attention…

Finally, so you want to contact me? The best bet is to LinkIn with me at http://www.linkedin.com/in/rwest. From there, we can work to email / phone / video … – whatever is necessary…

DailyRiczWest – behind the scenes v0.75

Ralph the News DogFirstly, let me introduce “Ralph the News Dog” (left) who replaces the generic cartoon I was using. If you follow me on Twitter, you’ll know that he passed away on the weekend, so I thought this would be a fitting tribute and better image to use. That shot was actually taken when he was 5, almost 5 years ago when we had not been in the UK for long and were living up in the Midlands… Anyway, to the code!

No major structural updates – they are coming… I’m well aware that I need to do some refactoring, but have not done it as the code fairly comfortably fits on a (slightly messy) page

use File::Copy;
use Time::Local;

$version = "0.75"; # RIP Ralphy

# Set up the Base Directories and files
$ifttt            = "/Users/rwest/Dropbox/ifttt";           # Location of If This Then That
$ifttt_favourites = "$ifttt/favourites.txt";
$twitter          = "/Users/rwest/Dropbox/Public/Twitter";  # Location of Public Twitter directory
$templates        = "$twitter/templates";

# Set up Date Strings
# Array of month strings for converting a month number in to a string
@month_text = qw( jan feb mar apr may jun jul aug sep oct nov dec );
# Get information for the day before this is invoked by subtracting seconds: 24h * 60m * 60s
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = (localtime(time - (24 * 60 * 60)));
$month = $month_text[$mon]; # Convert the month in to text from a number
$year += 1900;              # Perl Years are based from 1900, so we have to add this to get a real one

# Set up date based directories
$twitter_year = "$twitter/$year";                   # Twitter & Year directory
$twitter_year_month = "$twitter_year/$month";       # Twitter, Year & Month directory
# If the Twitter Year directory doesn't exist, make it
unless (-d $twitter_year)       { mkdir $twitter_year or die "Unable to create $twitter_year\n"};
unless (-d $twitter_year_month) { mkdir $twitter_year_month or die "Unable to create $twitter_year_month\n"};

# Assemble the String Date of form XX mon YEAR
$twitter_date = "$mday $month, $year";
print "DailyRiczWestDate : ", $twitter_date;

chdir("$twitter_year_month"); # Process in Year and Month directory under "Twitter"

# Assemble header
copy("$templates/header1.html", "header.html") or die "$templates/header1.html cannot be copied to $twitter_year_month";
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header1.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header2.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header2.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header3.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header3.html");
print HEADER $version; close(HEADER);
system("cat $templates/header4.html >> header.html");

# Process Body
copy($ifttt_favourites, "favourites.html") or die "$ifttt_favourites cannot be copied to $twitter_year_month";

open(FAVOURITES,"< favourites.html") or die("Cannot Read favourites.html");
open(BODY,"> body.html")             or die("Cannot Write body.html");

# Replace *BR* with <br> because ifttt will not pass through <br>
while (<FAVOURITES>) {
    s/\*BR\*/<br>/;
    print BODY;
}

close BODY; close FAVOURITES;

# Assemble Daily
system("cat header.html body.html $templates/footer.html > daily-$mday.html");
system("rm header.html favourites.html body.html");
$twodig_year = $year-2000;
system("mv $ifttt_favourites $ifttt/archive/favourites-$mday$month$twodig_year.txt");

There are a few new bits I’ve added here:

  • General rearrangement of initialisations in to more logical groupings
  • Addition of some more comments (always important in Perl!)
  • Handling any date rollover properly, i.e.
    • Computing the day before properly (1d = 24h * 60m * 60s) and then creating the date from that
    • Checking that directories exist and creating them if not
  • Finally, archiving the favourites.txt properly (I should probably structure this the same as the archives for the HTML pages, that way I Won’t be Repeating Myself ;-)

The next thing will be to do some basic structuring in Perl, which I haven’t done that much of as I don’t use it that often and usually for something quick and dirty. Seeing as my plans are much bigger than just what you’re seeing at the moment I’ll need this for the next major phase which will get me to the stage of looking like a real “online newspaper”

DailyRiczWest – behind the scenes v0.6

Newspaper MegaphoneHuoston – we have full automation! :-)

At last, I’ve bitten the bullet and actually automated the whole thing (for this month at least – more on that later). Up until now, I’ve been manually tweaking page info etc… but now I’ve broken the header in to 4 html fragments:

header1.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>

<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>DailyRiczWest -

header2.html

</title></head><body>
<big><big><span style="font-family: Arial Black;"></span></big></big>
<table style="text-align: left; width: 100%;" border="0" cellpadding="0" cellspacing="5">
<tbody>
<tr>
<td style="vertical-align: top; height: 1px; width: 1px;"><img style="width: 70px; height: 70px;" src="../../images/newspaperMegaphone.jpg" alt=""></td>
<td style="vertical-align: top;"><big><big><span style="font-family: Arial Black;">Daily</span></big></big><big><big><span style="font-family: Arial Black;">RiczWest</span></big></big><span style="font-family: Helvetica,Arial,sans-serif;"> for

header3.html

</span><br>
<span style="font-family: Helvetica,Arial,sans-serif;">=&gt; not
your
ordinary daily feed - version

header4.html

<br>
<small><span style="font-weight: bold;">NOTE: </span>Will be
continually evolving, i.e. improving quite quickly</small></span></td>
</tr>
</tbody>
</table>
<span style="font-family: Helvetica,Arial,sans-serif;"></span>

Ah yes, takes me back to the “good old days”, pre frameworks and a whole bunch of other crap ;-)

Anyway, to the code, which is starting to get a bit messy now, with the actual maintenance and archiving of files, along with the fully automated generation. It will definitely be time for some GitHubbing and refactoring, but for the mean time, here it is in all it’s glory (not quite ;)

#!/usr/bin/perl -w
# use strict;

# IMPORTANT NOTE: This WILL NOT handle month & year rollovers yet!

use File::Copy;
use Time::Local;

$version = "0.6";

$ifttt = "/Users/rwest/Dropbox/ifttt";
$ifttt_favourites = "$ifttt/favourites.txt";
$templates = "../../templates";

@month_text = qw( jan feb mar apr may jun jul aug sep oct nov dec );
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$month = $month_text[$mon];
$year += 1900;
$twitter_month = "/Users/rwest/Dropbox/Public/Twitter/2012/$month";
$daily_day = $mday-1;
$twitter_date = "$daily_day $month, $year";
print "DailyRiczWestDate : ", $twitter_date;

chdir("$twitter_month"); # Process in month directory

# Assemble header
copy("$templates/header1.html", "header.html") or die "$templates/header1.html cannot be copied to $twitter_month";
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header1.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header2.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header2.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header3.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header3.html");
print HEADER $version; close(HEADER);
system("cat $templates/header4.html >> header.html");

# Process Body
copy($ifttt_favourites, "favourites.html") or die "$ifttt_favourites cannot be copied to $twitter_month";

open(FAVOURITES,"< favourites.html") or die("Cannot Read favourites.html");
open(BODY,"> body.html") or die("Cannot Write body.html");

# Replace *BR* with <br> because ifttt will not pass through <br>
while (<FAVOURITES>) {
    s/\*BR\*/<br>/;
    print BODY;
}

close BODY; close FAVOURITES;

# Assemble Daily
system("cat header.html body.html $templates/footer.html > daily-$daily_day.html");
system("rm header.html favourites.html body.html");
system("mv $ifttt_favourites $ifttt/archive/favourites-$daily_day$month$year.txt");

Finally I have something that totally automatically:

  • Assembles the Page: Header (with date & version), Body and Footer
  • Archives the favourites.txt file should I need it in it’s raw form
  • Is designed to be invoked from cron on the day after the favorites are gathered

It’s far from ideal, as the code is no longer so clear and it won’t work properly when the month and year change, but I have 3 days to fix the month problem and 4 months to fix the year one ;-)

DailyRiczWest – behind the scenes v0.4

Newspaper MegaphoneDue to extraordinary circumstances (i.e. the hottest day this year) I didn’t quite get as far as I wanted with this release, but I did generate it from the command line in UNIX. I have 2 files under templates:

  • header.html
  • footer.html

and generated this file with the UNIX commands

  • sed -e 's/\*BR\*/\<br\>/g'
    ../../../../ifttt/favourites.txt > body.html

– this is the “Stream EDitor” which can perform various editing tricks from the command line. In this case, it’s doing a global search and replace for the string “*BR*” and replacing it with “<br>” to get the breaks in the HTML for reasonable formatting

  • cat ../../templates/header.html body.html
    ../../templates/footer.html > daily-18.html

– which basically ConcATenates the header, body and footer files in to one file

  • rm body.html

– that deletes the intermediate file

so there was no real manual editing, and I could obviously put this in a shell script with the above 3 commands.

DailyRiczWest – behind the scenes v0.3

Newspaper MegaphoneNot really much change for this version, so I thought I’d take a step back from the detail. As mentioned originally, apart from solving a problem for me, this is an experiment in “Social Coding”. So what do I mean by this? I’m certainly not up to the standards of the “Raspberry Pi Teenagers” who were in The Future of Programming – hint: it’s not a new IDE! but this is an experiment in my own context with the following overall objectives:

  • Solve my immediate need for a totally customisable personal paper
  • Get in to the habit of blogging more than once every 1 or so weeks
  • Play around with some social / web tech
  • Learn whatever else I can and have fun!

Oh – the “programming change” – that was pretty simple – I just used *br* instead of <br> in the template

<hr><br>
@{{UserName}} : {{Text}}*br*<br>
{{CreatedAt}} : <a href=”{{LinkToTweet}}”>
Link to Tweet</a> <a href=”{{FirstLinkUrl}}”>
Link to Article</a>*br*<br>

and did a search/replace on the generated file to put <br>’s in. So far, I’ve been “producing” the paper manually using KompoZer and for now the template is fairly sorted, so it’s time for some programming to get this puppy automated! Luckily, it’s the weekend…