Search results for: dailyriczwest

DailyRiczWest v0.9

Ralph MemorialIf you’ve been reading this blog for a while, you’ll recognise the title as ages ago I did a post DailyRiczWest – behind the scenes v0.1. That was quite a while and a bit has changed so I thought I’d do a quick update for people.

This program will take the concept of a personal paper like and spotify to it’s ultimate conclusion (well, for me at least ;). Instead of some “automated filter” picking things for you, the DRW works off any twitter items that I’ve favourited from any application (e.g. Flipboard, Tweetbot etc…). One advantage of this is that I can just tweet key items and pop everything else in a digest form. Here’s a link to an example of what was produced for yesterday.

DailyRiczWest-7oct13How does it work? Easy – it’s based upon an If This Then That recipe called Favourites to Dropbox for DailyRiczWest which just adds information on anything that you favourite to a file ifttt/favourites.txt in your Dropbox (assuming you have a “Public” folder).

There’s then a perl program and some templates which munge this in to an HTML digest of the previous days favourites. Eventually, I’d like to pull the content to the links summarized in to the digest but that’s a ways off. Being an agile kinda guy though, I’m doing this socially and releasing early and often.

Eventually I’d like to make this an easy to set up and stand alone project, but that’s a long ways off (as this is just a play project for me) but if you can wrap your head around how this works and set it up then let me know. If you have some problems, then feel free to ask…

DailyRiczWest – behind the scenes v0.75

Ralph the News DogFirstly, let me introduce “Ralph the News Dog” (left) who replaces the generic cartoon I was using. If you follow me on Twitter, you’ll know that he passed away on the weekend, so I thought this would be a fitting tribute and better image to use. That shot was actually taken when he was 5, almost 5 years ago when we had not been in the UK for long and were living up in the Midlands… Anyway, to the code!

No major structural updates – they are coming… I’m well aware that I need to do some refactoring, but have not done it as the code fairly comfortably fits on a (slightly messy) page

use File::Copy;
use Time::Local;

$version = "0.75"; # RIP Ralphy

# Set up the Base Directories and files
$ifttt            = "/Users/rwest/Dropbox/ifttt";           # Location of If This Then That
$ifttt_favourites = "$ifttt/favourites.txt";
$twitter          = "/Users/rwest/Dropbox/Public/Twitter";  # Location of Public Twitter directory
$templates        = "$twitter/templates";

# Set up Date Strings
# Array of month strings for converting a month number in to a string
@month_text = qw( jan feb mar apr may jun jul aug sep oct nov dec );
# Get information for the day before this is invoked by subtracting seconds: 24h * 60m * 60s
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = (localtime(time - (24 * 60 * 60)));
$month = $month_text[$mon]; # Convert the month in to text from a number
$year += 1900;              # Perl Years are based from 1900, so we have to add this to get a real one

# Set up date based directories
$twitter_year = "$twitter/$year";                   # Twitter & Year directory
$twitter_year_month = "$twitter_year/$month";       # Twitter, Year & Month directory
# If the Twitter Year directory doesn't exist, make it
unless (-d $twitter_year)       { mkdir $twitter_year or die "Unable to create $twitter_year\n"};
unless (-d $twitter_year_month) { mkdir $twitter_year_month or die "Unable to create $twitter_year_month\n"};

# Assemble the String Date of form XX mon YEAR
$twitter_date = "$mday $month, $year";
print "DailyRiczWestDate : ", $twitter_date;

chdir("$twitter_year_month"); # Process in Year and Month directory under "Twitter"

# Assemble header
copy("$templates/header1.html", "header.html") or die "$templates/header1.html cannot be copied to $twitter_year_month";
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header1.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header2.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header2.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header3.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header3.html");
print HEADER $version; close(HEADER);
system("cat $templates/header4.html >> header.html");

# Process Body
copy($ifttt_favourites, "favourites.html") or die "$ifttt_favourites cannot be copied to $twitter_year_month";

open(FAVOURITES,"< favourites.html") or die("Cannot Read favourites.html");
open(BODY,"> body.html")             or die("Cannot Write body.html");

# Replace *BR* with <br> because ifttt will not pass through <br>
while (<FAVOURITES>) {
    print BODY;

close BODY; close FAVOURITES;

# Assemble Daily
system("cat header.html body.html $templates/footer.html > daily-$mday.html");
system("rm header.html favourites.html body.html");
$twodig_year = $year-2000;
system("mv $ifttt_favourites $ifttt/archive/favourites-$mday$month$twodig_year.txt");

There are a few new bits I’ve added here:

  • General rearrangement of initialisations in to more logical groupings
  • Addition of some more comments (always important in Perl!)
  • Handling any date rollover properly, i.e.
    • Computing the day before properly (1d = 24h * 60m * 60s) and then creating the date from that
    • Checking that directories exist and creating them if not
  • Finally, archiving the favourites.txt properly (I should probably structure this the same as the archives for the HTML pages, that way I Won’t be Repeating Myself ;-)

The next thing will be to do some basic structuring in Perl, which I haven’t done that much of as I don’t use it that often and usually for something quick and dirty. Seeing as my plans are much bigger than just what you’re seeing at the moment I’ll need this for the next major phase which will get me to the stage of looking like a real “online newspaper”

DailyRiczWest – behind the scenes v0.6

Newspaper MegaphoneHuoston – we have full automation! :-)

At last, I’ve bitten the bullet and actually automated the whole thing (for this month at least – more on that later). Up until now, I’ve been manually tweaking page info etc… but now I’ve broken the header in to 4 html fragments:


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>DailyRiczWest -


<big><big><span style="font-family: Arial Black;"></span></big></big>
<table style="text-align: left; width: 100%;" border="0" cellpadding="0" cellspacing="5">
<td style="vertical-align: top; height: 1px; width: 1px;"><img style="width: 70px; height: 70px;" src="../../images/newspaperMegaphone.jpg" alt=""></td>
<td style="vertical-align: top;"><big><big><span style="font-family: Arial Black;">Daily</span></big></big><big><big><span style="font-family: Arial Black;">RiczWest</span></big></big><span style="font-family: Helvetica,Arial,sans-serif;"> for


<span style="font-family: Helvetica,Arial,sans-serif;">=&gt; not
ordinary daily feed - version


<small><span style="font-weight: bold;">NOTE: </span>Will be
continually evolving, i.e. improving quite quickly</small></span></td>
<span style="font-family: Helvetica,Arial,sans-serif;"></span>

Ah yes, takes me back to the “good old days”, pre frameworks and a whole bunch of other crap ;-)

Anyway, to the code, which is starting to get a bit messy now, with the actual maintenance and archiving of files, along with the fully automated generation. It will definitely be time for some GitHubbing and refactoring, but for the mean time, here it is in all it’s glory (not quite ;)

#!/usr/bin/perl -w
# use strict;

# IMPORTANT NOTE: This WILL NOT handle month & year rollovers yet!

use File::Copy;
use Time::Local;

$version = "0.6";

$ifttt = "/Users/rwest/Dropbox/ifttt";
$ifttt_favourites = "$ifttt/favourites.txt";
$templates = "../../templates";

@month_text = qw( jan feb mar apr may jun jul aug sep oct nov dec );
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$month = $month_text[$mon];
$year += 1900;
$twitter_month = "/Users/rwest/Dropbox/Public/Twitter/2012/$month";
$daily_day = $mday-1;
$twitter_date = "$daily_day $month, $year";
print "DailyRiczWestDate : ", $twitter_date;

chdir("$twitter_month"); # Process in month directory

# Assemble header
copy("$templates/header1.html", "header.html") or die "$templates/header1.html cannot be copied to $twitter_month";
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header1.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header2.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header2.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header3.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header3.html");
print HEADER $version; close(HEADER);
system("cat $templates/header4.html >> header.html");

# Process Body
copy($ifttt_favourites, "favourites.html") or die "$ifttt_favourites cannot be copied to $twitter_month";

open(FAVOURITES,"< favourites.html") or die("Cannot Read favourites.html");
open(BODY,"> body.html") or die("Cannot Write body.html");

# Replace *BR* with <br> because ifttt will not pass through <br>
while (<FAVOURITES>) {
    print BODY;

close BODY; close FAVOURITES;

# Assemble Daily
system("cat header.html body.html $templates/footer.html > daily-$daily_day.html");
system("rm header.html favourites.html body.html");
system("mv $ifttt_favourites $ifttt/archive/favourites-$daily_day$month$year.txt");

Finally I have something that totally automatically:

  • Assembles the Page: Header (with date & version), Body and Footer
  • Archives the favourites.txt file should I need it in it’s raw form
  • Is designed to be invoked from cron on the day after the favorites are gathered

It’s far from ideal, as the code is no longer so clear and it won’t work properly when the month and year change, but I have 3 days to fix the month problem and 4 months to fix the year one ;-)

DailyRiczWest – behind the scenes v0.5

Newspaper MegaphoneWhew! Bit of a Perl coding session, best summed up by by my twitter comment:

Apologies in advance to “The Ladies” but Perl is like a Burlesque Dancer. Very Sexy and when she does her stuff, you just have to admire ;-)

It’s been a while (probably 1y+) since I’ve coded Perl and it just reminds me what a fickle language that it is! After searching probably 20+ various google searches, I’ve finally figured it all out:

#!/usr/bin/perl -w
# use strict;

use File::Copy;

$ifttt_favourites = ".../Dropbox/ifttt/favourites.txt";
$twitter_month = ".../Dropbox/Public/Twitter/2012/apr";
$templates = "../../templates";

copy($ifttt_favourites, "body.html") or die "$ifttt_favourites cannot be copied to $twitter_month";

open(BODY,"< body.html") or die("Cannot Read body.html");
open(NEW_BODY,"> newbody.html") or die("Cannot Write newbody.html");

while (<BODY>) {
    print NEW_BODY;

close NEW_BODY;
close BODY;

system("cat $templates/header.html newbody.html $templates/footer.html > daily.html");
system("rm body.html newbody.html");

Seems so simple now, but there is so much “noise” about how to code Perl to do even simple tasks like search and replace that it’s a bit mind boggling!

Anyway, I’m almost there with the “automation” (the only thing I had to do was manually edit the Page Title Properties – and that was only because I’m running out of time). I now have a “program” that I can extend to automatically create the DailyRiczWest. The thing that really strikes me about Perl (yet again) is it’s awesome simplicity – it’s just so much work to reach that stage when you’re effectively a novice… Any comment’s on my code are welcome and I’ll soon put it up on GitHub when I have some time, but meanwhile, this at least makes generating my “Papers” a bit simpler.

How this will fit in with Twitter’s “grand new control-oriented plan” remains to be seen, but at the end of the day, I’m just getting a feed from If This Then That, while they are allowed to provide it… How this will fare under the new Twitter API Nazi Regime remains to be seen…

DailyRiczWest – behind the scenes v0.4

Newspaper MegaphoneDue to extraordinary circumstances (i.e. the hottest day this year) I didn’t quite get as far as I wanted with this release, but I did generate it from the command line in UNIX. I have 2 files under templates:

  • header.html
  • footer.html

and generated this file with the UNIX commands

  • sed -e 's/\*BR\*/\<br\>/g'
    ../../../../ifttt/favourites.txt > body.html

– this is the “Stream EDitor” which can perform various editing tricks from the command line. In this case, it’s doing a global search and replace for the string “*BR*” and replacing it with “<br>” to get the breaks in the HTML for reasonable formatting

  • cat ../../templates/header.html body.html
    ../../templates/footer.html > daily-18.html

– which basically ConcATenates the header, body and footer files in to one file

  • rm body.html

– that deletes the intermediate file

so there was no real manual editing, and I could obviously put this in a shell script with the above 3 commands.

DailyRiczWest – behind the scenes v0.3

Newspaper MegaphoneNot really much change for this version, so I thought I’d take a step back from the detail. As mentioned originally, apart from solving a problem for me, this is an experiment in “Social Coding”. So what do I mean by this? I’m certainly not up to the standards of the “Raspberry Pi Teenagers” who were in The Future of Programming – hint: it’s not a new IDE! but this is an experiment in my own context with the following overall objectives:

  • Solve my immediate need for a totally customisable personal paper
  • Get in to the habit of blogging more than once every 1 or so weeks
  • Play around with some social / web tech
  • Learn whatever else I can and have fun!

Oh – the “programming change” – that was pretty simple – I just used *br* instead of <br> in the template

@{{UserName}} : {{Text}}*br*<br>
{{CreatedAt}} : <a href=”{{LinkToTweet}}”>
Link to Tweet</a> <a href=”{{FirstLinkUrl}}”>
Link to Article</a>*br*<br>

and did a search/replace on the generated file to put <br>’s in. So far, I’ve been “producing” the paper manually using KompoZer and for now the template is fairly sorted, so it’s time for some programming to get this puppy automated! Luckily, it’s the weekend…

DailyRiczWest – behind the scenes v0.2

Newspaper MegaphoneToday I’ve changed the IFTTT rule so it outputs HTML and can hopefully just add a header and footer to almost automatically get a page. Or that was the plan…

IFTTT Uses a <br> to indicate a new line. Of course, if you know HTML, then you’ll know it’s used for the same purpose in that. In v0.1, I had to manually put in the newlines using an HTML editor – BTW KompoZer is the best free, cross platform editor I know of. So, I thought “I’ll outsmart IFTTT and use <br />” which is really the same thing, giving me the content of:

@{{UserName}} : {{Text}}<br /><br>
{{CreatedAt}} : <a href=”{{LinkToTweet}}”>
Link to Tweet</a> <a href=”{{FirstLinkUrl}}”>
Link to Article</a><br /><br>

Nope, it was too smart and realised that was the same thing, so I just ended up with double newlines! Back to manual edits…

As usual, I have an idea to get around that, so I’ll give it a go in v0.3

DailyRiczWest – behind the scenes v0.1

Newspaper MegaphoneWith the release of the first DailyRiczWest, if you’ve followed the link you’re probably thinking “what’s up here?”. I’ve noticed people putting up papers like and spotify and kind assumed that it was possible to link Favourites from Twitter in to these – it’s NOT! :-( I’ve searched the web and everyone wants to automatically feed you things.

Lately, I’ve discovered If This Then That and one of the things you can do is add to a file if something is favourited. So that’s step one. The next thing is to publish it, which is where you’re pre 2011 Dropbox account comes in. If you have one of these, then you’ll have a “Public” folder that you can share with no authentication – i.e. you can use it as a web server! That’s probably why it’s no longer available…

So, having my favourites.txt file I’ve manually copied it to a location and made it in to HTML. Quit a manual process, but it’s a start. The plan is to make this a social sharing project just in case it’s of interest to anyone. Even if it’s just me that’s interested, at least I have some documentation as the plan is to blog about each version within a day or so of it coming out.

The end goal is to end up with something like one of the curated papers, but that’s a fair way off, so in the Agile Spirit I’ll just released early and will hopefully be updating often, as long as work doesn’t get in the way…

Capturing your FlipBoard Magazine posts

FlipboardIconAs I’ve written about previously, although I’m not directly on Twitter at the moment, I am still curating information using FlipBoard Magazines and currently “publishing it” weekly as the Weekly RiczWest, which is basically a static page on WordPress with some links to my FlipBoard magazines that is available as a menu item at the top and is also tweeted using IFTTT (If This Then That).

DailyRiczWest-7oct13This is not really a satisfactory solution as I’d actually like to produce a “custom” magazine which contains only the content of the past week, which is the same idea I had with the Daily RiczWest. For all my gripes against Twitter, at least they make their content available via IFTTT (it wasn’t for a while, but now it’s back). Unfortunately, FlipBoard seems to be rather a closed system. This has obviously offended the “hacker” (and I mean that in a good way, not the common debased usage) so I’m looking to open it up and continue work on the core concept of taking a set of “highlighted posts” (by Flipboard Magazines, Twitter Favourites, …) and put them in to a custom stream and newspaper.

The first step is to free the information from FlipBoard to a neutral format that can be worked with. As usual, “the web” can help rescue us. FlipBoard have (marginally) “opened up” their magazines by giving them a URL which is updated and viewable by others on a browser. This gives us just enough room to actually capture the information, but how?

There are essentially three problems here:

  1. Sense changes to the web page
  2. Capture those changes
  3. Save them somewhere

3 will be solved by our old friend DropBox along with IFTTT for the high level orchestration, but it really could be anything.

For 1, before I found out I could only get my magazines “externally” from FlipBoard I was hoping they would have them exposed as RSS – unfortunately not :-( It’s a good idea though, so this lead to me searching for some way to convert web pages to RSS feeds. Luckily, there are a number of possible solutions, but I’ve chosen page2rss which is very easy to use (i.e. just give it the URL and it “does” the rest).

For 2, you simply set up an IFTTT Recipe that is triggered by your RSS feed and saves the ‘changes’ in the Web Page from 1

FlipBoard to DropBox 1

FlipBoard to DropBox 2

Once you have this set up you’ll get way too much information in your file as it seems to be a JSON Object – I’ve given a stripped down example below

"title":"Z Motherboards For Mini-ITX Builds - Reviews - Tom’s Hardware",
"excerptText":"Pre­vi­ous<p>The Mini-ITX Mar­ket Is Small, But Growing...<p>
ASRock ZE-ITX<p>ZE-ITX Software<p>ZE-ITX Firmware<p>Asus ZI-Deluxe<p>
ZI-Deluxe Software<p>ZI-Deluxe Firmware<p>EVGA Z Stinger<p>Z Stinger Software<p>Z …",
"authorDisplayName":"Thomas Soderstrom",

as in the next post I’ll be looking at parsing this for enough information to produce a WeeklyRiczWest newspaper. I’m hoping that we may even be able to use some of FlipBoards summary extraction features…

100th Post!

100th Blog PostI’m not the most prolific of bloggers, but after that first post on the 30th June 2012 “Hello world! Of Architecture and Change” this is now the 100th post – WOOHOO! As you can see above, I had a fairly modest celebration with a few Grenadier mates  – thanks guys ;-)

As with any Base-10 based moment, I think it’s time for a bit of reflection and cogitation… What better place to start than the “mission statement” from that first post:

If you’re interested in IT, Privacy, Science, Maths, Process, Systems
(of people and technical), Programming, Organisations and any other
topics that grab my attention, then you may want to follow this.

So how did I score?

The inevitable question is where now? Firstly, I think it’s time for a renewed mission statement:

To boldly go, where I've not gone before
To seek out new ways of being, living, thinking and working
To explore seemingly strange new worlds and ideas of other people
These are the voyages of me...
In physical, virtual, emotional & mental space: The final frontiers

Enterprise - NCC1701But a bit more specifically, what are my thoughts and plans for the future?

  • I like blogging about “anything”. I know this may be frustrating for some who very much stay on a particular “message topic”, as one day I’m talking about a product, another some programming, people or an organisation type. Well, that’s just me so I won’t change that…
  • I do want to increase the “technical content”, by which I mean around Architecture, Design and Programming. I got drawn down a people and process rabbit hole, which in some ways culminated with previous post on Toxic Waterfall. I’ll still blog in this area, but have a lot of technical ideas I want to explore and talk about along the way
  • I think topics like Maths, Science, Privacy and the like are probably better covered on Twitter, which I’ll be getting back on to next year after I’ve returned from a well earned break in Australia

Sam-TaegeukOr put more simply, I’ll probably stick with The Classics of Transformation:

  • People (& Organisations)
  • Process (with People and Organisations)
  • Technology (to implement Processes for People in Organisations)

but most importantly in a humane context