Tagged: dailyriczwest

DailyRiczWest v0.9

Ralph MemorialIf you’ve been reading this blog for a while, you’ll recognise the title as ages ago I did a post DailyRiczWest – behind the scenes v0.1. That was quite a while and a bit has changed so I thought I’d do a quick update for people.

This program will take the concept of a personal paper like paper.li and spotify to it’s ultimate conclusion (well, for me at least ;). Instead of some “automated filter” picking things for you, the DRW works off any twitter items that I’ve favourited from any application (e.g. Flipboard, Tweetbot etc…). One advantage of this is that I can just tweet key items and pop everything else in a digest form. Here’s a link to an example of what was produced for yesterday.

DailyRiczWest-7oct13How does it work? Easy – it’s based upon an If This Then That recipe called Favourites to Dropbox for DailyRiczWest which just adds information on anything that you favourite to a file ifttt/favourites.txt in your Dropbox (assuming you have a “Public” folder).

There’s then a perl program and some templates which munge this in to an HTML digest of the previous days favourites. Eventually, I’d like to pull the content to the links summarized in to the digest but that’s a ways off. Being an agile kinda guy though, I’m doing this socially and releasing early and often.

Eventually I’d like to make this an easy to set up and stand alone project, but that’s a long ways off (as this is just a play project for me) but if you can wrap your head around how this works and set it up then let me know. If you have some problems, then feel free to ask…

DailyRiczWest – behind the scenes v0.75

Ralph the News DogFirstly, let me introduce “Ralph the News Dog” (left) who replaces the generic cartoon I was using. If you follow me on Twitter, you’ll know that he passed away on the weekend, so I thought this would be a fitting tribute and better image to use. That shot was actually taken when he was 5, almost 5 years ago when we had not been in the UK for long and were living up in the Midlands… Anyway, to the code!

No major structural updates – they are coming… I’m well aware that I need to do some refactoring, but have not done it as the code fairly comfortably fits on a (slightly messy) page

use File::Copy;
use Time::Local;

$version = "0.75"; # RIP Ralphy

# Set up the Base Directories and files
$ifttt            = "/Users/rwest/Dropbox/ifttt";           # Location of If This Then That
$ifttt_favourites = "$ifttt/favourites.txt";
$twitter          = "/Users/rwest/Dropbox/Public/Twitter";  # Location of Public Twitter directory
$templates        = "$twitter/templates";

# Set up Date Strings
# Array of month strings for converting a month number in to a string
@month_text = qw( jan feb mar apr may jun jul aug sep oct nov dec );
# Get information for the day before this is invoked by subtracting seconds: 24h * 60m * 60s
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = (localtime(time - (24 * 60 * 60)));
$month = $month_text[$mon]; # Convert the month in to text from a number
$year += 1900;              # Perl Years are based from 1900, so we have to add this to get a real one

# Set up date based directories
$twitter_year = "$twitter/$year";                   # Twitter & Year directory
$twitter_year_month = "$twitter_year/$month";       # Twitter, Year & Month directory
# If the Twitter Year directory doesn't exist, make it
unless (-d $twitter_year)       { mkdir $twitter_year or die "Unable to create $twitter_year\n"};
unless (-d $twitter_year_month) { mkdir $twitter_year_month or die "Unable to create $twitter_year_month\n"};

# Assemble the String Date of form XX mon YEAR
$twitter_date = "$mday $month, $year";
print "DailyRiczWestDate : ", $twitter_date;

chdir("$twitter_year_month"); # Process in Year and Month directory under "Twitter"

# Assemble header
copy("$templates/header1.html", "header.html") or die "$templates/header1.html cannot be copied to $twitter_year_month";
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header1.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header2.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header2.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header3.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header3.html");
print HEADER $version; close(HEADER);
system("cat $templates/header4.html >> header.html");

# Process Body
copy($ifttt_favourites, "favourites.html") or die "$ifttt_favourites cannot be copied to $twitter_year_month";

open(FAVOURITES,"< favourites.html") or die("Cannot Read favourites.html");
open(BODY,"> body.html")             or die("Cannot Write body.html");

# Replace *BR* with <br> because ifttt will not pass through <br>
while (<FAVOURITES>) {
    s/\*BR\*/<br>/;
    print BODY;
}

close BODY; close FAVOURITES;

# Assemble Daily
system("cat header.html body.html $templates/footer.html > daily-$mday.html");
system("rm header.html favourites.html body.html");
$twodig_year = $year-2000;
system("mv $ifttt_favourites $ifttt/archive/favourites-$mday$month$twodig_year.txt");

There are a few new bits I’ve added here:

  • General rearrangement of initialisations in to more logical groupings
  • Addition of some more comments (always important in Perl!)
  • Handling any date rollover properly, i.e.
    • Computing the day before properly (1d = 24h * 60m * 60s) and then creating the date from that
    • Checking that directories exist and creating them if not
  • Finally, archiving the favourites.txt properly (I should probably structure this the same as the archives for the HTML pages, that way I Won’t be Repeating Myself ;-)

The next thing will be to do some basic structuring in Perl, which I haven’t done that much of as I don’t use it that often and usually for something quick and dirty. Seeing as my plans are much bigger than just what you’re seeing at the moment I’ll need this for the next major phase which will get me to the stage of looking like a real “online newspaper”

DailyRiczWest – behind the scenes v0.6

Newspaper MegaphoneHuoston – we have full automation! :-)

At last, I’ve bitten the bullet and actually automated the whole thing (for this month at least – more on that later). Up until now, I’ve been manually tweaking page info etc… but now I’ve broken the header in to 4 html fragments:

header1.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>

<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>DailyRiczWest -

header2.html

</title></head><body>
<big><big><span style="font-family: Arial Black;"></span></big></big>
<table style="text-align: left; width: 100%;" border="0" cellpadding="0" cellspacing="5">
<tbody>
<tr>
<td style="vertical-align: top; height: 1px; width: 1px;"><img style="width: 70px; height: 70px;" src="../../images/newspaperMegaphone.jpg" alt=""></td>
<td style="vertical-align: top;"><big><big><span style="font-family: Arial Black;">Daily</span></big></big><big><big><span style="font-family: Arial Black;">RiczWest</span></big></big><span style="font-family: Helvetica,Arial,sans-serif;"> for

header3.html

</span><br>
<span style="font-family: Helvetica,Arial,sans-serif;">=&gt; not
your
ordinary daily feed - version

header4.html

<br>
<small><span style="font-weight: bold;">NOTE: </span>Will be
continually evolving, i.e. improving quite quickly</small></span></td>
</tr>
</tbody>
</table>
<span style="font-family: Helvetica,Arial,sans-serif;"></span>

Ah yes, takes me back to the “good old days”, pre frameworks and a whole bunch of other crap ;-)

Anyway, to the code, which is starting to get a bit messy now, with the actual maintenance and archiving of files, along with the fully automated generation. It will definitely be time for some GitHubbing and refactoring, but for the mean time, here it is in all it’s glory (not quite ;)

#!/usr/bin/perl -w
# use strict;

# IMPORTANT NOTE: This WILL NOT handle month & year rollovers yet!

use File::Copy;
use Time::Local;

$version = "0.6";

$ifttt = "/Users/rwest/Dropbox/ifttt";
$ifttt_favourites = "$ifttt/favourites.txt";
$templates = "../../templates";

@month_text = qw( jan feb mar apr may jun jul aug sep oct nov dec );
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$month = $month_text[$mon];
$year += 1900;
$twitter_month = "/Users/rwest/Dropbox/Public/Twitter/2012/$month";
$daily_day = $mday-1;
$twitter_date = "$daily_day $month, $year";
print "DailyRiczWestDate : ", $twitter_date;

chdir("$twitter_month"); # Process in month directory

# Assemble header
copy("$templates/header1.html", "header.html") or die "$templates/header1.html cannot be copied to $twitter_month";
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header1.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header2.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header2.html");
print HEADER $twitter_date; close(HEADER);
system("cat $templates/header3.html >> header.html");
open (HEADER, ">> header.html") or die ("Cannot Append to header.html - after header3.html");
print HEADER $version; close(HEADER);
system("cat $templates/header4.html >> header.html");

# Process Body
copy($ifttt_favourites, "favourites.html") or die "$ifttt_favourites cannot be copied to $twitter_month";

open(FAVOURITES,"< favourites.html") or die("Cannot Read favourites.html");
open(BODY,"> body.html") or die("Cannot Write body.html");

# Replace *BR* with <br> because ifttt will not pass through <br>
while (<FAVOURITES>) {
    s/\*BR\*/<br>/;
    print BODY;
}

close BODY; close FAVOURITES;

# Assemble Daily
system("cat header.html body.html $templates/footer.html > daily-$daily_day.html");
system("rm header.html favourites.html body.html");
system("mv $ifttt_favourites $ifttt/archive/favourites-$daily_day$month$year.txt");

Finally I have something that totally automatically:

  • Assembles the Page: Header (with date & version), Body and Footer
  • Archives the favourites.txt file should I need it in it’s raw form
  • Is designed to be invoked from cron on the day after the favorites are gathered

It’s far from ideal, as the code is no longer so clear and it won’t work properly when the month and year change, but I have 3 days to fix the month problem and 4 months to fix the year one ;-)

DailyRiczWest – behind the scenes v0.5

Newspaper MegaphoneWhew! Bit of a Perl coding session, best summed up by by my twitter comment:

Apologies in advance to “The Ladies” but Perl is like a Burlesque Dancer. Very Sexy and when she does her stuff, you just have to admire ;-)

It’s been a while (probably 1y+) since I’ve coded Perl and it just reminds me what a fickle language that it is! After searching probably 20+ various google searches, I’ve finally figured it all out:

#!/usr/bin/perl -w
# use strict;

use File::Copy;

$ifttt_favourites = ".../Dropbox/ifttt/favourites.txt";
$twitter_month = ".../Dropbox/Public/Twitter/2012/apr";
$templates = "../../templates";

chdir("$twitter_month");
copy($ifttt_favourites, "body.html") or die "$ifttt_favourites cannot be copied to $twitter_month";

open(BODY,"< body.html") or die("Cannot Read body.html");
open(NEW_BODY,"> newbody.html") or die("Cannot Write newbody.html");

while (<BODY>) {
    s/\*BR\*/<br>/;
    print NEW_BODY;
}

close NEW_BODY;
close BODY;

system("cat $templates/header.html newbody.html $templates/footer.html > daily.html");
system("rm body.html newbody.html");

Seems so simple now, but there is so much “noise” about how to code Perl to do even simple tasks like search and replace that it’s a bit mind boggling!

Anyway, I’m almost there with the “automation” (the only thing I had to do was manually edit the Page Title Properties – and that was only because I’m running out of time). I now have a “program” that I can extend to automatically create the DailyRiczWest. The thing that really strikes me about Perl (yet again) is it’s awesome simplicity – it’s just so much work to reach that stage when you’re effectively a novice… Any comment’s on my code are welcome and I’ll soon put it up on GitHub when I have some time, but meanwhile, this at least makes generating my “Papers” a bit simpler.

How this will fit in with Twitter’s “grand new control-oriented plan” remains to be seen, but at the end of the day, I’m just getting a feed from If This Then That, while they are allowed to provide it… How this will fare under the new Twitter API Nazi Regime remains to be seen…

DailyRiczWest – behind the scenes v0.4

Newspaper MegaphoneDue to extraordinary circumstances (i.e. the hottest day this year) I didn’t quite get as far as I wanted with this release, but I did generate it from the command line in UNIX. I have 2 files under templates:

  • header.html
  • footer.html

and generated this file with the UNIX commands

  • sed -e 's/\*BR\*/\<br\>/g'
    ../../../../ifttt/favourites.txt > body.html

– this is the “Stream EDitor” which can perform various editing tricks from the command line. In this case, it’s doing a global search and replace for the string “*BR*” and replacing it with “<br>” to get the breaks in the HTML for reasonable formatting

  • cat ../../templates/header.html body.html
    ../../templates/footer.html > daily-18.html

– which basically ConcATenates the header, body and footer files in to one file

  • rm body.html

– that deletes the intermediate file

so there was no real manual editing, and I could obviously put this in a shell script with the above 3 commands.

DailyRiczWest – behind the scenes v0.3

Newspaper MegaphoneNot really much change for this version, so I thought I’d take a step back from the detail. As mentioned originally, apart from solving a problem for me, this is an experiment in “Social Coding”. So what do I mean by this? I’m certainly not up to the standards of the “Raspberry Pi Teenagers” who were in The Future of Programming – hint: it’s not a new IDE! but this is an experiment in my own context with the following overall objectives:

  • Solve my immediate need for a totally customisable personal paper
  • Get in to the habit of blogging more than once every 1 or so weeks
  • Play around with some social / web tech
  • Learn whatever else I can and have fun!

Oh – the “programming change” – that was pretty simple – I just used *br* instead of <br> in the template

<hr><br>
@{{UserName}} : {{Text}}*br*<br>
{{CreatedAt}} : <a href=”{{LinkToTweet}}”>
Link to Tweet</a> <a href=”{{FirstLinkUrl}}”>
Link to Article</a>*br*<br>

and did a search/replace on the generated file to put <br>’s in. So far, I’ve been “producing” the paper manually using KompoZer and for now the template is fairly sorted, so it’s time for some programming to get this puppy automated! Luckily, it’s the weekend…

DailyRiczWest – behind the scenes v0.2

Newspaper MegaphoneToday I’ve changed the IFTTT rule so it outputs HTML and can hopefully just add a header and footer to almost automatically get a page. Or that was the plan…

IFTTT Uses a <br> to indicate a new line. Of course, if you know HTML, then you’ll know it’s used for the same purpose in that. In v0.1, I had to manually put in the newlines using an HTML editor – BTW KompoZer is the best free, cross platform editor I know of. So, I thought “I’ll outsmart IFTTT and use <br />” which is really the same thing, giving me the content of:

<hr><br>
@{{UserName}} : {{Text}}<br /><br>
{{CreatedAt}} : <a href=”{{LinkToTweet}}”>
Link to Tweet</a> <a href=”{{FirstLinkUrl}}”>
Link to Article</a><br /><br>

Nope, it was too smart and realised that was the same thing, so I just ended up with double newlines! Back to manual edits…

As usual, I have an idea to get around that, so I’ll give it a go in v0.3

DailyRiczWest – behind the scenes v0.1

Newspaper MegaphoneWith the release of the first DailyRiczWest, if you’ve followed the link you’re probably thinking “what’s up here?”. I’ve noticed people putting up papers like paper.li and spotify and kind assumed that it was possible to link Favourites from Twitter in to these – it’s NOT! :-( I’ve searched the web and everyone wants to automatically feed you things.

Lately, I’ve discovered If This Then That and one of the things you can do is add to a file if something is favourited. So that’s step one. The next thing is to publish it, which is where you’re pre 2011 Dropbox account comes in. If you have one of these, then you’ll have a “Public” folder that you can share with no authentication – i.e. you can use it as a web server! That’s probably why it’s no longer available…

So, having my favourites.txt file I’ve manually copied it to a location and made it in to HTML. Quit a manual process, but it’s a start. The plan is to make this a social sharing project just in case it’s of interest to anyone. Even if it’s just me that’s interested, at least I have some documentation as the plan is to blog about each version within a day or so of it coming out.

The end goal is to end up with something like one of the curated papers, but that’s a fair way off, so in the Agile Spirit I’ll just released early and will hopefully be updating often, as long as work doesn’t get in the way…