Speedlinks Using Del.icio.us and Yahoo Pipes

Last week I started adding SpeedLinks to this blog. I’ve been looking for a good excuse to experiment with Yahoo Pipes for some time now. So, I wanted to find out if I could take the concept of the original Perl script I found, and create a pipe to do all the work (with little, to no programming). This original script basically does the following:

1) Allows you to store your Speedlinks with del.icio.us using a common tag like ‘speedlink’.

2) Fetches Speedlinks using the Del.icio.us API on a daily basis (tag=’speedlink’, date=’YYYY-MM-DD)

3) Parses the XML output using XPath, creates an HTML summary of the speedlinks, and automatically posts a blog entry.

For those of you that haven’t yet heard about Yahoo Pipes (from their website):

Pipes is an interactive data aggregator and manipulator that lets you mashup your favorite online data sources.
Like Unix pipes, simple commands can be combined together to create output that meets your needs:
* combine many feeds into one, then sort, filter and translate to create your ultimate custom feed.
* geocode your favorite feeds and browse the items on an interactive map.
* remix your favorite data sources and use the Pipe to power a new application.
* build custom vertical search pages that are impossible with ordinary search engines.
* power badges on your web site.
* extend your web site by accessing the JSON or RSS output from Pipes.

Pipes uses a slick visual editor that allows you to fetch and manipulate data sources (JSON and RSS), add user-defined inputs (dates, strings, numbers, urls), and filter the content as you wish (foreach w/ regex, sorting, content analysis, etc).

My first task in fetching my links from del.icio.us was figuring the best way to query. There were some small challenges:

1) The del.icio.us API requires basic authentication, and outputs in XML.
2) Even if I could parse the XML with pipes, you can query posts by date, but it is in UTC format (conversion is simple, but I don’t want to bother). And, there is no way to supply a date range (other than 24 hours from the date you provided).
3) The del.icio.us RSS feed (http://del.icio.us/rss/ericblue76/speedlink) doesn’t allow you to specifiy a date range.

Out of these limitations, my first Pipe was created – RSS Filter By Date. As you can see below, I defined 3 inputes (the RSS url, start date, and end date). Pipes has a nifty date format, so you can supply in a variety of syntaxes: simple english (today, yesterday), multiple formats (DD/MM/YY, YYYY-MM-DD), and also includes timezone (PST, PDT, UTC). The feed is fetched, and the date inputs are normalized to utime (which is UNIX epoch seconds), and each feed entry is parsed to see if the post is within the specified range.

The following screenshot illustrates what the pipe looks like after you run it. You can choose to download the query results in RSS or JSON format:

Now that the ‘RSS Filter By Date’ pipe was created, I proceeded to create a new pipe ‘Del.icio.us Speedlink Filter‘ that accepts your del.icio.us username, speedlink tag, and date range (default is UTC today, but can be converted to any date range) and linked to the RSS filter pipe.

Unfortunately, one major limitation I found with Pipe’s RSS output is that it strips a good bit of the original metadata associated with the original feed. This is discussed in the following post. Although you do get the original link, name, and description, the tag info is lost. If you want to include tag info in the speedlink results, you’ll have to download as JSON and parse/format to HTML.

Although using Yahoo Pipes to parse and fetch my del.icio.us links definitely works and is nifty, it doesn’t really simplify the original approach. But, this experiment was certainly fun, and it definitely shows the potential of Pipes to allow users to easily create, and manipulate data with little to no programming required.