In Dec ‘09 I shared a blog post about how to synchronize the scrapbook data between multiple computers. This was the first major step to sharing data between multiple devices, but still lacked some of the ubiquity that I desired.  In a nutshell I’ve made 2 major enhancements to Scrapbook:

1. An email ‘bridge’ to Scrapbook so I can email links from any device (PC, iPhone, iPad) and have them saved by Scrapbook
2. A centralized web-interface to browse/search/filter my scrapbook data.

I’ll start off with the less visually-stunning hack (email bridge), but by far the craftier of two.

Hack #1 – Scrapbook Email Interface

Whenever I began synchronizing my Scrapbook data between the 2-3 computers this solved a huge problem with being able to save webpages from anywhere.  Since 2009 a lot has changed, and devices like iPhone and iPad (yes, Apple fan boy to a degree) have changed the way we consume news.  Recently I’ve been using apps on the iPad like Zite and Flipboard to consolidate my Twitter, Facebook, and Googler Reader feeds into a single personalized newspaper.  This means that now > 50% of my reading time is spent from a device that has no visibility into my Scrapbook data.  I simply wanted a way to automatically email a link (built nativily into these apps) and have it automagically saved into my Scrapbook folder.  I could have simply cut corners and wrote a script to hand-edit the Scrapbook RDF Files and save the web page using something like wget or curl.  But, it just wouldn’t be the same…. I want the webpage saved EXACTLY as Firefox would normally render and save it.

This poses a bit of a technical challenge, since Scrapbook runs inside Firefox and there’s no native way to interface with a plugin running inside a browser.  After researching a number of approaches, I came across 2 Firefox plugins that let you build interfaces inside firefox (http, telent, etc.) that actually let you control the browser and execute Javascript.  Of the 2 plugins; POW and MozRepl, I decided to go with POW (Plain Old Webserver).  Both plugins are wicked cool in the sense that they’re non-traditional and very powerful.  POW runs a webserver inside firefox and let’s you run your ’server-side’ scripts as Javascript.  I’ve basically written a server process that runs INSIDE the client and executes XPCOM/Javascript to control the web browser windows and invokes the Scrapbook plugin API directly.

The setup process is simple:

1. Setup and install the POW and Scrapbook plugins in your browser
2. Configure POW to run a desired port and create a new directory /scrapbook/
3. Copy the index.sjs (server-side javascript) to this new /scrapbook/ directory
4. Setup a new email box or alias (e.g. yourusername+scrapbook@gmail.com)
5. Either run scrapbook2email.pl manually or run as a CRON job every couple minutes
6. Simply send emails to your new Scrapbook email, run the email script, and watch your pages be saved automatically

At a high-level this is accomplished with 2 scripts:

Email Interface script (Perl)

This script uses IMAP to retrieve scrapbook email requests from a designated folder. Along with doing basic sender/recipient validation, the script is also aware of plain text/multipart messages.  Once the email request is parsed, the link of the requested web page to be saved will be extracted.  Given the request URL the script will then contact the POW server and pass the requested URL (e.g. http://127.0.0.1:6670/scrapbook/?url=http://yourwebpagetobesaved.com/?articleID=3q4e3332).  Note that this version of the script requires that Firefox/POW be running and makes no attempt to launch for you.

For a copy of the script click here (GitHub).

Scrapbook/POW Bridge (Server-Side Javascript)

This script does the heavy lifting, and is essentially running at the other end of the POW server URL (http://127.0.0.1:6670/scrapbook/). Once the requested URL is detected the browser will spawn a new tab, automatically execute the Scrapbook Capture request, and save the webpage to a new top-level folder (e.g. Unfiled/MM-DD-YYYY). This script was tested with Scrapbook v.1.3.7.

For a copy of the script click here (GitHub).

It’s nifty now to email a link to my Scrapbook Bot and wihin a couple minutes a little notify popup shows in Firefox indicating my page was saved.

Hack #2 – Scrapbook Browser

This code was actually written back in Dec ‘09 after I wrote the synchronize blog post (and around the time I wrote the Document Viewer), however I haven’t shared until now.  What I’ve done is write a simple Perl/JQuery web app that used Simile’s Exhibit to view Scrapbook data in a tile, table, or timeline.  This interface also has a file/folder view so you can browse snippets just like you can through the native Scrapbook plugin interface within Firefox.

Here are some screenshots:

Tile View

Timeline View

Table View

Folder View

To download the code click here (GitHub).

What is JIT?

The JIT is an advanced JavaScript infovis toolkit based on 5 papers about different information visualization techniques.
The JIT implements advanced features of information visualization like Treemaps (with the slice and dice and squarified methods), an adapted visualization of trees based on the Spacetree, a focus+context technique to plot Hyperbolic Trees, and a radial layout of trees with advanced animations (RGraph).

Read more at http://blog.thejit.org/javascript-information-visualization-toolkit-jit/