In Dec ‘09 I shared a blog post about how to synchronize the scrapbook data between multiple computers. This was the first major step to sharing data between multiple devices, but still lacked some of the ubiquity that I desired.  In a nutshell I’ve made 2 major enhancements to Scrapbook:

1. An email ‘bridge’ to Scrapbook so I can email links from any device (PC, iPhone, iPad) and have them saved by Scrapbook
2. A centralized web-interface to browse/search/filter my scrapbook data.

I’ll start off with the less visually-stunning hack (email bridge), but by far the craftier of two.

Hack #1 – Scrapbook Email Interface

Whenever I began synchronizing my Scrapbook data between the 2-3 computers this solved a huge problem with being able to save webpages from anywhere.  Since 2009 a lot has changed, and devices like iPhone and iPad (yes, Apple fan boy to a degree) have changed the way we consume news.  Recently I’ve been using apps on the iPad like Zite and Flipboard to consolidate my Twitter, Facebook, and Googler Reader feeds into a single personalized newspaper.  This means that now > 50% of my reading time is spent from a device that has no visibility into my Scrapbook data.  I simply wanted a way to automatically email a link (built nativily into these apps) and have it automagically saved into my Scrapbook folder.  I could have simply cut corners and wrote a script to hand-edit the Scrapbook RDF Files and save the web page using something like wget or curl.  But, it just wouldn’t be the same…. I want the webpage saved EXACTLY as Firefox would normally render and save it.

This poses a bit of a technical challenge, since Scrapbook runs inside Firefox and there’s no native way to interface with a plugin running inside a browser.  After researching a number of approaches, I came across 2 Firefox plugins that let you build interfaces inside firefox (http, telent, etc.) that actually let you control the browser and execute Javascript.  Of the 2 plugins; POW and MozRepl, I decided to go with POW (Plain Old Webserver).  Both plugins are wicked cool in the sense that they’re non-traditional and very powerful.  POW runs a webserver inside firefox and let’s you run your ’server-side’ scripts as Javascript.  I’ve basically written a server process that runs INSIDE the client and executes XPCOM/Javascript to control the web browser windows and invokes the Scrapbook plugin API directly.

The setup process is simple:

1. Setup and install the POW and Scrapbook plugins in your browser
2. Configure POW to run a desired port and create a new directory /scrapbook/
3. Copy the index.sjs (server-side javascript) to this new /scrapbook/ directory
4. Setup a new email box or alias (e.g. yourusername+scrapbook@gmail.com)
5. Either run scrapbook2email.pl manually or run as a CRON job every couple minutes
6. Simply send emails to your new Scrapbook email, run the email script, and watch your pages be saved automatically

At a high-level this is accomplished with 2 scripts:

Email Interface script (Perl)

This script uses IMAP to retrieve scrapbook email requests from a designated folder. Along with doing basic sender/recipient validation, the script is also aware of plain text/multipart messages.  Once the email request is parsed, the link of the requested web page to be saved will be extracted.  Given the request URL the script will then contact the POW server and pass the requested URL (e.g. http://127.0.0.1:6670/scrapbook/?url=http://yourwebpagetobesaved.com/?articleID=3q4e3332).  Note that this version of the script requires that Firefox/POW be running and makes no attempt to launch for you.

For a copy of the script click here (GitHub).

Scrapbook/POW Bridge (Server-Side Javascript)

This script does the heavy lifting, and is essentially running at the other end of the POW server URL (http://127.0.0.1:6670/scrapbook/). Once the requested URL is detected the browser will spawn a new tab, automatically execute the Scrapbook Capture request, and save the webpage to a new top-level folder (e.g. Unfiled/MM-DD-YYYY). This script was tested with Scrapbook v.1.3.7.

For a copy of the script click here (GitHub).

It’s nifty now to email a link to my Scrapbook Bot and wihin a couple minutes a little notify popup shows in Firefox indicating my page was saved.

Hack #2 – Scrapbook Browser

This code was actually written back in Dec ‘09 after I wrote the synchronize blog post (and around the time I wrote the Document Viewer), however I haven’t shared until now.  What I’ve done is write a simple Perl/JQuery web app that used Simile’s Exhibit to view Scrapbook data in a tile, table, or timeline.  This interface also has a file/folder view so you can browse snippets just like you can through the native Scrapbook plugin interface within Firefox.

Here are some screenshots:

Tile View

Timeline View

Table View

Folder View

To download the code click here (GitHub).

I never really considered myself a “highlighter” until a couple years ago.  Back in school I would, on occasion, highlight some interesting passages while doing homework or reading books and jot them down later.  More often then not though many of those highlights would go to waste.  After all, what good are highlighting interesting bits of text if you don’t use them later?  My highlight compulsion increased about 6 years ago when I dove head first into mindmapping and starting experimenting with a technique called MMOST (Mind Map Organic Study Technique).  In a nutshell, MMOST is a strategy for quickly digesting books and summarizing what you’ve learned into a mindmap so you can recall or reference at a later date.  For a great intro to the MMOST technique, check out the post on How to Understand a Business Book in Four Hours.  What does highlighting have to do with MMOST?  While I’m reading a book I’ll highlight the passages that stick out to me and use those as the basis for creating the mindmap summary.  It can take a lot of time, but the process of highlighting, reviewing, and creating the mindmap can significantly improve your recall and what you get out of a book (or any research project).

Another big change happened earlier this year when I started using an iPad.  I’ve been gradually accumulating more digital books (using PDFs and purchasing books through Amazon using Kindle).  After using Kindle for a short time I was blown away by the feature that let’s you highlight book passages and get summaries of the highlighted text and page number (The direct URL is http://kindle.amazon.com/your_highlights.  This is REALLY useful for accelerating the summarizing process and the beauty of it is that it’s automatic – the extraction just works!  Around the time I started using Kindle for iPad I discovered a fantastic PDF Document reader called GoodReader.

GoodReader is a full-featured document reader with some powerful features.  Not only can you take all of your documents on the go, you can access remotely using WebDAV, Google Docs, DropBox, Email, and other online services.  Starting a couple months ago it got even better by supporting PDF highlighting and annotations.  I thought to myself, “Hey, it would be great if I could somehow extract all my highlighted text just like Kindle.  I could TRIPLE the number of books I read and create summaries for almost all of them!”.  It turns out this IS possible, but it is no where near as simple as I initially hoped.  I dove down the deep rabit hole of reviewing the ~ 1,000 page Adobe PDF specification, hacked and tinkered with Perl and Java code, reviewed numerous open source and commercial offerings, and have emerged (slightly scathed but wiser) with some good solutions.

The Challenge

I won’t get into the nitty-gritty details here, but what would seem a simple operation of extracting highlighted text from a PDF turns out to be exceedingly difficult depending on what strategy you use.  In fact, as near as I can tell, there is no existing open source or commercial solution that can reliably extract the 100% text accurately from all documents.  The main challenge with PDF is that it isn’t a markup language like HTML that will explicitly tell you how text should be rendered.  For example:

This is an <b>example</b> <highlight>sentence that I would like to highlight</highlight>.

The PDF format, while parsable, uses concepts like dictionaries, objects, streams and coordinate systems that tell PDF readers how to correctly render the doc. What this means is that things like annotations (notes) and highlights are rendered separately from the text itself.  The best way to visualize this is to think of the highlighted PDF as having 2 distinct layers: the top layer is the highlight itself and the bottom layer is the text.  The straightforward strategy is to simply say: “Find the X,Y coordinates of the region of highlight, then find the X,Y coordinates of all text in that same region and simply copy it”.  Well, the unfortunate complexity is that in order to find the coordinates of the text you also have to take into consideration the font type and size of the font.  After many hours of hacking with only minimal success, I’ve concluded that this method is not currently possible without a lot of additional coding.  And, unless somebody can point me in the right direction, I haven’t found any open source or commercial offerings that do this.  OK, so you’re probably wondering why I’ve made you read this much of the post only to tell you it’s not technically possible.  It is possible, just using a slightly different method.

The Solutions

It turns out that you can automatically extract the highlight with 100% accuracy, but there is a caveat that requires a little more manual work.  It sounds much more painful than it really is.  The trick is to not only highlight the passage of text, but also copy the text and paste as an annotation (note) on top of the highlight.  For GoodReader it’s simply a matter of a couple extra clicks.  And for people who use Adobe Acrobat or Acrobat Reader, there is an option in most versions to automatically copy/paste text into a note whenever you select text to highlight (Go to Settings -> Commenting Preferences -> “Copy selected text into Highlight, Cross-Out, and Underline comment pop-ups.”).  Here’s how you accomplish using GoodReader as of v3.2.0:

1. Select the text you would like to highlight and select Copy.  As soon as you click Copy, the menu option above the text will remain.
2. Next select the Highlight option.  At this point the text will now be highlighted.
3. Tap the highlighted text and select the Open option.  A note dialogue will appear.
4. Hold down for 2 sections on the note until the Paste option appears and select.  Click Save.

Basically 6 quick clicks/taps and you’re done.  It’s not ideal, but certainly a good trade-off if it means you get to extract automatically and have 100% reliability.  Now, there are a couple options for easily extracting your highlights.

Option 1 – Use a PDF Reader to create highlight summaries

If you have the money, Adobe Acrobat has many features that let you view and print all of your annotations (notes, highlights, etc.).  Although not significantly cost prohibitive most people (myself included) don’t really want to spend money if you can find a comparable free or open source solution.  Adobe Acrobat Reader (the free version most people use) does allow you to view the highlights in a summary pane, but doesn’t allow you to extract and print (You’ll notice that if you don’t create the annotated note with your highlight the entry will show blank.)  The best free PDF viewer that I experimented with is Foxit Reader and it allows you to easily create a PDF summary of your highlights.  Simply go to Comments -> Summary Comments and you’ll be prompted to save a new PDF file that only contains the highlighted text along with the page number.

Option 2 – Programmatically extract highlights

For those inclined to hack, there are a couple open source options for parsing PDF files.  I first started experimenting with a great Perl module called CAM::PDF.  After a few weekends of tinkering around and subsequently needing to dig into the official Adobe PDF specificaiton I realized how complicated PDF parsing, rendering, and text extraction can be.  CAM::PDF does make it easy parse the overall structure of the document and extract text for an entire page, but it is very difficult to extract for exact coordinates (for a number of technical reasons).  At this point I was still trying to solve the problem with the original strategy of extracing text by x,y coordinates, and after researching for countless hours I realized my open source options were limited.  My next step was to experiment with PDFBox, an Apache open source JAVA PDF library.  After some searching I was very excited to at least scratch the surface and get preliminary results of text extraction based on the highlight x,y coordinates.  I soon discovered that needing to take the font style, orientation, and spacing into consideration to grab the exact text would prove to be time consuming.  I haven’t yet found other examples, or reached out on the mailing list, but I’m sure with sufficient determination and time this could be done.  Not wanting to devote this amount of time right now to solve this problem, I opted to go for the pragmatic solution of saving the note and extracting that.  For those interested, I’ve attached some very simple test code that will extract the annotated comment and I’ve included commented out code for doing very basic (and not yet accurate) extraction based on region/coordinates.  When I have more time I may make this a standalone executable so you can run from the command-line and bulk extract highlights from multiple documents:

Code block

import java.awt.geom.Rectangle2D;
import java.io.File;
import java.util.List;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.common.PDRectangle;
import org.apache.pdfbox.pdmodel.interactive.annotation.PDAnnotation;
import org.apache.pdfbox.util.PDFTextStripperByArea;
public class ExtractHighlights {
public static void main(String args[]) {
try {
PDDocument pddDocument = PDDocument.load(new File("sample.pdf"));
List allPages = pddDocument.getDocumentCatalog().getAllPages();
for (int i = 0; i < allPages.size(); i++) {
int pageNum = i + 1;
PDPage page = (PDPage) allPages.get(i);
List<PDAnnotation> la = page.getAnnotations();
if (la.size() < 1) {
continue;
}
System.out.println("Total annotations = " + la.size());
System.out.println("\nProcess Page " + pageNum + "...");
// Just get the first annotation for testing
PDAnnotation pdfAnnot = la.get(0);
System.out.println("Annot type = " + pdfAnnot.getSubtype());
System.out.println("Modified date = " + pdfAnnot.getModifiedDate());
System.out.println("Rectangle = " + pdfAnnot.getRectangle());
// Sample code taken from Canoo unit test - extractAnnotations
// See https://svn.canoo.com/trunk/webtest/src/main/java/com/canoo/webtest/plugins/pdftest/htmlunit/pdfbox/PdfBoxPDFPage.java
// Experimental - Not completely working since rectangle doesn't take font size/spacing into account
// PDFTextStripperByArea stripper = new PDFTextStripperByArea();
// stripper.setSortByPosition(true);
//
// PDRectangle rect = pdfAnnot.getRectangle();
// float x = rect.getLowerLeftX() - 1;
// float y = rect.getUpperRightY() - 1;
// float width = rect.getWidth() + 2;
// float height = rect.getHeight() + rect.getHeight() / 4;
// int rotation = page.findRotation();
// if (rotation == 0) {
//     PDRectangle pageSize = page.findMediaBox();
//       y = pageSize.getHeight() - y;
//}
//
// Rectangle2D.Float awtRect = new Rectangle2D.Float(x, y, width, height);
// stripper.addRegion(Integer.toString(0), awtRect);
// stripper.extractRegions(page);
//
// System.out.println("Getting text from region = " + awtRect + "\n");
// System.out.println(stripper.getTextForRegion(Integer.toString(0)));
System.out.println("Getting text from comment = " + pdfAnnot.getContents());
}
pddDocument.close();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}

Of all the APIs I reviewed PDFBox appears to be one of the best: enumerating through the annotations is easy, extracting the note is just as simple, and the basic API is there to extract highlights with no need for the note (just be prepared to dig in and do some work).  I also spent some time researching Adobe’s Javascript API and saw some forum posts where a person had mentioned they wrote a JavaScript plugin for Adobe Acrobat Reader that extracted the highlight without the need for the notes.  However, I could not find a working example.  With further research I’m sure this could be another option.

For the short-term, my practical solution is going to use Foxit Reader to create the highlight summaries.  Foxit works under Wine (linux) and I’ve been able to share my GoodReader docs over WiFi and mount that Goodreader share as a WebDav folder.  This means that once I’m done reading and highlighting a PDF I can easily open up in FoxitReader without needing to copy anything, generate the highlight summary, and save back to my Documents folder.  Longer-term I’ll probably elaborate on the PDFBox code and write a program to automatically extract the highlights and save as text, XML, or HTML.

Other Links of Interest

• My PDF Bookmarks from Del.icio.us (TONS of good links found during research)
• Python – Scrape Highlighted (Not portable, but uses a combo of Python, AppleScript and SkimPDF for Mac)
• Python – PDF Miner
• PDF Can Opener (Inspects PDF docs)
• Acrobat Exhibit Highlighter (Some highlight tools using Javascript to enhance Acrobat)
• Topic Grazer (Windows – helps with text extraction)

Happy Highlighting!

Being able to access your personal information/knowledge from multiple places is the ultimate realization of total information ubiquity.  Being able to access all of your personal bookmarks, notes, contact information, journal entries, and research data from any computer is obviously useful.  Being able to access all of your personal knowledge from a handheld device like an iPhone is absolutely exciting!  Without sounding totally nostalgic, this type of portability is in a large part a modern-day realization of what Vannevar Bush had envisioned in his article on the Memex (“As We May Think“).

“Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and, to coin one at random, “memex” will do. A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.”

Currently, my personal wiki and other research data (bookmarks, pdfs, mindmaps, etc) are stored on my private server (accessible only behind my firewall).  Unless I enable SSH access, my wiki content is not generally available from the Internet and I have no way to easily access remotely.  I’ve been thinking for a while on the best approach for making this data completely portable.  After some experimentation, I’ve found an easy method for making my personal wiki completely accessible in offline mode right on my iPhone.  At a high-level all you really need to do are 2 things:

1) Find software that can take a snapshot of your wiki content and make it available for offline viewing

2) Find software that lets you save a copy of your snapshot wiki, store on the iPhone, and view in a web browser (actually both on the phone itself and another PC)

Creating a backup of your wiki

There are a lot of applications out there that act as ’spiders’ that crawl your website and save local copies of your pages so you can view in offline mode (no need for an Internet connection).  After trying a handful, one of the better applications I tested was HTTrack (available for Windows and Linux). I should note that I really did try to make this work with Scrapbook.  To date I’ve used Scrapbook to capture copies of pages with no problems.  However, it turns out that backing up a wiki pushes it to its limit… Scrapbook only does one serial http connection at a time, doesn’t have a configurable delay between requests (default is 1 sec and this takes too long), filtering options are not extensive enough, and the process of dynamically updating the HTML to support relatives links took way too long.  In the end, HTTrack ended up being the best solution for a complete wiki backup.

HTTrack is a highly configurable crawler that allows you to create a complete snapshot of your wiki (Mediawiki in my case).  Crawling a wiki turns out to be a little more complicated that your typical website.  Because wiki’s offer a number of functions (editing of pages, viewing history, printing and exporting in other formats) there are certain links that should not be included in the backup.  After some trial and error, I discovered that since I used Semantic Mediawiki I needed to be even more careful with the links I wanted to include (many of the Special and Property pages took FOREVER to index).

I tried the windows version of HTTrack (even under Wine on Linux) and the web client version as well.   However, was not completely impressed with how it worked.  What I wanted was a command-line script to run the backup.  Luckily, I found a couple websites that have used HTTrack for this purpose and decided to use for my own needs.  Here is a copy of the script i used to create the offline snapshot of my wiki:

Code block

#! /bin/sh
# Inspired by blogpost from http://www-public.it-sudparis.eu/~berger_o/weblog/2008/05/30/offline-backup-mediawiki-with-httrack/
# -w mirror web sites (--mirror)
# -O backup directory
# -%P extended parsing, attempt to parse all links, even in unknown tags or Javascript (%P0 don't use) (--extended-parsing[=N])
# -N0 Saves files like in site Site-structure (default)
# -s0 follow robots.txt and meta robots tags (0=never,1=sometimes,* 2=always) (--robots[=N])
# -p7 Expert options, priority mode: 7 > get html files before, then treat other files
# -S Expert option, stay on the same directory
# -a Expert option, stay on the same address
# -K0 keep original links (e.g. http://www.adr/link) (K0 *relative link, K absolute links, K3 absolute URI links) (--keep-links[=N]
# -A25000 maximum transfer rate in bytes/seconds (1000=1kb/s max) (--max-rate[=N])
# -F user-agent field (-F "user-agent name") (--user-agent )
# -%s update hacks: various hacks to limit re-transfers when updating (identical size, bogus response..) (--updatehack)
# -x Build option, replace external html links by error pages
# -%x Build option, do not include any password for external password protected websites (%x0 include) (--no-passwords)
site=wiki:8080/memex
topurl=http://$site
backupdir=~/websites/memex
httrack -c4 -w $topurl/Special:Allpages \
-O "$backupdir" -%P -N0 -s0 -p7 -S -a -K0 \
-F "Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)" \
-%s -x -%x  \
"+*$site/index.php?*" \
"+*$site/mindmap*" \
"-*Special*" \
"-*Property*" \
"-$site/index.php?title=Property:*" \
"-$site/index.php?title=Special:*" \
"-*$site/Discussion:*" \
"-*$site/Help*" \
"-*/docs/*" \
"-*/wikifiles/*" \
"-*month=*&year=*" \
"-*action=edit" \
"-*action=formedit" \
"-*action=history" \
"-*printable=yes" \
"-*oldid=*" \
"+*$site/images/*" \
"+*.css" \
"+*.js"

The best feature of HTTrack is that it will download all content, including Javascript and Flash, and update all links and make them relative.  This way the entire website can be viewed offline and made portable.  Overall, my entire wiki backup was ~ 30MB of markup and content (of course, excluding all audio and video).  And, in the future I need to come up with a solution for exporting my mindmaps.  Since iPhone does not yet support flash I’ll need some other way to allow for embedded viewing of my mindmap content.  Anyways, at this point all of my content is now ready for copying to my iPhone.

Storing my Wiki on my iPhone

OK, this is the really nifty part.  The one thing I really missed about my old 60GB IPod Video was the ability to mount it over USB and use it just like an external hard drive.  I used to haul around TONs of my data and could easily share between Windows, Linux and Mac.  Unfortunately when the iPhones and iPod Touch’s came out, you could no longer mount the iPhone and copy files (without hackery of course).  Luckily there are a number apps that let you use your iPhone as a storage device.  One of the BEST applications out there is an app called AirSharing.

With Air Sharing, you can:

• Mount your iPhone or iPod touch as a wireless drive on a Mac, windows, or Linux computer, over Wi-Fi, or connect from your computer’s web browser.
• Drag-drop files between your iPhone or iPod touch and your computers.
• View documents in many common formats.

What’s really useful is that you can mount your iPhone using WebDAV and transfer files just like a regular drive.  The incredibly cool bonus is that you can also access your content from another computer.  If you’re connected to the same Wi-Fi network, you can use any PC to browse (e.g. http://iphone-local:8080/wiki/) and access your content just like it was on the original server.  For an added layer of security, while you’re on the go you can setup an AdHoc wireless network and connect privately between your computer and the iPhone to access your personal knowledge base.

Of course, accessing the content on your iPhone from another PC is an added bonus.  The real power in this solution is the ability to browse your wiki on the iPhone without needing any Internet access (3G Or WiFi).  Simply open up your Airsharing app and browse directly to your wiki folder and click on index.html.  Wala!, your browsing your personal wiki just like usual.

I exported the majority of the text content from my wiki (preserving the original formating, with Javascript support).  In fact, I even shared my digital scrapbook that I blogged about last week.  but you can also choose to export your entire document collection and multimedia files (video, MP3s, etc).  This is incredibly useful for taking your knowledge on the go and having all of your data RIGHT at your finger tips.  Here are some screenshots of my personal knowledge manager wiki right on my iPhone:

All Articles

Workout Journal

Learning

Documents

Background

For my own personal knowledge management setup, I’m very interested in tracking a number of different ‘things’:

* Documents – PDFs, word documents, mindmaps, etc.

* Notes – Journal entries, book summaries, personal notes (think wiki text)

* Links – Bookmarks (personal or social sites like del.icio.us)

* Multimedia – Audio / Video

* Snippets – Captured web pages (full or partiallly snipped content)

When I first mentioned my ‘Digital Scrapbook’, I wasn’t dropping any hints about me having any crafty hobbies, I generally refer to my system for storing Snippets as my Scrapbook.  This name is no doubt in large part due to the fact that I’ve been using the popular Firefox plugin ScrapBook to manage my digital snippets for a few years now.

ScrapBook is a fantastic solution for storing local copies of web pages for research (with highlighting, editing, and annotation), saving snips of important sections of sites, recording purchase confirmations or receipts, and saving your travel itineraries.  One major thing it has been lacking though is the ability to synchronize or share the Scrapbook with other computers. I use multiple computers (a couple laptops: Mac & Window and a central desktop: Linux) so my goal is to have consistent and up to date data between all systems.  And, up until now, I’ve had no way to integrate this save data into my wiki-based knowledge management system.

I started investigating a solution for this a number of months ago and stumbled across a related (and powerful) research tool called Zotero.  I haven’t had a chance to use Zotero in depth, but one new feature in the beta version that stuck out to me was the ability to synchronize your data with a remote server.  On the surface this feature looks good (and probably is for most people – data sync to Zotero server and webdav support for documents), but I was looking for a solution where I have more control over where the data is hosted.  Although I’m usually not concerned with hosting my data with most providers, I often save private financial information in my Scrapbook (credit reports, financial statements, account numbers, etc.) so I’d like to have control over where the data is saved and how it’s encrypted.  Further research eventually sparked a few ideas for a solution.

Synchronizing and Sharing ScrapBook Data

I decided to find a way to explore a setup using some file sharing/sync services after reading an article on syncing Scrapbook using Dropbox.  I had never used Dropbox before and after giving it a brief testdrive it looked very promising.  Hey, you get a 2GB account for free so that’s definitely an added bonus!  Although Dropbox has some killer features (a big one being an iPhone app to access your files), I opted to experiment with another sync service.  I’ve been using JungleDisk for a couple years as my Amazon-S3 backed offsite backup solution, and was curious if this could be used.  After downloading the latest version (3.0.2 for Linux)  I discovered that it now supports file/directory synchronization between computers.  After about 10-15 minutes of setup and file syncing I had a working solution between my laptop and desktop computers.  Here’s what you’ll need to do:

Step 1: Download and install the latest version of the Scrapbook plugin for Firefox on your 1st computer.  For a good quick intro/tutorial to Scrapbook, check out this video from Lifehacker.

Step 2: Setup an alternate Scrapbook location that resides outside of your Firefox profile directory (Prefrences -> Organize -> Save data to)

Step 3: Setup your preferred sync solution and use the directory provided in Step 2.  I preferred JungleDisk for my setup, but there are other services like Dropbox, Box.net, SugarSync, etc.  Check out the Activty Owner wiki for a detailed list of sync services.  And, although I haven’t personally tried yet, I’m sure there are some other non-hosted open source sync solutions like Unison (cross-platform) that could be used.

Step 4: For your 2nd (or subsequent computers) repeat steps 1 through 3.

Wiki Integration (Extra Credit)

OK, for me this was the icing on the cake.  Since my Scrapbook data is now on the same computer as my wiki I thought it would be nifty to somehow integrate directly into some of my wiki pages.  I found out that Scrapbook supports the ability to export your Scrapbook hierarchy as a tree in HTML (from Scrapbook Sidebar: Tools -> Output Tree as HTML).  Although this isn’t completely automatic (yet) this gave me the the content I needed to add to my wiki.  Now, since wikis by there very nature dont’ typically allow you to embed other HTML pages I needed to find a way to make this work.

Step 1: Setup a directory on your webserver to serve content from your Scrapbook directory (setup in Step 2 above) (e.g. http://yourwebsite/scrapbook).  This can either be on the same server as your wiki or another, it doesn’t really matter.

Step 2: Verify the output of the directory tree looks good.  If you enabled frames, the URL should be something like http://yourwebsite/scrapbook/tree/frame.html.

Step 3: For MediaWiki users there are various ways to directly embed pages in your wiki content.  I found that the AnySite extension did the trick for me.  Enable the extension, pick a wiki page where you want to display your ScrapBook data and you are set!  For example, here is my content:

* Link to [http://wiki:8080/wikifiles/scrapbook/tree/frame.html ScrapBook Tree]
<anyweb mywidth=”1024″ myheight=”768″>http://wiki:8080/wikifiles/scrapbook/tree/frame.html</anyweb>
[[Category:Documents]]

I also discovered a somewhat related research project at Stanford called Vispedia.

“Imagine taking a unique journey through the depths of your mind and stepping into a world of limitless possibilities where every one of your thoughts, habits, behaviors, beliefs, decisions and actions are revealed to you in vivid detail. Within this scenario “causes“ are identified and “effects“ are measured, effectively unlocking the hidden patterns that are continuously re-creating your life experience on a daily basis.”

What is the MasterMind Matrix?

“In the simplest terms, the MasterMind Matrix Chart presents you with a visual Concept Map of Your Personality. It progressively unlocks and expands your understanding of your current patterns of behavior, while helping you develop new habits that can guide your life in the direction of your choosing.”

You can download the free version of the map, or read about it in depth and buy the premium version ($50USD).

Most traditional personal knowledge management (PKM) or personal information management (PIM) applications offer the same basic set of features:

* Storage of notes and documents
* Search functionality and keyword/tagging capability
* Outline view in a traditional hierarchy, or user-defined views
* Task management, calendar, and contact management (mainly PIM, not KM)

These are essential features, however don’t offer too much to the more visually-inclined knowledge junkies. For visual learners and information visualization fans, having a graphical representation of knowledge and seeing how things relate is a must have feature.  Luckily, in the past few years there has been a rise in the number of knowledge management applications that offer this capability.  The following is a list of interesting /unique / effective tools for knowledge management and information visualization (not listed in any particular order):

15. DeepaMehta

Link: http://www.deepamehta.de/
Platforms: Win, Mac, Linux
Cost: Free (Open Source)

DeepaMehta is a “networked semantic desktop” that replaces the traditional computer desktop. DeepaMehta rids the user from dealing with applications, files and directories. Instead, the DeepaMehta user arranges information of any kind and origin into supportive topic maps. Topics may be e.g. projects, emails, webpages, notes, articles, contacts, or meetings. Users define their own topic types. Old-fashioned applications, windows and files are no longer in the sight of the user, but the meaningful relationships between real world topics.

14. Tinderbox

Link: http://www.eastgate.com/Tinderbox/
Platforms: Mac
Cost: $179

Tinderbox stores and organizes your notes, plans, and ideas. It can help you analyze and understand them. And Tinderbox helps you share ideas through Web journals and web logs. Tinderbox maps your notes as you make them. Build relationships by arranging notes, organizing them with shape and color, linking them. Tinderbox lets you record ideas quickly and keep them where you’ll find them again when you need them.

13. Vue

Link: http://vue.tufts.edu/
Platforms: Win, Mac, Linux
Cost: Free (Open Source)

At its core, the Visual Understanding Environment (VUE) is a concept and content mapping application, developed to support teaching, learning and research and for anyone who needs to organize, contextualize, and access digital information. Using a simple set of tools and a basic visual grammar consisting of nodes and links, faculty and students can map relationships between concepts, ideas and digital content.

12. eyePlorer

Link: http://www.eyeplorer.com/eyePlorer/
Platforms: Web
Cost: Free

EyePlorer allows you to explore and process knowledge. Search engines help you find links and documents – they require you to follow these links and open the respective document in order to access information. eyePlorer.com, powered by vionto, provides immediate access to facts. It visualizes facts as well as relationships between facts. Furthermore, eyePlorer.com allows you to collect, process and publish interesting bits of information. eyePlorer.com is a visual knowledge workbench.

11. BeeDocs Timeline

Link: http://www.beedocs.com
Platforms: Mac
Cost: $65

Bee Docs Timeline is software for Mac OS X that makes it easy for you to present historical events in a way that reveals connections and clarifies relationships.

Make timeline charts of world history, family trees, fictional events or business deadlines. Timelines can help you understand and present history with new perspective!

See: http://eric-blue.com/2009/04/14/how-to-create-stunning-3d-timelines/

Post continued, click below to navigate to the next page….

I have to admit, when I first came across this software last week I was instantly impressed.  I’ve had a fascination with software that allow you to visualize and comprehend information in different and exciting ways.  Bee Docs Timeline, an application for Mac OS X (sorry Windows and Linux fans), is a great example of how to create and present smooth, professional looking timelines in a relatively easy fashion.

Here’s a brief summary of their product:

“Bee Docs Timeline is software for Mac OS X that makes it easy for you to present historical events in a way that reveals connections and clarifies relationships.

Make timeline charts of world history, family trees, fictional events or business deadlines. Timelines can help you understand and present history with new perspective!”

This application has a number of features, the most compelling being the ability to view the timeline in 3D, and even save it as a movie (Quicktime).  Not only can you create and import traditional timeline data, but you can also import from iPhoto, iTunes, and RSS feeds.  For illustration purposes, I created a 3D timeline video of my blog posts for the last few months (albeit less frequent posts than normal thanks to Twitter).

“One of the great things about an iPhone or Android phone is being able to play Pacman while stuck in line at the post office. Sometimes though, we yearn for something more than just playing games or watching videos.

What if you could also access literature’s greatest works, such as Emma and The Jungle Book, right from your phone? Or, some of the more obscure gems such as Mark Twain’s hilarious travelogue, Roughing It? Today we are excited to announce the launch of a mobile version of Google Book Search, opening up over 1.5 million mobile public domain books in the US (and over half a million outside the US) for you to browse while buying your postage.”

Continue reading at http://booksearch.blogspot.com/2009/02/15-million-books-in-your-pocket.html

Abstract. The use of semantic representations in document-oriented
environments – as formal annotations or embedded instances of a formal
knowledge base – is seen as an enabling technology for intelligent services
which may help knowledge workers in tasks like finding, structuring, or
assessing information. A very high level of formalization may even more
directly support problem solving, e.g., by the application of inferencing
services. A coupling of textual and formal representations in document-
centered knowledge work raises, amongst others, two questions: i) How
can the acquisition of formal knowledge in such an environment be facili-
tated? ii) How can the potential complexity of formal annotations during
a document’s life and use cycle be adequately handled?
We present the Mymory workbench as an approach to tackle these chal-
lenges. Mymory is based on a semantic wiki system and supports manual
as well as automated annotations of wiki documents. These annotations
can be framed by automatically obtained models of the user’s work con-
text, thereby a situation-oriented structuring of the annotations is being
achieved which can be exploited in semantic search and adapted docu-
ment presentation. In the paper, the general approach and architecture
of the system is being described. Example use cases show how document-
centered knowledge work can look like in such an environment and give
an idea which services can be possible with contextualized fine-grained
annotations.
2008_EKAW_Mymory