Knowledge To Go: Put Your Wiki On Your IPhone

Building my own personal knowledge manager has been quite a journey.  Over the last couple years I’ve taken a “piecemeal” approach and slowly built up the features of my system one component at a time.  One major feature that has always been on my mind is data portability.  Last week I wrote an article on how to sync your digial scrapbook between multiple computers and even sync to your wiki.  This feature had me thinking about how I could take portability to the next level.
Being able to access your personal information/knowledge from multiple places is the ultimate realization of total information ubiquity.  Being able to access all of your personal bookmarks, notes, contact information, journal entries, and research data from any computer is obviously useful.  Being able to access all of your personal knowledge from a handheld device like an iPhone is absolutely exciting!  Without sounding totally nostalgic, this type of portability is in a large part a modern-day realization of what Vannevar Bush had envisioned in his article on the Memex (“As We May Think“).

“Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and, to coin one at random, “memex” will do. A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.”

Currently, my personal wiki and other research data (bookmarks, pdfs, mindmaps, etc) are stored on my private server (accessible only behind my firewall).  Unless I enable SSH access, my wiki content is not generally available from the Internet and I have no way to easily access remotely.  I’ve been thinking for a while on the best approach for making this data completely portable.  After some experimentation, I’ve found an easy method for making my personal wiki completely accessible in offline mode right on my iPhone.  At a high-level all you really need to do are 2 things:

1) Find software that can take a snapshot of your wiki content and make it available for offline viewing
2) Find software that lets you save a copy of your snapshot wiki, store on the iPhone, and view in a web browser (actually both on the phone itself and another PC)

Creating a backup of your wiki
There are a lot of applications out there that act as ‘spiders’ that crawl your website and save local copies of your pages so you can view in offline mode (no need for an Internet connection).  After trying a handful, one of the better applications I tested was HTTrack (available for Windows and Linux). I should note that I really did try to make this work with Scrapbook.  To date I’ve used Scrapbook to capture copies of pages with no problems.  However, it turns out that backing up a wiki pushes it to its limit… Scrapbook only does one serial http connection at a time, doesn’t have a configurable delay between requests (default is 1 sec and this takes too long), filtering options are not extensive enough, and the process of dynamically updating the HTML to support relatives links took way too long.  In the end, HTTrack ended up being the best solution for a complete wiki backup.
HTTrack is a highly configurable crawler that allows you to create a complete snapshot of your wiki (Mediawiki in my case).  Crawling a wiki turns out to be a little more complicated that your typical website.  Because wiki’s offer a number of functions (editing of pages, viewing history, printing and exporting in other formats) there are certain links that should not be included in the backup.  After some trial and error, I discovered that since I used Semantic Mediawiki I needed to be even more careful with the links I wanted to include (many of the Special and Property pages took FOREVER to index).
I tried the windows version of HTTrack (even under Wine on Linux) and the web client version as well.   However, was not completely impressed with how it worked.  What I wanted was a command-line script to run the backup.  Luckily, I found a couple websites that have used HTTrack for this purpose and decided to use for my own needs.  Here is a copy of the script i used to create the offline snapshot of my wiki:
[codesyntax lang=”text”]
#! /bin/sh
# Inspired by blogpost from http://www-public.it-sudparis.eu/~berger_o/weblog/2008/05/30/offline-backup-mediawiki-with-httrack/
# -w mirror web sites (–mirror)
# -O backup directory
# -%P extended parsing, attempt to parse all links, even in unknown tags or Javascript (%P0 don’t use) (–extended-parsing[=N])
# -N0 Saves files like in site Site-structure (default)
# -s0 follow robots.txt and meta robots tags (0=never,1=sometimes,* 2=always) (–robots[=N])
# -p7 Expert options, priority mode: 7 > get html files before, then treat other files
# -S Expert option, stay on the same directory
# -a Expert option, stay on the same address
# -K0 keep original links (e.g. http://www.adr/link) (K0 *relative link, K absolute links, K3 absolute URI links) (–keep-links[=N]
# -A25000 maximum transfer rate in bytes/seconds (1000=1kb/s max) (–max-rate[=N])
# -F user-agent field (-F “user-agent name”) (–user-agent )
# -%s update hacks: various hacks to limit re-transfers when updating (identical size, bogus response..) (–updatehack)
# -x Build option, replace external html links by error pages
# -%x Build option, do not include any password for external password protected websites (%x0 include) (–no-passwords)
site=wiki:8080/memex
topurl=http://$site
backupdir=~/websites/memex
httrack -c4 -w $topurl/Special:Allpages \
-O “$backupdir” -%P -N0 -s0 -p7 -S -a -K0 \
-F “Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)” \
-%s -x -%x  \
“+*$site/index.php?*” \
“+*$site/mindmap*” \
“-*Special*” \
“-*Property*” \
“-$site/index.php?title=Property:*” \
“-$site/index.php?title=Special:*” \
“-*$site/Discussion:*” \
“-*$site/Help*” \
“-*/docs/*” \
“-*/wikifiles/*” \
“-*month=*&year=*” \
“-*action=edit” \
“-*action=formedit” \
“-*action=history” \
“-*printable=yes” \
“-*oldid=*” \
“+*$site/images/*” \
“+*.css” \
“+*.js”
[/codesyntax]
The best feature of HTTrack is that it will download all content, including Javascript and Flash, and update all links and make them relative.  This way the entire website can be viewed offline and made portable.  Overall, my entire wiki backup was ~ 30MB of markup and content (of course, excluding all audio and video).  And, in the future I need to come up with a solution for exporting my mindmaps.  Since iPhone does not yet support flash I’ll need some other way to allow for embedded viewing of my mindmap content.  Anyways, at this point all of my content is now ready for copying to my iPhone.
Storing my Wiki on my iPhone

OK, this is the really nifty part.  The one thing I really missed about my old 60GB IPod Video was the ability to mount it over USB and use it just like an external hard drive.  I used to haul around TONs of my data and could easily share between Windows, Linux and Mac.  Unfortunately when the iPhones and iPod Touch’s came out, you could no longer mount the iPhone and copy files (without hackery of course).  Luckily there are a number apps that let you use your iPhone as a storage device.  One of the BEST applications out there is an app called AirSharing.

With Air Sharing, you can:

• Mount your iPhone or iPod touch as a wireless drive on a Mac, windows, or Linux computer, over Wi-Fi, or connect from your computer’s web browser.
• Drag-drop files between your iPhone or iPod touch and your computers.
• View documents in many common formats.

What’s really useful is that you can mount your iPhone using WebDAV and transfer files just like a regular drive.  The incredibly cool bonus is that you can also access your content from another computer.  If you’re connected to the same Wi-Fi network, you can use any PC to browse (e.g. http://iphone-local:8080/wiki/) and access your content just like it was on the original server.  For an added layer of security, while you’re on the go you can setup an AdHoc wireless network and connect privately between your computer and the iPhone to access your personal knowledge base.
Of course, accessing the content on your iPhone from another PC is an added bonus.  The real power in this solution is the ability to browse your wiki on the iPhone without needing any Internet access (3G Or WiFi).  Simply open up your Airsharing app and browse directly to your wiki folder and click on index.html.  Wala!, your browsing your personal wiki just like usual.
I exported the majority of the text content from my wiki (preserving the original formating, with Javascript support).  In fact, I even shared my digital scrapbook that I blogged about last week.  but you can also choose to export your entire document collection and multimedia files (video, MP3s, etc).  This is incredibly useful for taking your knowledge on the go and having all of your data RIGHT at your finger tips.  Here are some screenshots of my personal knowledge manager wiki right on my iPhone:
All Articles

Workout Journal

Learning

Documents