On Saturday I stopped by the Apple store and picked up an UP device by Jawbone. The UP is a wristband and iPhone app that will track your fitness and sleep activity and help encourage you to live a more active life style. It’s very similar to the Fitbit device, and lets you track:

• steps
• distance
• calories burned
• active/inactive times
• workouts
• sleep (awake/light/deep sleep)

So far I’m really impressed with both the wristband and iPhone app.  However, I was a little disappointed the wristband didn’t support ANT or Bluetooth for data synchronization.  The data sync is unique and “low impact” (battery-wise) in that it plugs into the iPhone via the headphone jack and uploads data directly to the app. After some digging I was surprised to see no way to send or save the data.  I was curious if the data was just saved to the app, or also synchronized with another server.

As with most of my Quantified Self gadets (Fitbit, Zeo, Withings, Neurosky, etc.) my first reaction was to “free my data”.  My understanding is that Jawbone has exposed an API to partners, and will soon be releasing a more public API and ability to download data shortly.  Similar to my initial journey with the Fitbit, I was inspired to find an “unofficial” way to get access to the data (e.g. my Fitbit Hacks).  I was able to get access and save my personal data for about a year before an official method became available.  Luckily Fitbit and other companies like Zeo (especially Zeo) have taken an open approach with their APIs and have embraced the developer community.  My hope is that UP will do the same.

In the spirit of filling in the data gaps until that day comes and an official announcement is made I decided to do a bit of exploration yesterday morning.  Using my favorite HTTP/SSL sniffer (Charles) and some tinkering with the app I was able to decipher how to get access to all of the UP data.  I haven’t written a data scraper script yet, but I was able to verify a number of API requests from the command-line using LWP POST and GET.

I wrote up some detailed notes in my wiki yesterday and have posted a snapshot here:

Jawbone UP API – http://eric-blue.com/projects/up-api/

My hope is this should give others enough information to download their own data for the purpose of backing up and analysis with other software.  Since this isn’t officially supported and appears to be used only by their app I would use sparingly (no excessive requests) and don’t do anything that would violate a TOS/AUP (use at your own risk).  Otherwise, enjoy!

My other Quantified Self-related projects:

TRAQS.me (Tools for Reporting & Analysis of the Quantified Self)

Fitbit Hacks (the original “unofficial” API)

Zeo (Hacking your way to a better night’s sleep)

NeuroSky Brainwave Visualizer

Mindstream – Neurosky EEG Data streamer

For whatever reason, at home I’ve been primarily sticking with a single monitor.  I use Ubuntu and natively use virtual workspaces, but still doesn’t give me the screen real estate that a 2nd monitor provides.  Rather than invest in a 2nd monitor right now, I thought it would be pretty slick to use my iPad as a second “virtual” monitor.  It turns out this is possible AND very easy!

What is a virtual monitor?  Through the magic of VNC i can actually use all of my apps on my main Linux PC and at the same time add touch screen support!  What do you need?

• A computer (PC, Linux, or MAC) with some VNC server installed and a VNC viewer.  I don’t really have a preference on VNC software, but you can check out RealVNC (http://www.realvnc.com/)

• An iPad running some VNC-capable software.  I haven’t researched extensively but I believe there are a number of solutions.  The app I’ve been running for years now is iSSH.  iSSH is sophisticated terminal software that is capable of SSH and support VNC connections.

• A nifty iPad stand that turns your iPad into a swivel monitor (OK, this isn’t needed, but puts some icing on the cake).  I’d recommend the RocketFish iPad Stand.

How do you set this up?  Easy…

1. You can opt to control your PC with VNC like most users do.  However, if you want a real “virtual” monitor/desktop you’d want a distinct workspace.  By default VNC uses :0 (your main desktop) as it’s default display.  You can create a new VNC server instance on a fresh desktop at :1.  I’m using TightVNC so the syntax may vary, but the command to setup is easy:

vncserver -alwaysshared :1

AND to kill the isntance when your done or want to start over

vncserver -kill :1

Note: the -alwaysshared is needed if you want to also control the iPad “virtual” monitor from your main PC

2. If you haven’t done so download and install iSSH from the App Store (currently $9.99 USD).  Setup either an SSH with VNC, or Raw VNC connection to your main computer.  Make sure to specify :1 as the port.
3. If you want to also control this new iPad virtual desktop from your main PC you can run vncviewer. On the command line (assuming Linux) run:

vncviewer :1

Note: You should see this display mirrored on your iPad as well.  You now have the choice to control exclusively from the iPad or share from the main PC.

Here’s a screenshot of my iPad monitor setup using the RocketFish stand and running Firefox from my iPad.

In Dec ‘09 I shared a blog post about how to synchronize the scrapbook data between multiple computers. This was the first major step to sharing data between multiple devices, but still lacked some of the ubiquity that I desired.  In a nutshell I’ve made 2 major enhancements to Scrapbook:

1. An email ‘bridge’ to Scrapbook so I can email links from any device (PC, iPhone, iPad) and have them saved by Scrapbook
2. A centralized web-interface to browse/search/filter my scrapbook data.

I’ll start off with the less visually-stunning hack (email bridge), but by far the craftier of two.

Hack #1 – Scrapbook Email Interface

Whenever I began synchronizing my Scrapbook data between the 2-3 computers this solved a huge problem with being able to save webpages from anywhere.  Since 2009 a lot has changed, and devices like iPhone and iPad (yes, Apple fan boy to a degree) have changed the way we consume news.  Recently I’ve been using apps on the iPad like Zite and Flipboard to consolidate my Twitter, Facebook, and Googler Reader feeds into a single personalized newspaper.  This means that now > 50% of my reading time is spent from a device that has no visibility into my Scrapbook data.  I simply wanted a way to automatically email a link (built nativily into these apps) and have it automagically saved into my Scrapbook folder.  I could have simply cut corners and wrote a script to hand-edit the Scrapbook RDF Files and save the web page using something like wget or curl.  But, it just wouldn’t be the same…. I want the webpage saved EXACTLY as Firefox would normally render and save it.

This poses a bit of a technical challenge, since Scrapbook runs inside Firefox and there’s no native way to interface with a plugin running inside a browser.  After researching a number of approaches, I came across 2 Firefox plugins that let you build interfaces inside firefox (http, telent, etc.) that actually let you control the browser and execute Javascript.  Of the 2 plugins; POW and MozRepl, I decided to go with POW (Plain Old Webserver).  Both plugins are wicked cool in the sense that they’re non-traditional and very powerful.  POW runs a webserver inside firefox and let’s you run your ’server-side’ scripts as Javascript.  I’ve basically written a server process that runs INSIDE the client and executes XPCOM/Javascript to control the web browser windows and invokes the Scrapbook plugin API directly.

The setup process is simple:

1. Setup and install the POW and Scrapbook plugins in your browser
2. Configure POW to run a desired port and create a new directory /scrapbook/
3. Copy the index.sjs (server-side javascript) to this new /scrapbook/ directory
4. Setup a new email box or alias (e.g. yourusername+scrapbook@gmail.com)
5. Either run scrapbook2email.pl manually or run as a CRON job every couple minutes
6. Simply send emails to your new Scrapbook email, run the email script, and watch your pages be saved automatically

At a high-level this is accomplished with 2 scripts:

Email Interface script (Perl)

This script uses IMAP to retrieve scrapbook email requests from a designated folder. Along with doing basic sender/recipient validation, the script is also aware of plain text/multipart messages.  Once the email request is parsed, the link of the requested web page to be saved will be extracted.  Given the request URL the script will then contact the POW server and pass the requested URL (e.g. http://127.0.0.1:6670/scrapbook/?url=http://yourwebpagetobesaved.com/?articleID=3q4e3332).  Note that this version of the script requires that Firefox/POW be running and makes no attempt to launch for you.

For a copy of the script click here (GitHub).

Scrapbook/POW Bridge (Server-Side Javascript)

This script does the heavy lifting, and is essentially running at the other end of the POW server URL (http://127.0.0.1:6670/scrapbook/). Once the requested URL is detected the browser will spawn a new tab, automatically execute the Scrapbook Capture request, and save the webpage to a new top-level folder (e.g. Unfiled/MM-DD-YYYY). This script was tested with Scrapbook v.1.3.7.

For a copy of the script click here (GitHub).

It’s nifty now to email a link to my Scrapbook Bot and wihin a couple minutes a little notify popup shows in Firefox indicating my page was saved.

Hack #2 – Scrapbook Browser

This code was actually written back in Dec ‘09 after I wrote the synchronize blog post (and around the time I wrote the Document Viewer), however I haven’t shared until now.  What I’ve done is write a simple Perl/JQuery web app that used Simile’s Exhibit to view Scrapbook data in a tile, table, or timeline.  This interface also has a file/folder view so you can browse snippets just like you can through the native Scrapbook plugin interface within Firefox.

Here are some screenshots:

Tile View

Timeline View

Table View

Folder View

To download the code click here (GitHub).

1. Find an alternative to iTunes that let’s me access ALL of my music on the go with a low barrier to syncing/listening
2. Cleanup my music collection (meta tags, filenames, etc.) since my iTunes use (abuse) has left many songs forgotten and neglected
3. Free my Music… while I’m in the mood to completely ditch iTunes I might as well get rid of all my DRM-protected music.  I’m doing this for both philosophical and practical reasons.

SubSonic (http://www.subsonic.org/)

Subsonic is an open source music (free) java-based streaming server for Windows, Mac, and Linux.  Right of the bat, I have to say it’s one of the most impressive open source apps I’ve come across.  I had it setup on Ubuntu in < 5 minutes and it works flawlessly!  Subsonic provides a web interface so you can access and listen to your music from anywhere.  I have it hosted on my own personal server and have tested it seamlessly over both 3G and Wi-Fi. The great thing about this product is there are applications for iPhone, Android and Windows 7 that let you stream music to your phone.  The app I chose for iPhone was iSub Music Streamer.  It cost $5 USD, but is probably the best money I’ve spent in the app store.  Things I really like?

• Transcoding – Even if you don’t have all .MP3s on your server, Subsonic can convert and stream other types on the fly (like AAC, AIF, M4A).  You can also customize audio bitrate based on bandwith usage (Wi-Fi or 3G), but by default I don’t limit.
• Caching – Once you’ve played the song it will be cached.  You can also queue or cache songs ahead of time and force the app into offline mode.
• Playlists – You can create both server-side and client side playlists.  This makes it really easy to share with multiple devices (iPhone(s), iPad, PC, media center, etc.)
• Podcasts – This feature completely eliminates the need for Iphone.  Plug in an RSS Podcast URL and Subsonic will download your podcast .MP3s and keep them updated on demand.

A viable alternative to iTunes?  You bet!  No sync times and music is available from anywhere.  I’m rediscovering my music collection all over again and listening to music I haven’t really bothered with in years.

Jaikoz Audio Tagger (http://www.jthink.net/jaikoz/)

Years ago I used a very good Windows program called MP3Tag to cleanup my music collection.  This program would let you bulk-edit the MP3 ID3 tags (Artist, Album, Track, Year, etc.) and rename files.  This program still works well but I found an even better alternative.  Jaikoz (20GBP-~$33USD) is another amazingly slick app. It does everything that MP3 tag does plus more:

• Integration with online metadata sites like MusicBrainz and Discogs to automatically identify files
• Acoustic fingerprinting using Amplified.  If you have a file with no identifiable information in the filename or metadata Amplified will most likely find it (think Shazam or SoundHound)
• Adds album cover art and lyrics

A few years ago I blogged about an incident where my first iPod Video music db became corrupt and I lost information on about 500+ songs (meta data was non-existant in most of them and files were renamed to something like QD3K33.mp3).  Unbelievably Jaikoz was able to recover about 95+% of the files and even added album cover art and lyrics!

Noteburner (http://www.noteburner.com/)

Since Rhapsody and Amazon started offering .MP3 downloads all of my online music purchases have been through them.  However, I still have (had) hundreds of DRM-protected songs that I purchased through iTunes and Rhapsody.  Back in 2008 I went through some pain to burn the Rhapsody purchases to CD and then rip them again back to .MP3.  This time around I wanted to find a way to make it easier.  I’ve used virtual CDRW software in the past like CDBurnerXP and it worked well.  Noteburner, a worthly alternative ($39.95), offers many of the features of CDBurnerXP but also burns to CD and converts to MP3 in a single-step process!  Simply install, select as your preferred CD-RW drive for burning and burn the playlist.  Noteburner will automatically create the virtual CD, read the CD-TEXT provided by itunes, and create the .MP3 file.  Once you have all your converted files, simply load them into Jaikoz and get the full range of meta tags including album cover art and lyrics.

My total cost for this little project was less than $75 and I couldn’t be happier.  I now have a clean music collection that I can access from anywhere in the world, and don’t have to feel the iTunes pain ever again.  C-ya!!

Late last year I wrote a personally pivotal blog post on Total Recall, Personal Informatics and Life Logging.  This was an eye-opening post for me since I discovered the work of Gordon Bell, learned more about his MyLifeBits project which parallels my own Personal Memex project (albeit MyLifeBits was started 10 years ago and has quite a head start) and picked up his book Total Recall.  The book has reaffirmed my belief that a need for an intuitive and consumer-friendly personal memex is on the horizon, and made me aware of a number of devices on the market today that that help assist with e-memory and personal informatics.   From the article on what personal informatics is all about:

“(Personal Informatics is)… characterized as the monitoring and displaying of information about our daily activities through intelligent devices, services and systems. This information allows us to see trends and opportunities for change that we would otherwise miss.  With the rise in network and RFID technology we are pointing to a time where personal informatics can play an important role in our lives. If people can access this information about their daily routines, and interact with their own personal data currently invisible to them: would they make more informed decisions?”

And, from the PersonalInformatics website:

“Personal informatics is a class of tools that help people collect personally relevant information for the purpose of self-reflection and self-monitoring. These tools help people gain self-knowledge about one’s behaviors, habits, and thoughts. It goes by other names such as living by numbers, personal analytics, quantified self, and self-tracking.”

Immediately after reading these articles, and further researching in the book, I decided to start experimenting with some of these devices.  Although there are a lot of different type of lifelogging and personal informatics devices, the area that caught my interest for a practical application is for fitness.  Having devices that can automatically track your weight, fitness activity, and workouts and keep you informed on what decisions you should be making is powerful.

I’ve already evaluated the FitBit device for tracking fitness activity (calories burned, miles walked, sleep patterns) and have successfully created an API for pulling this information directly from fitbit.com (See my Fitbit Hacks projects).  I’ve also been using the WiThings Wi-Fi Scale for the last 6 months.  This scale is amazing in that as soon as you step on the scale it will automatically connect over Wi-Fi and upload your weight, BMI, and lean mass info to the WiThings site which can be viewed on your PC or iPhone.  For workouts I’ve been using the iFitness application for both the iPhone and iPad and love the simplicity of tracking workouts.  And, I just recently came across the Zeo Personal Sleep Coach which provides very detailed information on the quality of your sleep (I haven’t tried it yet, but it looks amazing and I’ll be getting a device shortly and will evaluate).

All of these devices (combo of hardware and software) have a couple things in common:

* They make tracking personal information VERY easy.  In most cases they just work automatically (e.g. step on scale, clip device onto your belt, etc.).

* Data is ready made available for viewing: you can see trends via graphs and make informed decisions

But, there are also some major downsides:

* Getting data from these 3rd party websites is not always easy

* There is no cross integration (e.g. Weight gathered from WiThings can automatically be uploaded to Fitbit or my workout logs, etc.)

I’ve been thinking about ways to integrate all of this data from various products into a single interface.  My ultimate goal is to make this a sub-project of my Personal Memex project.  All areas of your fitness and health could theoretically be tracked from a single place.  I created a quick diagram to illustrate (Large diagram also available on Gliffy):

So, this looks fancy, but what are the practical applications?

1) Keeping your weight under control.  Being able to see your trends, coupled with current activity (coming from FitBit and workout sheets) will allow you to gauge how active you need to be.

2) Staying motivated to workout.  Again, being able to see trends on how many times a week you’re going to the gym or going outside to be active (walking, hiking, running) and getting real-time notifications is a way to help you stay on track.

3) Observing your sleeping patterns to see how it relates to your overall stress and productivity.

This may seem like a high degree of information logging and complexity for most people.  But, I think the real benefit is this: Low-maintenance and High-reward = Great results.  Most of these devices are already tracking this information, and with minimal development, systems could be built to collect, aggregate, and present this information in ways that can help you shift your behaviors and actions.  These are all preliminary ideas right now, but my hope is to start prototyping this over the coming months, integrate into my existing personal memex system, and then release this proof of concept as an online demo.

Of all the apps I use, I have to say that GoodReader is at the top of my list!  GoodReader is a really comprehensive PDF viewer with a large number of features.  Although I’ve been a fan of AirSharing, GoodReader offers it’s main feature of sharing files using Wi-Fi, and some others not present in other apps: excellent support for large PDF files (>75 MB), the ability to connect over the network to view documents (WebDAV, FTP, Google Mail, etc.). At this point I’ve pretty much copied the majority of my My Documents folder.  And, to access the rest of my docs I simply need to VPN into my home server, connect over WebDAV and quickly download.

Now, GoodReader can support a number of file formats including .txt, .html, and various video and audio formats.  But, I’ve been trying to standardize my doc collection into a common format: PDF.  I have a handful of websites that I’ve been wanting to convert to PDF format for offline viewing and have found the PERFECT open source command-line solution: wkhtmltopdf to the rescue.  What makes wkhtmltopdf unique is that it uses Webkit (WebKit is the engine of Apples Safari, which is a fork of the KDE KHtml) to render pages and works on Windows, Mac, and Linux.  Since it’s based on Webkit this means that the resulting PDF looks pretty much like it would in Safari and the format remains intact (including HTML links which is a big plus).

To test this out I thought it would be good to take the Squashed Philosophers site with me.  For anybody with an interest in philosophy, Squashed Philosophers from Glyn Hughes is really an excellent cliff notes-like site that takes that ideas and works of the greatest philosophers of history and compresses into a summary that can be ready in ~30 minutes (condensed but very high quality).  I figured this would be a great thing to take on the go.  I wrote a quick little Perl script that grabbed all of the sub-pages from the main page, and executed a simple command for each URL. Example:

# wkhtmltopdf http://www.btinternet.com/~glynhughes/squashed/ancientgreeks.htm ancientgreeks.pdf
Loading pages (1/5)
Resolving links (2/5)
Counting pages (3/5)
Printing pages (5/5)
Done

The resulting PDF looks really good (embedded in my Document Browser for preview).

http://linuxoniphone.blogspot.com/2010/04/ive-been-working-on-this-quietly-in.html

Since I purchased my new iPhone 3G a few weeks ago, I’ve been happily syncing it with my MacBook Pro. As much as I love my Mac laptop, most of my day to day work is done primarily on my personal desktop running Ubuntu Linux. I thought now would be the ideal time to move all of my music-related activities entirely from OS X to Linux. So, I recently migrated all of my music there (35+GB!), and have been trying out a handful of open source solutions for managing my music collection (SongBird, Rhythmbox, Amarok, etc.). What I’ve quickly learned is that most of these applications work fairly well, however they don’t integrate with the Apple Store (love it or hate it, they’ve got great content) and you can’t sync yet with the iPhone 3G.

Since I’m already running VMware Workstation (XP SP2), I thought it would be a breeze to download the latest version of iTunes and sync away. That dream unfortunately did not come true! It took a little while but after I imported my music library from a shared folder, I decided to plugin the iPhone. I waited….. waited….. and nothing happened! I restarted iTunes and got a most obscure error message:

“iTunes could not connect to iPhone “” because an unknown error occurred. (0xE8000035)”

After an inordinate amount of Googling, It seems this has been a common problem affecting both VMware and VirtualBox users. There was a mixture of comments. Some users said things where working fine for them, others gave up with tears in their eyes. From what I could gather, here were the most common suggestions for solving the problem:

1. Upgrade from Service Pack 2 to Service Pack 3
2. Enable USB 2.0 in your VM preferences
3. Try plugging the phone into a different USB Port
4. Set ehci.present to TRUE in the .vmx file
5. Try running the VM in VMWare Player instead of VMWare Workstation

I can safely say that *none* of these suggestions helped. The first, and most time consuming, thing I tried was upgrading to SP3. No luck! USB2.0 and EHCI where already enabled so that couldn’t be it either. Different USB ports didn’t help, and VMWare Player behaved exactly the same. As an absolute last resort, I decided to download the latest version of Workstation. My version 6.0.2 was from about 7 months ago, and I noticed that 6.0.5 was released at the end of Aug. The solution that worked for me?

Upgrade to the latest version of VMware!!

Why did this work? I browsed through the release notes, and saw mention of an iPhone-specific fix in version 6.0.3:

“With Workstation 6.0.2 the iPhone might be seen as a camera device by a Windows guest but iTunes did not list the device in its device list. In Workstation 6.0.3, the iPhone is usable and visible in iTunes for Windows guests.”

Problem solved and my iPhone is now happily syncing away with my virtual instance of Windows XP. I figured I would share this post and spread the word in case there are other hopeful, but stumped, iPhone 3G users out there.