First Steps into the Quantified Self: Getting to Know the Fitbit API

Today I have been experimenting with the Fitbit API.

fitbit Logo

Fitbit Logo

Here are some rather amateurish steps for getting data from Fitbit’s “cloud”.

Simple Client Setup for Public Data

1. Get and setup a Fitbit product.

I have access to a set of Fitbit scales (the “Aria”). The Fitbit Force looks quite good; I may get one of those when they are launched in the UK to record steps.

2. Register an “App”

Goto https://dev.fitbit.com. Login using your Fitbit password from 1). Click the link to add an “app”.

3. Setup Libraries

Install oauth2:

sudo pip install oauth2

Clone python-fitbit:

mkdir Healthcare
cd Healthcare
git clone https://github.com/orcasgit/python-fitbit.git

4. Record Consumer Key & Secret

After registering your “app” in 2) above, you have access to a “Consumer Key” and a “Consumer Secret”. You can access these details by going to “Manage My Apps”. I then store these in a “config.ini” file I keep in my working project directory. For example:

[Login Parameters]
C_KEY=*key_string*
C_SECRET=*secret_string*

5. Access Public Data

With the client variables we can access public data on Fitbit.

import fitbit
import ConfigParser

#Load Settings
parser = ConfigParser.SafeConfigParser()
parser.read('config.ini')
consumer_key = parser.get('Login Parameters', 'C_KEY')
consumer_secret = parser.get('Login Parameters', 'C_SECRET')

#Setup an unauthorised client (e.g. with no user)
unauth_client = fitbit.Fitbit(consumer_key, consumer_secret)

#Get data for a user
user_params = unauth_client.user_profile_get(user_id='1ABCDE')

You can get your user ID by accessing your Fitbit profile page. It is displayed in a URL just above any photo you may have. The last command returns a JSON string similar to this one:

{u'user': {u'city': u'', u'strideLengthWalking': 0, u'displayName': u'USERNAME', u'weight': 121.5, u'country': u'', u'aboutMe': u'', u'strideLengthRunning': 0, u'height': 0, u'timezone': u'UTC', u'dateOfBirth': u'', u'state': u'', u'encodedId': u'1ABCDE', u'avatar': u'https://pic_url.jpg', u'gender': u'NA', u'offsetFromUTCMillis': 0, u'fullName': u'', u'nickname': u'', u'avatar150': u'https://pic_url.jpg'}}

As I am only looking at getting a current weight reading (as taken from the last Aria scales measurement) I could stop here, as long as I have made my “body” data public (viewable by “Anyone” on the Fitbit profile > privacy page). However, as I wanted to get a BMI and body fat reading I continued with user authentication.

User Authenication

The python-fitbit library has a routine called gather_keys_cli.py. I followed this routine in iPython to get the user key and secret for my own Fitbit account.

1. Fetch Request Token

client = fitbit.FitbitOauthClient(consumer_key, consumer_secret)
token = client.fetch_request_token()
print 'FROM RESPONSE'
print 'key: %s' % str(token.key)
print 'secret: %s' % str(token.secret)
print 'callback confirmed? %s' % str(token.callback_confirmed)
print ''

2. Allow Access to User Fitbit Account

I first followed the steps:

print '* Authorize the request token in your browser'
print ''
print 'open: %s' % client.authorize_token_url(token)
print ''

I had registered my callback URL as http://localhost/callback. I had not actually implemented a request handler for this URL on my local machine. However, I learnt this was not a problem – the parameters I needed were included in the callback URL. Even though the page got a ‘404’ I could still see and extract the ‘verifier’ parameter from the URL in the browser.

3. Get User Key and Secret

First I stored the ‘verifier’ parameter from the URL as a string (verifier = “*parameterfromURL*”). Then I ran these steps to get the user key and secret:

print '* Obtain an access token ...'
print ''
print 'REQUEST (via headers)'
print ''
token = client.fetch_access_token(token, verifier)
print 'FROM RESPONSE'
print 'key: %s' % str(token.key)
print 'secret: %s' % str(token.secret)
print ''

4. Save User Key and Secret and Use in Request

The last step is to save the user key and secret in the “config.ini” file. I saved these as “U_KEY” and “U_SECRET” in a similar manner to the consumer key and secret. Hence, these variables could be retrieved by calling:

user_key = parser.get('Login Parameters', 'U_KEY')
user_secret = parser.get('Login Parameters', 'U_SECRET')

Finally, we can use both sets of keys and secrets to access the more detailed user data:

authd_client = fitbit.Fitbit(consumer_key, consumer_secret, user_key=user_key, user_secret=user_secret)

body_stats = authd_client._COLLECTION_RESOURCE('body')

Todo

Some further work includes:

  • Adapting the python-fitbit routines for the iHealth API.
  • Building a script that gets data and imports into a website datastore.
  • Using the body data to alter a SVG likeness.

Doing Useful Things with WeMo Motion

Using both the WeMo Motion rules and IFTTT allows you to do certain things with this motion detector. However, to expand our possibilities it would help if we could store our motion data and make it accessible to the programs that we write.

20131004-063848.jpg
To store our motion data in a database we need a bit of a convoluted process. It goes something like this:

20131004-063802.jpg
First we set up an IFTTT recipe to send an email to a Gmail account when motion is detected.

20131004-064517.jpg
Having done this you will get a series of emails:

20131004-064837.jpg
I recommend setting up a separate Gmail account for automation to avoid spamming yourself with IFTTT emails. It would also make things more secure for the next steps. To make things easier when using multiple IFTTT recipe emails, I set up a filtering rule in Gmail to automatically label all emails like this as “Motion”.

The next step is to write some Python code to access our emails, process messages and store data in an SQLite database.

  • The email processing makes use of the imaplib and HeaderParser libraries; and
  • The database processing makes use of the sqlite3 libraries.

A first function accesses all unread emails with a particular label and returns an array of the subject lines of those emails.

def read_subjects(label):
	obj = imaplib.IMAP4_SSL('imap.gmail.com', '993')
	obj.login('username@gmail.com', 'password')
	obj.select(label)  # <--- it will select inbox
	typ ,data = obj.search(None,'UnSeen')

	subjects =[]

	for num in data[0].split():
		data = obj.fetch(num, '(BODY[HEADER])')

		header_data = data[1][0][1]

		parser = HeaderParser()
		msg = parser.parsestr(header_data)
		#print msg['Subject']
		subjects.append(msg['Subject'])

	return subjects

A second set of functions then processes each subject line to extract a ‘datetime’ that the motion occurred and a motion sensor name.

def store_motion(subjects):
#Initilise temporary array for data
	rows = []
	unread_count = len(subjects)-1

	#Process and store unread mail items
	for j in range(0,unread_count):
		#print subjects[j]
		#Extract date/time of last motion
		extracted_date = extract_date(subjects[j])
		#Extract motion time
		motion_time = datetime.datetime.strptime(extracted_date, "%B %d, %Y at %I:%M%p")
		#Extract sensor name
		s_name = extract_sensor(subjects[j])
		#Add (motion time, sensor name) tuple to rows
		rows.append((motion_time, s_name))
	#print rows
	storemotioninsql(rows)

def extract_date(word):
	date_index_start = word.find("ion: ")+5
	date_index_end = word.find(" at")+11
	date_out = word[date_index_start:date_index_end]
	return date_out

def extract_sensor(word):
	name_end = word.find("' ")
	word_out = word[1:name_end]
	return word_out

A third function stores the prepared ‘datetime’ and motion sensor name in an SQLite database.

def storemotioninsql(rows):
	#Save in database
	con = lite.connect('motion.db')

	with con:

	    cur = con.cursor()

	    #Create a READINGS table if it doesn't already exist
	    cur.execute('CREATE TABLE IF NOT EXISTS motion (r_datetime TIMESTAMP, r_s_name TEXT)')

	    for row_values in rows:
	    	#print row_values
	    	cur.execute('INSERT INTO motion VALUES(?,?)', (row_values[0], row_values[1]))

All that remains is to set these functions up in a python script and then use cron to schedule it to run every 15 minutes (crontab -e etc…).

Databases for Sensor Data in Python

Here are some options to persistently store sensor data in a database using python.

Anydbm

For simple key-value pairs the anydbm module looks the best.

It stores values as strings. Because of this you may need to use another module called pickle to convert non-string values into strings.

For example:

import anydbm, pickle
from datetime import datetime

#Open a database - if it doesn't exist: create it
db = anydbm.open('test_data.db', 'c')

time_now_in = datetime.now()
string_version_in = pickle.dumps(time_now_in)

#Store value
db['1'] = string_version_in
#Close database
db.close()

#Open database
db2 = anydbm.open('test_data.db')

string_version_out = db2['1']
time_now_out = pickle.loads(string_version_out)
print time_now_out

db2.close()

SQLite

For more advanced database features the next step is sqlite3.

A tutorial on sqlite3 can be found here.

You may need to install sqlite3:

sudo apt-get install sqlite3

Then in python you can use SQL statements using code such as:

import sqlite3 as lite

#... - data in variables

# Set up a new database connection - db will be created if it doesn't exist
con = lite.connect('energymonitor.db')

#with the db connection (simplifies commit etc)
with con:

cur = con.cursor()

#Create a READINGS table if it doesn't already exist
cur.execute('CREATE TABLE IF NOT EXISTS readings (r_datetime TIMESTAMP, r_watts INT, r_temp INT)')

#Store sensor readings
cur.execute('INSERT INTO readings VALUES(?,?,?)', (time, watts, temp))

This then stores my energy monitor readings in a database called (unoriginally) "readings":

Simple Energy Monitoring on Raspberry Pi

So. Reading an LCD screen was more difficult than I first assumed. My code is still good for reading my Thermostat values. However for energy monitoring and £20 I found a better solution.

Looking on the net a particular model of energy monitor was recommended for easy monitoring: the Current Cost EnviR.

This monitor has two big advantages:

  • A USB data cable that allows a serial port connection; and
  • Simple XML output over this connection.

These two features were a breath of fresh air after dealing with the proprietary mess that is the Owl/Electrisave energy monitor (just google).

A little trick: the EnviR was supplied by EON to its customers. While the EnviR is listed for about £50-70 new you can also buy cheap it on eBay by way of this EON link. Look for items listed as EON energy monitor and compare with the picture above (also look for the white RJ45 to USB cable – the monitor also comes in black).

Once you have a monitor get code from https://github.com/mapkyca/Current-Cost-EnviR

cd ~/Code
git clone https://github.com/mapkyca/Current-Cost-EnviR

(Many thanks to Marcus Povey for this http://www.marcus-povey.co.uk.)

If pip is not installed:

sudo apt-get install python-pip

Install pySerial:

sudo pip install pyserial

I got an error that the serial port was already open – this was resolved by deleting the following line:

meter.open()

I then found that my energy usage was shown as 0 Watts. Doing a print out of the read data I noticed that there were three channels. I had plugged my monitor sensor into the third input on the energy monitor transmitter: hence the reading was on channel 3 (<ch3>):

Through trial and error I worked out that channel 1 is the central sensor socket on the transmitter:

Success!


Todo:

  • Setup time correctly on monitor (or ignore and use datetime – datetime.datetime.now());
  • Set up a local database and store the tuple: (timestamp, energy, temp);
  • Clone data in Pachube/Cosm/Xively (now owned by LogMeIn!)?