Saturday, January 11, 2014

(Off topic) Wearable computer project build details, part 1

I've had an interest in wearable computing for a long time -- and after Glass was announced and it became clear that Google wanted to produce wearable computing technology that was limited to gluing powerpoint to your face, I decided to get my act together and build something practical for once.

Since this has been a fairly long-running project, compared to my previous attempts, I figured that I might as well document it.

Step 1: Monocular Display

So, a friend sent me a partially disassembled head mounted display. Having had read about the use of monocular displays, and mechanisms by which people had turned binocular displays into monocular displays (mostly tutorials for MyVu-brand displays wherein the wires were simply cut), I decided to take what was essentially a freebie display and turn it monocular.

Disassembling the display

I had a pair of goggles that I bought at an anime convention (steampunk is really big at anime conventions), so I actually just mounted the disassembled display on one of the thin clear plastic lenses with a combination of screws and velcro, then mounted the lens on the inside using velcro.

Mounting the display board and one of the control boards to the lens

The control box with its battery has been opened up and mounted with velcro on the side of the goggles
The lens is mounted to the inside with velcro
The completed monocular display mod, for this revision

Step 2: Panic, Worry, Waste Money

I was always planning to use the Raspberry Pi as the heart of the wearable computer, but I actually had quite a hard time getting the software to work. In the end, it turned out that I was using a mislabeled SD card -- it claimed to be one of the brands that was supported, but it was actually a card from another manufacturer that had been re-labeled and re-sold. SD cards, despite adhering to the SD standard, can vary significantly enough between manufacturers that a project like the Raspberry Pi doesn't actually support arbitrary brands, particularly for booting. I got a different card that was branded the same, and it worked just fine.

Initially, I thought that I had burned out the control box for the display I had. I was missing the adapter for charging it, so I opened it up and attached a USB cord directly to it (because I couldn't find an appropriately sized barrel connector); only later did I actually attempt to hook it up, and found that it was displaying a white screen regardless of what I was doing with the Pi. Later on, when I got a duplicate display, I discovered that this was actually the fault of the composite video adapter I was using -- I was attempting to use a composite audio adapter for composite video, and getting no signal. The duplicate display came with both the charger and the appropriate adapter, so in the end I didn't need to modify that box at all, nor did I need to replace it.

Anyhow, between this and lacking a handful of other components, it took a while before I got around to hacking together the rest of the project.

Wearing the display 
From the side

Step 3: Software

I wrote this software using a library that I had written for another project, and that is supposed to be released as open source some time in the near future. It's not released yet, so I'm avoiding really talking about it. Suffice it to say that it's similar to a previous project.

Software written for this project -- a zzstructure navigator -- running on the Pi, but connected to a normal TV

Step 4: Hook Shit Together

In the end, since my monocular display worked just fine, all I needed to do was connect it to the Pi. I ended up making things a little bit neater, for the sake of my sanity.

I took the separate audio and video cables leading from the Pi to the adapter and wrapped them with a bit of spiral wrap I had lying around from earlier projects. Having these catch on things individually would be extremely annoying, once I started actually wearing the thing -- I've had wires from earlier wearable computer projects catch on things like chairs, pets, and moving vehicles, with all the expected consequences. The number of cables required to attach a first generation twiddler to a laptop manufactured ten years later may well be the reason that particular unit broke (and it's very hard to find twiddlers on ebay).

I wrapped everything in a small zippered leather case. This case was, when I obtained it, marketed as a wallet -- although it doesn't actually fit in a normal-sized front pocket -- and explicitly intended for users of PDAs. As a wallet, it failed -- it was too large and couldn't hold much of anything in the way of bills or cards. As a PDA carrier, it failed -- it was actually too big for any of the various models of palm pilot or blackberry I had since I obtained it, and the velcro patch on the inside never managed to keep them in place. However, it's perfectly sized for the Pi, complete with plugs sticking out of the side and an SD card sticking out of the back. The wifi dongle I attached stuck out the top, making it impossible to zip the thing up completely, but it didn't cause enough of a gap to radically risk the hardware in the case of a sudden downpour.

I also used sugru to seal up a couple of the holes in the plastic case my Pi uses that I didn't think I was likely to use, and to insulate the unused side of the two-in-one USB cord I'm using to power the Pi. 

The full layout, at the time of writing

 The monocular display, functioning

Recent Changes

Using velcro for holding sensitive optics in place is a huge pain -- every time I get things displaying right, the lens moves as soon as I move my head. The velcro was self-adhesive, and it had trouble consistently sticking to the lens housing and to the plastic pane used by the goggles, so the lens would sometimes even physically fall out. So, I mounted some sugru on the pane where the velcro was, and I intend to use small screws and bolts to adjust the lens distance.Update: the sugru didn't work -- I couldn't get an appropriate distance with it, and it was hard to trim down just with the tools at hand. However, I was in luck: despite the lens being much closer to the eye in the goggle mount than in the original system, it turned out that if I flipped the lens around in the housing and then flipped the housing around, I got perfect focus at the intended distance. So, I just superglued the housing onto the pane of the goggles, and clipped the lens in backwards:
The lens housing being glued in. The screws are just there as extra anchor points -- they are flush with the back of the housing and glued on as well. The lens has small protrusions that clip it into the housing with either orientation, so this made it pretty easy; I now have the larger half of the lens facing my eye from the outside of the housing. Keep in mind that this puts the lens very close to the eye (so, watch out for infections), and that superglue produces fumes that are irritating to the eye (so wait for it to air out before using the display for long periods). The housing won't actually fit square with the display because there are some screws that poke through from the other side holding the display itself onto the pane, so I have it shifted to one side; one side of the screen is slightly blurry, so perhaps I should have aligned the display the other way.

The unhoused control box had some problems. It has a giant lithium-ion battery pack held on with some not-very-sticky double-sided tape, which on several occasions fell off with enough force to pull free the power wires. It also was too heavy to stick on to the velcro on the goggle's vent (this is where it is mounted in the side photo, which is why it is tilted to the side). I had kept it around just by chance, but because I was so sure I had burnt it out, I misplaced some pieces of the housing and could not re-house it in the original case. So, I just wrapped it in electrical tape and taped it to the band of the goggles, after cutting some holes for connectors and for the power switch. The cable leading from the control box to the display is fairly long and stiff, so I looped it up and taped that to the control box as well, leaving just enough slack to connect the display's control board.

Future work

I still haven't solved the problem of input. Many people use tiny bluetooth keyboards of the type intended for powerpoint presentations for this, but I consider full QWERTY keyboards to be best used with two hands and unwieldy with one -- and what's the point of a wearable computer if you need two hands to operate it. I'm considering building a chordite derivative, or perhaps a clone of the twiddler -- both are fairly popular one-handed chording keyboards intended to be held in the hand (as opposed to some chording keyboards that are only good if put flat on a table). I'd like to have a joystick or a direction pad for moving the mouse pointer, unlike the twiddler (which uses an accelerometer). I have some teensys I can use for emulating a genuine keyboard and mouse, but I'm circumspect about adding more programmable hardware to this project; I may simply use the Pi's GPIO port. UPDATE: turns out that the twiddler hasn't actually used accelerometers for mousing since the 2.0, about ten years ago. I dropped $200 on a Twiddler 2.1, so I will update when I can test it. It seems like everything will work fine with it aside from using the SUPER key as a meta key; since I am running Awesome with the default configuration on the Pi, I'll need to reconfigure the command key to be CTRL-ALT instead.

Another possible input system is eye tracking. There was an interesting student project at Cornell using cheap phototransistors and LEDs for eye tracking by bouncing infrared light off the sclera, and it seems like that should be easy to incorporate.

I've been interested in Thad Starner's presentations on a project at Georgia Tech wherein glove-mounted vibrating motors are used for non-intentional learning. In his experiment, subjects were taught to play simple melodies on the piano using these vibrating gloves, which simulated the muscle memory of rote practice. There are many things I'd like to be able to do with my hands, and some of them can probably be accelerated with this, so I've bought five vibrating motors and intend to wire them up to the Pi's GPIO port and train myself with them, as soon as I can find a suitable glove. Update (2014-01-20): Started on the glove, but I don't really have the appropriate equipment so I'm having my girlfriend finish it for me. Here's the beginning of it:

On a less exciting note, I still haven't received the battery pack I intend to use. So, I have that to look forward to in a few days, at which point I'll be able to test how long the unit can run. I was surprised at how long the display ran on a full charge, although I expect to need to break out a cable to quickly charge it up from the main battery pack. I wouldn't like to hook it up to the Pi itself, since it doesn't take too kindly to sudden drops in board voltage; instead, I might simply attach a USB splitter to the battery pack and run one line directly to the display charger, ready to be plugged in whenever power gets too low. I'm not really sure how long the battery will last when running the Pi with hardware hooked up and also charging the display, so I may end up upgrading to this battery pack instead.

In the end, I intend to stick both the battery pack and whatever boards I need for bridging the Pi GPIO with my own implemented I/O devices inside the leather case, although I may not be able to fit them. The battery pack looks pretty water-resistant, so maybe I'll just keep it in another pocket.

Update (2014-01-19):

I appear to have somehow fried one of my Raspberry Pi boards. I initially thought that the power supply wasn't putting out enough amperage to run the board, but it turns out (luckily) that I was incorrect -- it works just fine with my other Pi. I have no idea what caused the failure. Unfortunately, it's clear that the wifi dongle (which did work once upon a time) is now no longer functional; I may need to replace it, or it may simply be that the battery is incapable of supplying enough power to run it properly.

The second Pi came with its own (rather nice) housing, and I'm avoiding putting it in the same box as the other one on the off chance that the housing is related to the damaged board. In terms of likelier causes, though, it looks like there's some damage to the SD card holder, which may be from stress on the card itself -- having your SD card stick out of the housing by more than an inch is a design flaw, and perhaps one of the most glaring in the Pi. I suspect I will need to build a small extension to the housing for the express purpose of accommodating a normal-sized SD card, or purchase a much shorter one and transfer the image there.

My twiddler still hasn't arrived, although I am increasingly hopeful about its capabilities. It looks like I'll be able to get away with using it for everything, but that I will need to change all my keybindings from SUPER to something else because the twiddler doesn't support SUPER as a meta-key. If it's easy enough (as this teardown implies) I may add a hold switch, because I believe every portable device with buttons requires a hold functionality.

Update (2014-01-23):
The main problem with the monocular display is that the goggles fog up quickly. Even completely removing the vent covers doesn't help much. So, I was considering dehumidification mechanisms.

One option (the easy one) is to put a bunch of dessicant packages into the goggles. Unfortunately, dessicants tend to be toxic, and they also need to be replaced frequently. Best-case, you can use salt (which isn't toxic but which you wouldn't want in your eyes) and bake it to get the water out. I'd like to avoid this.

The other is to use cooling to cause condensation on some heat-sink or coil outside the goggles and collect or otherwise discard the condensed water (the way that both air conditioning units and professional dehumidifiers work). The trick is that most of these have moving parts (which you don't really want attached to your face).

Luckily, there are solid-state semiconductor-based systems for cooling: peltier junctions. And, better still, all the peltier junction produces is a difference in temperature between the two sides -- meaning that we can also produce air movement by exploiting the difference in pressure between hot air and cold air.

I'm not fully convinced that this will work, but the idea is that the expanding air on the top (hot) side of the peltier pump will push through the tube into the heat sink on the cool (bottom) side, and the movement of the air will create just enough suction to cause the air in the goggles to pass through the cool heat sink and condense some of its water, which will then drip off. I will probably need to mount a thermistor on the top and build an automatic shut-off that will kill the power to the heat pump when it gets too hot.

Update (2014-02-01):
I received the Twiddler, and last Tuesday I took the unit out for a test at the local hackerspace:

and also at work:
The battery life seemed pretty OK, but I hadn't gotten a replacement wifi adapter yet so I didn't trust it.

Today, I installed the new wifi adapter (an Edimax EW-7811Un, based around the RTL8188CUS and very common for Pi users) and took the machine for a walk. I wore it and kept it on and connected to wifi from around 11AM to around 5PM without recharging, and this was fairly successful.

There are two caveats. One is that the adapter doesn't seem to like mixed 802.11BG networks (which is what my router provides), and an unsecured 802.11BG network shows up as a WEP-enabled ad-hoc network. I switched my router into B mode and this fixed the immediate problem. However, I still couldn't connect -- and it turned out that if I plugged in the battery pack, I could connect just fine.

The SD card did just fine as well, which was somewhat unexpected since the previous one had its holder shear in short order after much less wear. I credit a combination of more support in the laser-cut box I'm using and the unexpectedly good performance of the hacked-together SD-card protector I stuck on. I had a bunch of rubber belts lying around that were just slightly smaller than the perimeter of the box, and I wrapped one around the end of the box such that it would extend out just past the end of the SD card.
The belt wraps around the end, and juts out just past the edge of the SD card. The battery is held on with rubber bands, for now.

The wifi dongle is just below the keyboard connector here, so it can't be seen. The battery's wire trails around.
As a side note to the de-fogging saga, I've noticed that fogging doesn't happen nearly as much outside in the cold (for all the obvious reasons), but that this doesn't really help because outside light coming through the left side of the goggles actually overpowers the light from the display. In addition to defogging mechanisms, I will need to figure out a way to block out light. I may roll that into attempts to encase the electronics and protect them from the elements.

Update 2014-02-14:
A few weeks ago, while installing an opaque mask on the lens, the connections between two lens-mounted boards broke. On the display side, there's a socket, but on the side hooking to the board to which the (usb-lookalike) connector going to the controller connects, our wires are surface-mount soldered to vias. Anyone trying to replicate this project should probably desolder these wires and solder them on the other side of the via, because the torque from the cable leading to the controller is enough to pull the velcro off (and so it was only a matter of time before surface-mount connections like that pulled apart).

I created the mask just out of thin cardboard and electrical tape, using a spare lens as a template and cutting out a rectangle for where the display was affixed. I placed it between the lens and the screw-on abutment that holds replacement lenses in.

After adding the mask and redoing the solder, I folded velcro around the edge opposite the wires I just soldered in and affixed the board to the velcro vertically; I also changed the position of the cable to avoid producing so much torque on the wires.

The next step for this project is the production of a protective casing for the lens-mounted boards. I wanted such a casing to be transparent, but it looks like barring getting a glass blower or an expert in lexan-bending to produce a small box, I will not be able to produce a transparent case. I may need to use an opaque printed case.

Update 2014-04-25:
Adafruit published a tutorial for an extremely similar design. Their version is slicker and easier to follow, so if you are trying to build something like this, I recommend primarily following their directions (particularly for the disassembly portion). If you are trying to build this unit exactly, follow their disassembly instructions and then follow mine beginning with step #4.

A future path I'd like to explore is to use something like this for legitimate overlay. My general idea is to remove the goggles entirely, use the adafruit version of the mount with the elbow inverted (so that the display goes in the opposite direction) or straight (so that the display juts straight out), then mounting an extra fresnel lens (if necessary) and a half-silvered mirror at a 45 degree angle in front of the eye:


  1. hey john! this is a fun project, we'd like to email about making it a learning guide on please send us an email! -ladyada (