Screenshots

As promised here are some screenshots;






And some of what it looks like on the actual screen;

Progress update

This week I've mostly been working on the business plan part of my project, which is going quite well. I've also found an interesting article on BBC News about humans and computers becoming closer together. It says; "The keyboard, mouse and monitor will increasingly be replaced by more intuitive forms of interaction and display, including tablet computers, speech recognition systems and fingertip-operated surfaces." This is exactly what my project is setting out to achieve, blurring the boundaries between physical and digital.

I've also been working on getting Wiimote input working, along with some general refactoring of the code to make it a bit nicer. I'm using the WiiremoteJ library along with Bluecove and my Macbook (as it has bluetooth built in). Bluetooth is a massive pain and took me ages to get it working nicely.

However, the Wiimote input now works; it picks up the IR light and pressing the button "clicks" on things. I'm not quite sure yet how it will work with finding the angle of point since both the IR light position and the accelerometer are affected by the pointing direction of the controller. I'm going to get some IR LEDs hooked up later this week so hopefully will be able to get a better understanding of how it will work then.

Next tasks are to get the shelves cut and get the internal lighting sorted in the bookcase as this will really allow me to see how everything works together.

I've also knocked up a quick webpage at http://duality.benorgan.co.uk

Some more screenshots should be posted later this week as I've made a few more tweaks to the GUI.

Touchpacity is no more...

After an hour of searching through thesauruses, Wikipedia and using the Dotamator I've decided on a new name for my project.

Duality

du·al·i·ty (dū-ăl'ĭ-tē, dyū-) pronunciation
n.
The quality or character of being twofold; dichotomy.

GUIness

I've been playing around with my Java GUI today and have made a few aesthetic tweaks. Hope you like it!

Javadoc

Javadoc documentation is now available at http://benorgan.co.uk/duality/javadoc/. This will be kept up to date as the project progresses.

Progress update

It's been a while since I posted an update on my project, so here it is!

So I've finally got the short-throw projector which allows me to project onto the screen at a decent brightness without using mirrors and such! It's amazing how big the image is given such a short throw, in fact the image is too big now and I've had to make the image smaller to get it to fit on the screen. The image is also much brighter than my previous arrangement and I had to turn down the brightness and contrast as it was too bright as default. It looks like the width of shelves I want to have should be all OK with the new projector too.

Short throw projector in the bookcase, just fits!

I've also got hold of a wide-angle webcam from Creative which has an 80 degree viewing angle apparently. This is needed so I can view the whole screen from the webcam inside the bookcase. When placed at the back of the bookcase the webcam can indeed see the entire screen. I had planned to attach the webcam to the underside of the bookcase shelf but I also realised a problem, in that the webcam needs to be at the front of the shelf otherwise the shelf will obscure the view of the screen. Although eventually the shelves will be much narrorwer this still means the webcam won't quite see all of the screen. However, I think it's the best I'm going to get given the bookcase dimensions. The webcam also has correction for the wide angle lens built into the drivers which is quite handy as I was worried this would cause problems!

Upon testing out the webcam with my OpenCV program I found that it wasn't as "good" as the other cheap old webcam was. I assume this is because firstly the webcam can see a lot more and secondly it's a lot more sensitive. Adjusting the settings of the drivers and my program gave some improvements but it was still not as good as previously. After much time fiddling I decided that the card/tape I was using to detect the position of the finger was too dull. I got a few packs of large brightly coloured sticky dots and tried with these instead. Using these the performance seemed much better. There were also now issues due to light reflecting back off the screen from the projected image which hadn't been a problem previously as the projector wasn't as bright. Adjusting the brightness and contrast of the projector helped a bit.

I'm still not very happy with the OpenCV detection as it doesn't seem to be as robust as before and I don't think detecting angle of point is going to be an option, which will be a pain. I'm going to play around with different lighting. I've also implemented some stuff to load and save settings from the OpenCV program which should make adjustments easier as I can build on what I did last time rather than starting from scratch each time.

The webcam duct taped to the back of the bookcase

Since my last post I've also spent a fair amount of time working on my Java code. I've added functionality to look up book information from the Amazon API based on the ISBN number read from an XML file. The Amazon API is massive and it took some time to figure out which bits I needed to use, but it all works fine now. The only thing I need to work on is extracting keywords and reviews but currently it gets author, title, publisher, date and an image all from the Internet. I've also made the Java code more general and tidied it up a bit, although I think that's going to be an ongoing process!

The image the new projector shows; much brighter and no problems with keystone or focus, notice the new GUI and information from Amazon

I also met with a guy from the Bristol Museum a couple of weeks ago, they were very interested in what I am doing and he said they'd be interested in seeing it in action and/or running a demo at their demo case in the museum.

Finally I've also been thinking about the Computer Sciency bits of my project. I think the main questions I need to answer are;

• Where is it best to display the information on screen given that people want to see the physical objects and the digital information. Also, what information should be displayed?
• How do you figure out what the person is pointing at given that they could be pointing at the screen or an object in the bookcase. In both cases the actual position of their finger tip and the angle it is pointing as viewed by the webcam will be dependant not only on what they are pointing at but also on the distance from the screen, as the webcam views the world in persepective. This is exagguarated with a wide angle lens.
  
• Once the problem of what the user is pointing at is solved then I still need to figure out if they are pointing at the screen or a book as there are two layers of information. Moving between these layers could be done in a number of ways, for example, explicitly by having a way to switch between the layers (e.g. a close button) or by using some sort of heuristic.
• How should the user get feedback about what they are pointing at? A cursor on the screen seems a simple solution, but is this too much like a normal computer?
• How can a user select objects? The options here are dwell time, clicking by moving the finger somehow or using a different input method for clicking (voice for example)

I'm sure there will be more questions as time goes on, hopefully next time I'll have some answers too!

New Enterprise Competition

As part of the Business Plan component of my project we were asked to submit an entry to the University of Bristol New Enterprise Competition. Being a bit of a business geek I've entered every year I've been at Bristol, but didn't win once.

So I was very surprised when they announced my entry, Touchpacity, as one of the seven finalists this year! Not only do I win £200 as a cash prize but I also get help and advice from some really experienced people, get to go to the Enterprise Dinner in June and have the chance of winning some of the £35,000 prize fund.

I've got to develop a full business plan and presentation for before the dinner in June, but I need to do that anyway for my course so it's all good.

Java GUI

Also this week I've been spending a lot of time developing my GUI application in Java. Lots of time spent figuring out how they work again and trying to remember how Listeners work.

Anyway, I now have an application which will read information from an XML file (or other sources once I write the code) and take input from the OpenCV application I knocked up a few weeks ago (or other sources once I write the code). It then uses these to create an on screen representation of the books which can be selected using your finger pointing at a webcam. As there is no notion of "clicking" I am currently using dwell time to select things. I've also made an on-screen keyboard where you can "type" in letters and search for strings within the books and it then highlights all the relevant books on-screen.

A screenshot of the amazing GUIness

Picture of me using the input to select books. Notice my (green clad) finger at the top left in front of the webcam. The image is then processed using OpenCV to detect the position of my fingertip (on the right of the screen) and used to control the cursor on my application (left of the screen).

Next week I'm planning on getting my application working on the screen and spending a lot more time on the input (specifically figuring out if my plan is actually going to work!).

I'm still lacking a name (and not sure about Touchpacity). New ideas include Tiffany and Midas. I will buy anyone who comes up with a name I end up using a pint.

More projection...

I spent a bit more time this week experimenting with the projection onto the screen. I managed to find a mirror (which was actually part of a table mat set from a very cheap shop in Wales for £5) which I could use to reflect the projection off. This not only gives me control over where the image goes but also increases the throw of the projector, making the image on the Holoscreen larger.

Hi-tech mirror setup for reflecting the projected image

Image after being reflected off the mirror, with some keystoning this will be about the right size. However, it's only really visible with the lights off due to the low power of the projector

As you can see the mirror makes the image much bigger although it's not very bright. I'm still waiting for another projector (either a standard one or a fancy short-throw one) so in the mean time I bought in my personal projector to see how much difference a bright projector made (although the bulb is past-due to be replaced). I forgot to bring the DVI adapter I needed so unfortunately the image is just of the projector logo, but you get the idea.

As you can see this projector is much brighter, even with the lights on. Certainly bright enough for what I need.

And a close up shot, ignore the lines on the screen; they are due to the DLP projector and you can't see them in real-life

The next thing I wanted to find out was how deep the shelves could be before they restricted the image. Obviously this depends on a number of factors; mostly the angle of projection (determined by the mirror), the depth of the shelves and the height of the lowest shelf. I stuck a bit of card hanging off a shelf at the top of the bookcase at the depth where I want the shelves to be (I'm planning on cutting them). This would then provide an object to obstruct the projection so I could see how big of a problem it would be. If you look closely at the image (apart from my reflection) you can see the cardboard inside dangling down, and then you can see a black area where the card is causing a shadow right at the top of the projected image. As you can see this shouldn't present too much of a problem as the shelves can be fairly low down before interfering with the image.

Pic showing how the shelves might cause problems for the projected image (ignore the reflection of me taking a photo!)