Java GUI

Also this week I've been spending a lot of time developing my GUI application in Java. Lots of time spent figuring out how they work again and trying to remember how Listeners work.

Anyway, I now have an application which will read information from an XML file (or other sources once I write the code) and take input from the OpenCV application I knocked up a few weeks ago (or other sources once I write the code). It then uses these to create an on screen representation of the books which can be selected using your finger pointing at a webcam. As there is no notion of "clicking" I am currently using dwell time to select things. I've also made an on-screen keyboard where you can "type" in letters and search for strings within the books and it then highlights all the relevant books on-screen.

A screenshot of the amazing GUIness

Picture of me using the input to select books. Notice my (green clad) finger at the top left in front of the webcam. The image is then processed using OpenCV to detect the position of my fingertip (on the right of the screen) and used to control the cursor on my application (left of the screen).

Next week I'm planning on getting my application working on the screen and spending a lot more time on the input (specifically figuring out if my plan is actually going to work!).

I'm still lacking a name (and not sure about Touchpacity). New ideas include Tiffany and Midas. I will buy anyone who comes up with a name I end up using a pint.

Holoscreen is attached!

Yes I know it's Friday night and I should be at the pub but my colleagues are growing anxious at my lack of blogging so I thought I better post something to pacify them.

This week I finally got the DNP Holoscreen attached to the bookcase. It's a temporary bodge at the moment just so I can figure out if / how everything is going to work. I'm hoping to neaten things up and adjust the positioning of the screen so it fits onto the front of the bookcase a bit better next week, but this will do for now. Below are a few pics;

DNP logo and felt covered hinge which attaches the glass to the bookcase

Dodgy cardboard washer and bolt arrangement (this will be sorted out soon!)


View of the bookcase with the glass screen closed (notice the overlap on the right, that will be fixed soon too!)

View of the bookcase with the door open. Only one shelf at the moment. Eventually the shelves will be a bit shallower.

Also this week I've been playing round with OpenCV and managed to detect coloured bits of card which I stuck to my finger (yes I looked crazy when testing it, waving my cardboard covered finger around in mid-air). Using this I can find out the position of my finger in front of a webcam and also the angle (by doing some maths on the two known points). The angle part isn't working very well yet (need to add some Kalman filtering or similar) but the position seems to work pretty well. Managed to knock up a simple Java app which let me move the pointer around the screen using my finger and select boxes by hovering over them for a second. I'm still not sure whether to use this or the Wiimote (or something else), but hopefully testing next week should sort that out.

Picture of my hand with the bits of card on. My OpenCV program looks for the green and blue coloured parts of the image and uses backprojection, thresholding and blob detection to work out the positions of the finger tip and joint. This information is then sent over the network to my Java app.

Finally I've also been finalising the business plan competition entry this week, which is due in on Tuesday.

I'm still looking for a better name than Touchpacity, suggestions would be most welcome.

Anyway, time to stop working and head to the pub!

Project progress

So far I've read a shed load of papers and done a lot of thinking and planning.

I've also been playing around with some OpenCV stuff to detect where fingers are pointing but at the moment I'm thinking this might be too complicated and unreliable to use. I also hooked up my mate's Wii controller to my PC by Bluetooth and played around with WiinRemote to control the cursor on the screen. Although this would require users to hold a device it's a lot more accurate and reliable and means I don't have to spend weeks working on a computer vision problem (which isn't really under the remit of my project).

I've also made some progress with the hardware; the bookcase is built and has hinges and I've been playing around with NVKeystone to give the keystone correction I need for the projection onto the screen.

Today I fired up Netbeans and wrote my first few lines of code - it was a bit disconcerting starting from a completely blank screen realising that eventually this will be my project! So far I've got a Java GUI full screen with a custom cursor. Not very exciting, but it's a start!