Project “webcam events” - UberVU hack fest

Yesterday I participated to “Hack Days” event, hosted by UberVU. The theme if this meet-up was “making life easier”. Me and my friend Andrei were working on a little program that uses the webcam to watch for certain “trigger images”. Each of this images has an event attached, so when the image appears, the event happens. The event should be anything in command line, a python or perl script, or a program with parameters.

Why would you need something like that ? Well, for example, you are in bed and you watch a movie and your mobile phone rings, you want to pause the movie. You raise a little card and the movie pauses. After you talk on the phone, you raise another card and the movie resumes; in this case, the event is local, but the system should permit sending the signal on the network, using sockets. For example, you can have the webcam watch for defects on the assembly line and send some signals to a central server, etc.

So, this “framework” should contain 2 parts : you capture the “trigger images” and define the event, you start the webcam service and wait for the trigger to appear.

Andrei is working in Java on Windows, I’m working in Python on Ubuntu. The GUI should be in Java and the image detection should be in Python.

At first, we tried to take captures in the GUI using Java Media Framework, but jmf 2.1.1 installer for linux is broken, you can’t install anything. To fix it, you have to edit the installer with VIM, search for “tail”, replace “tail +309 $0 > $outname” with “tail -n +309 $0 > $outname”, then you install, then you “chown your_user /usr/bin/JMF-2.1.1e/lib/jmf.properties”, to be able to make any changes to the properties file. And after all this, we discovered that jmfRegistry doesn’t recognize my laptop webcam… Even if the webcam can be used by Cheese and OpenCV…

Finally, we managed to connect the Java GUI, with Python image recognition, using sockets.

This is a print screen of the interface:

And this is the code for python:

import cv
import glob
import socket

HOST = '192.168.1.2'
PORT = 7777

print('Connecting to socket server...')
ss = socket.create_connection((HOST,PORT), 3600) # Timeout 1h
ss.send('hello server!\n')

capture = cv.CreateCameraCapture(0)
frame = cv.QueryFrame(capture)
pngs = glob.glob('*.png') # The trigger images must be PNG
imgs_to_find = [cv.LoadImage(img, cv.CV_LOAD_IMAGE_COLOR) for img in pngs]
W,H = cv.GetSize(frame)

while True:

    frame = cv.QueryFrame(capture)
    cv.Flip(frame, frame, 1)

    for i in range(len(imgs_to_find)):
        to_find = imgs_to_find[i]
        w,h = cv.GetSize(to_find)
        width  = W - w + 1
        height = H - h + 1
        result = cv.CreateImage((width,height), 32, 1)

        cv.MatchTemplate(frame, to_find, result, cv.CV_TM_SQDIFF_NORMED)
        (minval,maxval,minloc,maxloc) = cv.MinMaxLoc(result)
        (x,y) = minloc

        # The smaller the value, the more precise is the detection
        if minval <= 0.09:
            print 'Found:', pngs[i], minval
            ss.send(pngs[i]+'\n')
            ss.recv(1024)
            # De-comment the next 2 lines to view the webcam image in "standalone mode"
            # cv.Rectangle(frame,(int(x),int(y)),(int(x)+w,int(y)+h),(255,0,0),3,0)

    # cv.ShowImage('View', frame)

    if cv.WaitKey(10) >= 0:
        break

Maybe you find it useful.

Thank you Liviu, thank you UberVU ! We really enjoyed your event !

Originally posted at: https://crconstantin.tumblr.com/post/19741864549/