Tuesday, December 20, 2011

Automated F3F base detection: 40 fps reached

Although I didn't tell anybody, I made some progresses with my automated F3F base detection program.
One of my goals was to reach a processing rate corresponding to the camera feed rate which is 30 frames per second. With my first trials, I could hardly reach 12 fps. But with an improved complementary combination of two motion detection techniques, the program now reaches about 40 fps which is quite enough for the intended purpose!
For those interested, the principles behind the motion detection are following.

Background subtraction:
The frame preceding the glider entry is grabbed into memory. Once the next frame with the moving glider is loaded, the program subtract each background pixel RGB (red, green, and blue) values from the corresponding current pixel RGB values. Where the current color corresponds to the background color, we set the value to white (or 1). At the pixel locations where the glider "appeared", there his a variably strong difference (above a threshold) between  the glider color and the background color, so we set this location value to black (or 0). Once the subtraction is done with all pixels, we obtain a bitmap image in which the detected glider appears in black on a white background. In the process, we can average the X and Y coordinates of each "positive" pixel pertaining to the glider so we know where in the frame the geometric center of the glider is located. But since it's the nose of the glider we have to detect crossing the base line, we rather store the maximal X coordinate when the glider has to cross from left to right, and the minimal X coordinate when the glider has to cross the opposite base line.
Background subtraction on a 640x480 pixel frame is quite a processor intensive. It necessitates more than 27 million color component subtractions every second! Since we want to detect only what is above the horizon line, it's less pixels but still heavy to process fast enough. I could work with a lower video resolution but then the alignment precision decreases. The trick I found here consists in processing not every single pixel, but only a fraction of them, let's say in a ratio 1:4. This drastically improves the processing rate by the same (inverted) ratio without hampering the detection's precision too much.

Instead of manual threshold, I tried to implement a variable threshold based on average luminance of the frame. This seems to work great on the color footage I used for testing. Nevertheless, the threshold can still be augmented manually if needed.
Once in a while, the glider's colors are below detecting threshold because it's colors are too much alike the background colors. In such cases, frame subtraction gives no result. The glider is there in the frame, but with this technique, there's no way to compute it's location. We could lower the threshold but then some background pixels would be detected together with the glider and we do not want such "noise".

Blob detection:
In those cases when there are no computed glider coordinates after background subtraction, the program grabs the blob detection technique.
The complex blob detection algorithm is a little complicated to explain here, but the principle is that it recognizes color patterns in frames and delineate them. With proper threshold settings for dark and light patterns, it analyses the frames and detects the patches called blobs. Once done, we can get the average coordinates or the minimal and maximal X coordinate and go on towards base detection rules.

Base crossing rules:
For the base crossing rules to work steadily and accurately, we need to know not only the present glider location, but a certain number of it's preceding locations as well. So we know not only where the glider is, but also from where he came. In my program, I store the 20 last glider locations as X/Y coordinates (min, max, average).
The base crossing rules are rather simple:
- when the glider is not visible, all coordinates are set to zero.
- when the glider enters the video frame in the appropriate direction for base crossing, a "entered" flag is set to true.
- if the glider leaves the scene without crossing, the "entered" flag is false, all coordinates are zeroed, and the program switches to the other base camera.
- but if the glider passes beyond the base line coordinate and the entered flag was raised, the base crossing is triggered and the hit counter is incremented.
- as soon as the glider leaves the frame again or when the glider goes undetected, the program switches to the other base camera.

Next steps will be:
- to do only local blob detection around the expected glider location to decrease processing time even more
- to find a way to compile an executable program (exe) that work independantly from the development platform
- to send base crossing signal to by F3F buzzer
- to send base crossing signal to my F3F timing system ;-)
- to use radio-transmitted video feeds so the PC stays in run center
- to inquire whether there would be some accessible graphic processing dev board I could use to replace the netbook. It would be perfect to have the hardware (processing board, camera and RF transmission) in two moisture-safe boxes. Ideally with a hardware budget around than 300 USD (software budget is secret)? If people have suggestion for an affordable dev board that could handle it, I'm interested!

Here is an example recorded tonight. As said earlier, the record is only 12 fps although the program processes at 30-40 fps.


  1. Could you post the raw video files somewhere? I have some optimization ideas and I'd like to try coding something up for it, but I don't have any source material.

    OpenCV compiles on ARM, there are hundreds of dev boards easily within your price range, many have video input and acceleration. How do you get the live video into your system - RCA capture card, HDMI, or USB webcam?

  2. Thanks for your feedback. I was thinking about some ARM boards. For now, I try porting on Android.
    On the PC I implemented USB webcam.
    Right now I can't give away raw footage because I also had to borrow it. I'll ask the owner if it's OK to share. In the meantime, pm me at kitnep at gmail dot com so I have our email address.