| | | |

Computational Video

Tracking Talkers with a Large Distributed Microphone Array

The objectives of the project are to investigate methods of analyzing data from a large distributed array of microphones to localize one of more talkers within a room. In addition, video cameras will also be used to help locate potential talker candidates. The project started in 2006 and is expected to end in 2009.

There are several innovative aspects to this project. The first is the way in which audio and video data are used together to dynamically track people. The second is how the distributed microphones will be used for dynamic beamforming. Beamforming will be used to capture utterances from talkers who are some distance from any microphone. However, only a subset of the microphones will be used for beamforming at any one time. Research is required to develop methods to select an optimum set of microphones which can best capture the sound at any instant.

This project follows on from an earlier project called Beamed Microphone Arrays which has been completed. What is new is the use of a large distributed array of microphones for beamforming, along with a set of distributed cameras to determine the position of a human talker and to capture the utterances.

Dozens of microphones are placed in a room, along with a number of video cameras. The microphones and cameras are calibrated so that their three dimensional position is known. Head detection and face tracking software is used to find the people in each camera view, and then their three dimensional location is computed by triangulation. The result is a set of possible locations for the people. This information in combination with directional information derived from the microphone array is used to guide the sound pickup process.

We aim to generate an effective video and audio signal equivalent to that produced by a table-mounted system, but in a much less obtrusive manner. Many applications are cited in the scientific literature for beamforming microphone systems. Most address the need for an unencumbered and natural interactive environment using hands-free technology.

The focus of this project will be on using this technology in the context of a video conference room, or the control of large interactive displays. Our research results could have an impact on the following sectors: video conferencing, security, public kiosks.

This project is a joint research activity between the NRC-IIT CVG and the Acoustics and Signal Processing Group at NRC-IMS. The technology would allow people to have more natural video conference sessions, and it would also facilitate the use of broadband visual communications for collaboration.

Research Contact

David Green (Dave)
NRC Institute for Information Technology
1200 Montreal Road
Building M-50, Room C-344
Ottawa, ON K1A 0R6
Telephone: +1 (613) 993-6572
Fax: +1 (613) 952-0215
E-mail: David Green

Business Contact

Dr. George Forester
Group Coordinator, Business Development
Business Development Office
NRC Institute for Information Technology
1200 Montreal Road
Building M-50, Room 203
Ottawa, ON K1A 0R6
Telephone: +1 (613) 993-3478
Fax: +1 (613) 952-0074
E-mail: George.Forester@nrc-cnrc.gc.ca