Video Analysis

American Sign Language Recognition, Gesture Recognition, Human Activity Classification

We use a detection-based motion representation using relational distributions. Motion is represented as changing probabilities. Unlike previous approaches, we do not require the signer to wear specialized colored gloves, and we do not need explicit tracking of the body parts. We use a random sampling-based algorithm for capturing the shapes and spatial configurations of objects in an image using low-level primitives, e.g. edge points, and efficiently computing the histograms used for representing motion. Histograms are embedded into a low-dimensional space by preserving the probabilistic distances like Bhattacharya, Matusita, Chernoff, Kullback Leibler (KL) and symmetric-KL distance measures, and motion recognition is performed in the new space using matching of the trajectories.

Data Source: Boston University

Learning from Long Video Sequences

Given multiple sequences with one common sign, we automatically extract the common sign from the sequences. Right-hand side images show two sequences - I BUY TICKET WHERE? and YOU CAN BUY THIS FOR HER, with a common sign BUY (marked in red). The adjacent signs are marked in magenta. The unmarked frames between the signs indicate the frames corresponding to movement epenthesis. The movement epenthesis also effects how the sign is signed. This effect makes the automated extraction, modeling and recognition of signs from continuous sentences more difficult when compared to just plain gestures, isolated signs or fingerspelling.

Molecular Imaging & Reconstruction

Estimating Astigmatism in micrographs

We developed an algorithm to estimate the astigmatism present in the micrographs captured using an electron microscope, using a RANdom SAmple Consensus (RANSAC)-based approach. When the astigmatism – a parameter in the contrast transfer function of an electron microscope – is not estimated and corrected for, the resolution of 3D reconstruction suffers significantly.

Unsupervised approach to extract bad/noisy particle images

We developed an unsupervised classification algorithm to improve 3D reconstructions of protein macromolecules from extremely noisy digital images captured using an electron microscope. Several digital images of the molecule that have the same viewing parameters are averaged to improve the signal to noise ratio. We developed an automated algorithm to identify and reject images of molecules having inconsistent viewing parameters, or having extremely low signal to noise ratio. The green, blue, red and the non-colored boxes, in order indicate the worse to the best subclasses.