Projects

Learning causal models and reasoning perceptual causality from video

This includes the following sub-tasks:

1. Detecting the visible fluent changes using appearance features, such as opening door, light is turned on;
2. Learning causal relations (causal-effects) between actions and detected fluent changes.

Parsing 3D scenes by reasoning physical relations, stability and safety

We have collected a dataset of point cloud data for categories of scenes and have software to fit the point clouds into 3D shape primitives (not perfect) and filling in the missing voxels to generate a solid 3D input. This project includes the following subtasks:

1. Inferring the 3D physical supporting relations between the geometric primitives;
2. Calculating the potential energy landscapes and stability with respect to various disturbance fields, such as gravity, wind and human activities;
3. Calculating the potential energy landscapes and stability with respect to various disturbance fields, such as gravity, wind and human activities;
4. Reasoning about material properties and thus inferring weight and friction;
5. Reasoning the safety of objects and scenes for robot applications

Scene understanding by inferring attractions relations, hidden objects and intents

From outdoor surveillance videos, such as the courtyard scene and aerial video in Fig. 9 collected by a hex-rotor, we will use existing method to detect and track humans and vehicles. Using the trajectories, the inference process will compute the following aspects jointly:

1. Reasoning the hidden objects that attract agents, such as food truck, vending machine, dumpster, mailbox, picnic table etc.;
2. Computing the potential attractive/repulsive fields generated by these objects;
3. Inferring the hidden relations (i.e. intents) between the agents and the attraction objects probabilistically and thus predicting early events;
4. Reasoning the hidden and changing minds/intents of agents;
5. Reasoning the relations between humans and their roles involved in group activities.