The University of Kentucky Center for Visualization & Virtual Environments

ImageNet 3D Research

A group of army commanders sits in a conference room in Washington D.C. – a world away from a battlefield in Afghanistan. Despite the vast physical distance between soldiers in combat and these army commanders, the commanding officers can oversee military action using unmanned sensors. Although this seems like a movie scene, the ImageNet project at the Vis Center is making this a reality. Researchers from across the University collaborate on ImageNet, with the final goal of improving the Army’s ability to prepare for battle through enhanced data gathering, imaging, and display technologies. The project involves software engineering, networking, computer vision, aerial image gathering through remote-controlled planes, and graphics. ImageNet breaks down barriers between disciplines to create new technology that could never be developed in one area.

Two vehicles will gather data: a LIDAR truck and an unmanned aerial vehicle. For example, a LIDAR sensor scanning truck could drive through a residential neighborhood in Afghanistan. The LIDAR scanner will collect 3D points. The truck is also equipped with a “ladybug” camera that takes 360-degree photos. However, the truck’s camera cannot capture the view from above. Unmanned aerial vehicles (UAVs) will fly over the area taking aerial photographs. A control station will collect the data, apply algorithms to the data to extract meaning, and then pass the meaningful material to a remote control center. The algorithms will automatically extract information, such as any suspicious activity or changes in the area. ImageNet will show the commanders in the remote theater sifted information about the battle zone so that the commander will be informed and able to quickly deploy resources.

With the system gathering so much data, an automatic pipeline needs to reduce manual workload. For instance, a human could never pick through the millions of points in a point cloud. Making sense of the 3D data is Dr. Ruigang Yang’s job; he heads the 3D vision team at UK. Dr. Yang uses the 2D images and 3D point clouds to create 3D semantic models. The team focuses mainly on man-made buildings at the moment. With a semantic model, a computer identifies an object as a house, a wall, a roof, and so on. “That is a classic computer vision problem – how to get a computer to understand what it sees.” Computer scientists have chipped away at this dilemma for years with limited success, but Dr. Yang remains hopeful. Since the 3D vision portion of ImageNet focuses on a specific class of objects, man-made buildings, Dr. Yang believes they will succeed.

ImageNet has many applications, both in war and peacetime. If a commander deploys a sensor to take many photos of an area, the sensor takes so many photos that a commander could not look at each one. Researchers must develop a way to filter the inundation of information so officers focus on useful information. The plan uses computers to develop an automatic pipeline to process semantic information. The computer sorts information by knowing what the points show; for example, the LIDAR sensor creates a 3D point cloud of a house. The computer goes beyond recognizing the points’ locations. Instead, the computer analyzes the point cloud so it understands that the points form a house.

ImageNet is useful for meaningful situation awareness. For example, laser scanning recognizes tiny issues on highways such as small surface cracks or the roadway leaning towards the wrong side. The UK 3D team used the LIDAR truck to scan Nada Tunnel in the Red River Gorge. In the future, they will scan the tunnel again to see if the tunnel’s structure changed. ImageNet monitors a site that is difficult to scan, such as a tunnel, and catches structural changes early.
ImageNet expands the resources decision-makers use. An army commander would know everything of significance happening on a battlefield, the President would see the damage resulting from a natural disaster, and an urban planner would have an accurate model of a proposed development. Researchers at the Vis Center work on a wide variety of projects, but they all share one common goal: seeing the world in a new way. ImageNet will change the way we see our world through its delivery of significant information about a situation.

NOTICE: Some websites to which these materials provide links for the convenience of users are not managed by the University of Kentucky. The University does not review, control, or take responsibility for the contents of those sites. Equal Opportunity/Affirmative Action Employer.