Carnegie Mellon University’s Robotics Institute, in collaboration with the Civil and Environmental Engineering departments at CMU and Northeastern University, has recently launched the Aerial Robotic Infrastructure Analyst (ARIA) project. Funded under the National Science Foundation’s National Robotics Initiative, the ARIA project is developing new methods to rapidly model and analyze infrastructure using small, low-flying robots. Our one page flier provides a brief overview of the project. The article below gives a more detailed account of the problem ARIA is addressing and our approach.
A recent review of our nation’s aging bridges, dams, and other infrastructure highlights the need to rehabilitate or replace structurally deficient or functionally obsolete structures (ASCE, 2009). Current inspection methods involve expensive, specialized equipment, and are labor-intensive as well as potentially dangerous. Furthermore, inspections are subjective and result in representations that are difficult to compare over time.
The Aerial Robotic Infrastructure Analyst (ARIA) project introduces a new concept in infrastructure inspection. Rather than putting inspectors in harm’s way, ARIA uses small, low-flying robots, coupled with three-dimensional imaging, and state of the art planning, modeling, and analysis to provide safe, efficient, and high-precision assessment of critical infrastructure. We illustrate the concept with bridges because they are among the most prevalent and important types of infrastructure requiring regular inspection, but the ideas can be applied to many types of infrastructure, including electrical transmission towers, dams, and buildings.
America’s Aging Infrastructure
America’s infrastructure is deteriorating at an alarming rate. The American Society of Civil Engineers (ASCE) has provided a Report Card that gives our nation’s infrastructure an overall grade of D [ASCE, 2009]. The report highlights that one in four of the nation’s bridges are structurally deficient or functionally obsolete and are thus in need of more frequent inspections while the bridges await significant repair or replacement. The consequences of our deteriorating infrastructure can be significant in terms of loss of life as well as economic impact. The 2007 collapse of the I-35W bridge over the Mississippi River killed 13 people and injured 145. Economically, increased travel distances cost an estimated $400K per day [Minn DoT, n.d.], not to mention $234 million to replace the bridge and over $50 million in insurance claims [Phelps, 2010].
Infrastructure Inspection Today
Infrastructure inspection is an expensive, manual, and labor-intensive process. Inspectors physically observe various critical components of the structure and identify any deterioration or potential problems. Evaluations are subjective and can vary significantly from inspector to inspector.
Inspection results are recorded in lists and tables, sometimes accompanied by hand-drawn sketches or markings on blueprints to indicate the locations of problems. The tabular format makes it difficult to relate observations to specific locations on the physical structure, and it limits comparison of a component’s deterioration over time. With today’s technology, there is no way to directly track the progress of corrosion on a specific beam.
Large structures require specialized equipment, such as snooper trucks that allow inspectors to access the undersides of bridges, or the inspectors must climb over and under the structure, which can be dangerous, even with training and safety equipment.
Recently, micro air vehicles (MAVs) have begun to be used for infrastructure inspection, mostly using imagery or videos. For example, Aibotix has developed an intelligent MAV with an onboard camera that can be used for inspection. However, video-based approaches give a narrow “soda-straw” view of the situation, and inspectors can easily become disoriented when navigating complex, unfamiliar environments. It can be challenging to relate a close up of a structural defect to its location within the overall structure.
The Aerial Robotic Infrastructure Analyst
The ARIA Project is taking infrastructure inspection to a new level. ARIA will use small, low-flying robots (i.e., MAVs), coupled with 3D imaging and advances in planning, modeling and analysis to provide safe and efficient, high-precision inspection and damage assessment of structures. Rather than just observing, ARIA will actively construct a semantically rich 3D model of the structure, which will enable new methods of analysis and immersive interaction with the data.
Consider this future scenario. An inspector arrives at a bridge site and removes a large suitcase from the trunk of her car. She opens the case and pulls out ARIAbot – a small multi-rotor MAV – and a tablet computer. After a couple of pre-flight diagnostics, ARIAbot takes off. It begins by making a quick pass over and under the bridge. On the computer, a rough 3D model of the bridge begins to take shape. Different components are automatically identified and segmented into objects – footings, columns, beams, girders, and the deck. ARIAbot conducts the standard inspection process, collecting 3D data and close-range imagery of all the critical components. The inspector notices that from the previous year’s report, one of the girders has some significant cracking. She highlights the girder for a detailed analysis. ARIAbot flies in for a closer look at the problem girder, applying algorithms to assess the location and size of the cracks. The result is overlaid on the integrated 3D model and linked to the model from the previous inspection. Flipping between the two models, the inspector can see the crack is getting worse. The inspector requests a simulation, and ARIAbot translates the 3D model into a finite element model (FEM) that incorporates the revised crack detections. The simulation results show that the bridge is safe for now, but estimates that a repair will be needed in a year – several years sooner than planned. Later, back at the Department of Transportation, the results of the inspection are transferred into the Bridge Management System, and the repair is scheduled for the following spring.
Our vision of ARIA addresses many of the limitations of existing infrastructure inspection processes:
- Comprehensive coverage – Unlike ground-based systems or manual physical inspection that spot check areas deemed to be important, ARIA can comprehensively cover all visible surfaces of the structure.
- Safe access – The robot can safely fly to high locations and inspect the structure at close range, even over water or other hazards where ground-based sensors cannot reach.
- Offline access – Since ARIA creates an integrated 3D model containing all measurements, inspectors can later revisit the data and recheck the results.
- Objective measurement – Unlike subjective and potentially inconsistent evaluations from inspectors with different experience and training, the ARIA algorithms are designed to provide objective inspections that are stable over time.
- Longitudinal analysis – Repeated inspections of the same structure can be linked together to analyze the progression of deterioration over time.
The ARIA Project is organized into three core objectives:
- Rapid infrastructure modeling and analysis. The most significant technical challenge of ARIA is to rapidly and accurately model infrastructure in a manner that is useful for inspection tasks. The process consists of four key operations: 1) controlling the robot and creating a 3D point cloud map; 2) transforming the point cloud data into a semantic, component-based model; 3) visually analyzing the model to identify defects; and 4) converting the semantic model into a finite element model (FEM) and simulating the resulting model for structural assessment. The result of the process will be an integrated infrastructure model (IIM), which links the robot’s raw observations with derived results, such as the component-based model, the structural analysis FEM, and inspection algorithm outputs.
- Immersive, engineer-centered inspection and assessment. Using the IIM, engineers and inspectors can immersively interact with the modeling and inspection process. The 3D environment provides improved situational awareness so that inspectors can rapidly orient themselves with respect to the structure. The model enables virtual inspections in real time as well as after the fact.
- Robotic inspection assistant. As the ARIA robot gathers and processes data, it generates potentially useful knowledge. The robot acts as an inspection assistant, providing appropriate feedback to the inspector and suggesting or reprioritizing inspection goals.
The ARIA Platform
The ARIA platform is a custom-designed octo-rotor MAV. It is equipped with a lightweight single line laser scanner that rotates at about 1 Hz, providing accurate, but low-resolution 3D measurements up to a range of 30 meters. The platform also sports three video cameras – two forward-looking stereo cameras and a wide-angle, high-resolution inspection camera pointing upward. An inertial measurement unit (IMU) and GPS provide relative and absolute position estimates, and wireless communication enables data transmission to and from the base station.
Rapid Infrastructure Modeling and Analysis
The first, and most challenging objective of the ARIA project is to rapidly create accurate 3D models of infrastructure, not just in terms of low-level geometry, but also in terms of high-level, semantically rich models. Each of the operations that make up this objective is, in itself, a difficult problem.
Mapping Using UAVs. In order to create an accurate 3D model of the infrastructure, we need to accurately estimate the ARIA robot’s position as it navigates the environment. This problem is well studied, and is known as the simultaneous localization and mapping (SLAM) problem. In the case of infrastructure inspection, we can incorporate information from the image and 3D sensors as well as the IMU and GPS. One challenge in this environment is the imprecise nature of the GPS signal. If the ARIA robot flies under a bridge, the GPS position estimate will be degraded because many of the satellites will be occluded. Position estimates based on 3D and imagery rely on particular levels of scene complexity. We are investigating methods for dynamically estimating uncertainty models for different measurement modalities and developing an understanding of how these models are coupled between modalities.
Semantic 3D Modeling. The ARIA platform will produce point cloud models of the infrastructure, which can be useful for some purposes. However, much more can be accomplished if we can transform the raw point cloud data into a semantically rich, component-based model. For example, if we can recognize the columns of a bridge, then the inspector can specify to the ARIA platform, “Give me a visual inspection of each of the columns,” rather than tediously manually marking each column in a point cloud. Furthermore, transforming component-based geometric models into a finite element model is a potentially simpler and more powerful pathway to structural analysis. One of the challenges for semantic 3D modeling of infrastructure is the vast variety of designs and styles of structures. We are attacking the problem by incorporating top-down knowledge in the form of shape grammars, which are rules or guidelines dictating how different components are related to one another. We are also studying how to adapt algorithms to work across different styles of infrastructure through domain transfer techniques.
Visual Analysis and Inspection. Many of the problems with infrastructure can be detected and analyzed visually. Visual inspection of infrastructure surfaces includes detection and monitoring of cracks, spalling (concrete deterioration), and regions of rust. While these problems have been studied in the past, most existing approaches rely on bottom-up image-processing approaches or ad-hoc heuristics. Our goal is to investigate methods that consider the physical phenomena that give rise to these defects, incorporating the latest methods from computer vision and machine learning into the inspection process.
Structural Modeling and Assessment. The key question for an inspector is whether the structure being assessed is in danger of collapse. Often, the best way to answer this question is through simulation, typically using finite element modeling (FEM). For many reasons, it is difficult, if not impossible, to directly transform a point cloud into an FEM for damage assessment. We believe, however, that by first transforming the point cloud into a semantic model, the problem becomes much easier. Given a 3D model consisting of connected beams, trusses, girders, columns, and such, a FEM can be created comparatively easily. We are studying this process and evaluating the ability to incorporate damage estimates from visual analysis algorithms into the models.
Immersive, Engineer-centered Inspection and Assessment
Infrastructure inspection is inherently a 3D process. However, current inspection methods distill the 3D information into 2D plans or non-visual representations (i.e., tables and checklists). Such a separation between the physical representation and the inspection results can lead to missed information and mistakes. For example, if an inspector counts column numbers incorrectly, an inspection result for one column could be attributed to a different one. Since the data from different inspections is not physically aligned, it is difficult to compare and monitor changes over time.
If, instead of separating the inspection results from the physical model, we allow the two to be intimately tied together, we can benefit from the spatial relationships dictated by the physical model. The ARIA project is developing an immersive visualization environment using the integrated infrastructure model as its foundation. In our research, we have found that immersive 3D environments can be beneficial for tasks ranging from vehicle tele-operation to building inspection. In this project, we are evaluating the use of an immersive model for interacting with the robotic inspection assistant, for conducting virtual inspections off line, and for analyzing and tracking deterioration of infrastructure over time.
A Robotic Inspection Assistant
While it may be possible to eventually design a fully autonomous system for infrastructure inspection, such an approach is likely to lead to user frustration and limited applicability in the near term. The ARIA robot is intended to work interactively with an inspector. We are working to develop planning and control algorithms that learn an inspector’s preferences based on observations and then adapt their operation to meet those needs. Using these algorithms will enable the robot to act as an assistant or apprentice to the inspector, adjusting its automation level appropriately to the situation.
One of the central challenges for the inspection assistant is dealing with the robot’s competing objectives. The inspector or visual inspection algorithms provide objectives in terms of inspection goals, minimum resolution of specific observations, and accuracy of the 3D model. The vehicle itself must manage positional uncertainty, safety, feasible flight paths, and limited battery life. We are investigating hybrid search-based/optimization motion planning algorithms to address this challenge.
The Potential of ARIA
The ability to rapidly create comprehensive, accurate, and semantically rich 3D models using micro air vehicles has potential beyond the inspection domain. The same techniques can be applied to modeling and analyzing disaster sites, construction sites, or historical sites. The disaster at Fukushima is a good example of how ARIA would benefit other domains. After the accident, a call was issued to the robotics community for robots to help with handling the catastrophe and determining the state of the reactors. Unfortunately, the destruction and debris limited the ability of the ground-based robots that were called into action to reach parts of the site that would easily be mapped by a MAV. Perhaps the next time disaster strikes, the ARIA robot will be on the scene to help out.
[ASCE, 2009] American Society of Civil Engineers (ASCE), “Report Card for America’s Infrastructure,” American Society of Civil Engineers, Reston, Virginia 2009.
[Minn DoT, n.d.] Economic Impacts of the I-35W Bridge Collapse. St. Paul: Minnesota Dept. of Employment and Economic Development, Minnesota Dept. of Transportation, http://www.dot.state.mn.us/i35wbridge/rebuild/municipal-consent/economic-impact.pdf.
[Phelps, 2010] David Phelps and Rochelle Olson, Firm to pay $52M in I-35W bridge collapse, Minneapolis Star Tribune, August 24, 2010,