DIGITIZING REALITY: AUTOMATED 3D POINT CLOUD DATA PROCESSING USING AI
How can today's advanced technology solve the challenges that many organizations face after obtaining vast 3D point cloud datasets, including the management, storage, registration, fusion and extraction of useful and actionable information?
Instruments for digitizing the 3D real environment are becoming smaller, more lightweight, lower-cost and more robust. Accordingly, they are finding increasingly widespread usage, not only on surveying tripods for the highest accuracy, but also on mobile platforms such as autonomous vehicles, drones, helicopters, aircraft, robotic vacuum cleaners, trains, mobile phones, satellites and Martian rovers. Lidar uses laser scanning, while photogrammetry records images from one or more cameras which may be moving. Each laser scan records tens of millions of data point positions and colours in a point cloud, and hundreds of such point clouds may be combined. This article discusses the challenges that many companies and organizations face after obtaining vast 3D point cloud datasets, including the management, storage, registration, fusion and extraction of useful and actionable information.
Cloud computing
The first challenges users face in performing 3D point cloud data processing include:
Data Storage: The amount of data recorded grows exponentially with time, creating large data repositories.
Processing: The computing power required increases as new algorithms with useful functionality are released and with the volume of data.
Sharing: There are multiple stakeholders spread geographically around the world on mobile platforms who all need to view the most up-to-date data at the same time.
Previously, a software application ran on a dedicated server in a data centre but, if the computer hardware broke down, the user either had to find a backup (which had to be standing by and ready) or would suffer an interruption in service. Many companies guarantee a 24/7 level of service and so cannot tolerate this. However, Cloud Computing now gives users access, over a network, to applications running on a set of shared or pooled servers in a globally communicating network of data centres, giving speed and productivity improvements, resulting in increased competitiveness.
Figure 1: 30 Terrestrial laser scans of a central London library, fully automatically aligned using the Vercator software.
Big data analytics
Users face the difficult challenge of how to boil down the vast amounts of 3D point cloud data to generate useful and actionable information. Current methods for creating Digital Twin BIM models of buildings require users to inspect vast 3D point clouds to manually recognize and mark the outline positions of surfaces, straight edges, walls, floors, ceilings, pipes, and objects, which is time-consuming and susceptible to error. Some semi-automatic methods on laptops require users to recognize and mark part of these and the program finds the rest. Again, such objects can be mislabelled. Fully automatic methods are becoming available on laptops but do not find all the useful information, so users must add and correct what is found. Sometimes the automatic method makes so many mistakes it is quicker for the user to find and mark the structures manually.
“Useful information” in one application may be different from that in another application. For example, in autonomous vehicles, it is an accurate 3D terrain model which can be used for safe navigation. In electricity pylon scanning, it is whether the pylon has its safety warning sign in place clearly visible and whether nearby vegetation is gradually encroaching on the power lines. In railway scanning, it is whether there has been any slippage or sag as well as an estimate of when gradually encroaching vegetation will become a hazard. Electricity supply companies and Network Rail are under UK government obligations to regularly inspect their assets and to perform preventative maintenance to ensure continuity of supply and travel.
Geometrical object recognition
Correvate has developed a suite of machine learning geometric image processing methods for fully automated basic object recognition – walls, floors (figures 2 and 3), edges (figure 4) and pipes (see figure 5).
See below for the rest of the article.
Figure 2: Automatic Wall and Floor Recognition in a recently poured concrete shell of a building under construction in London (16 aligned scans).
Figure 3: Automatic Wall Recognition in a recently poured concrete shell of a building under construction in London (16 aligned scans).
Figure 4: Automatic Edge Detection followed by fitting of straight-line segments in UCL circular/octagonal library under the iconic central dome (21 aligned scans)
Figure 5a (top), Pipe scan and 5b (bottom), Automatic Pipe Recognition in a Boiler Room 3.5 million point cloud; 98% cylinders correctly found (2 aligned scans red and blue).
Artificial Intelligence
Artificial neural networks are extremely simplified models of living brains, which are trained and learn like people rather than being programmed by a master programmer. The learned knowledge or skills are stored in a distributed manner in the strengths or weights of the neuron interconnections. Some artificial neural networks learn on their own while others require a teacher or instructor to tell them when they are right or wrong. Gradually, they get better and better at performing a task during the iterative learning cycles which usually take a long time and require thousands of examples of the training data. Artificial neural networks are particularly good at recognition, classification and optimization tasks. However, their performance depends crucially on how they are trained, the types and the amount of training data. Many types of neural network have been developed and, most recently, Convolutional Neural Networks (CNN) used to perform Deep Learning have become very popular and achieve very good results. In the case of object recognition, if the neural networks are only trained with examples of objects one wants to find, then all input data will be classified as one of those objects, even if it is not one of those objects. So, the performance of the neural network is only as good as the way it was trained and the data that was used to train it. Neural networks are not as new as you might imagine given their current popularity in the media. Over 30 years ago, Selviah (1989) proved that the weighted interconnection layer of neural networks performs the same operation as a collection of correlators, operating in parallel, matching images from a database with input data and then the non-linear part of the neurons decide which image matches the input most closely. The clever part is the way in which the training automatically works out what images to store in the database in the first place.
In the conference room image, figure 6, you see the impressive recognition results after training a new type of CNN with data from the Stanford Large-Scale 3D Indoor Spaces Dataset (S3DIS) using 70,000 3D objects of 13 types, structural objects: ceiling, floor, wall, beam, column, window, door, and movable objects: table, chair, sofa, bookcase, board and clutter in 11 types of room. Each category of object is marked in a different colour, for example, ‘chairs’ are marked in yellow, ‘boards’ are marked in orange, ‘beams’ are marked in red, ‘door’ is marked in green, ‘walls’ are marked in dark green, ‘floor’ is marked in blue, etc. The accuracy of classification of objects is around 93.5% comparable to human accuracy. The objects to be recognized can be chosen for each application simply by changing the training database.
Figure 6: AI Automatic 3D object recognition. Plan view of original point cloud data for a conference room and 3D recognised objects. The ceiling was removed for clarity in viewing the inside of the room.
Artificial intelligence in the cloud As the AEC sector embraces digital technology, the amount of data produced grows exponentially, creating large data repositories. To generate useful and actionable information from this ‘big data’ requires leveraging smart analytical tools such as AI that are becoming more accessible, especially when hosted from the cloud. Both the cloud computing infrastructure and artificial intelligence supply the tools to leverage and enable digital technology by providing convenient methods of working at scale, thus lowering the barriers to entry for users to these new ways of working. Artificial intelligence (AI) neural network and deep learning require vast databases of thousands of examples for training, which can be conveniently stored in elastic expandable cloud storage on demand. AI software requires highly parallel processing on many parallel processors to carry out the training in a reasonable time, again easily available in cloud computing infrastructures.
Intelligent combination and use of available techniques such as laser scanning, automatic alignment, cloud computing and artificial intelligence can not only speed up analysis of vast data sets but also improve accuracy and release human activity to ensure that a product is correct and useful.
BENEFITS OF USING THE CLOUD Auto-Application Updates: applications are updated automatically, so the user always has access to the most up-to-date optimised software and bug fixes. Responsivity: dedicated development support teams continuously monitor user experience to optimise and, if necessary, rewrite code. Scalability, flexibility and agility: Scalable elastic cloud environments on pools of servers, storage and networking resources scale up and down according to the number of users and the volume of their usage. They automatically scale up and down as users’ needs change. Capital expenditure free: users have access to the highest power computers. There is efficient use of hardware as users do not need to purchase, manage and maintain large amounts of computer and storage hardware, resulting in lower hardware, power, cooling and IT management costs. Users only pay for what they use as the cloud resources automatically scale, so it is easier for small businesses to manage their business at any time of day, from anywhere. High speed: multiple computers run in parallel so many different parts of the same point cloud can be processed at the same time and many different users have no effect on speed or quality. Security: the data is stored and communicated securely with a level of encryption chosen by the user. If security is a paramount concern, the software can run on a private cloud without internet connections in-house. Clouds can be configured to make use of certain data centres, such as within one country if intercountry security is a concern.
Availability: if one server is busy or not available then another server takes its place to provide full availability. Disaster Recovery: data is stored in multiple locations at the same time so if storage hardware in one data centre breaks down, the calculation proceeds with little interruption as the data is backed up elsewhere. Data archiving facilities are automatically provided. Latency: if latency is important, the cloud can be configured so that local clouds provide low latency to the user. Increased collaboration: many users, located globally, and mobile users, can store, process, share and view datasets at the same time without any loss of speed or responsivity. Reliability: the application software can make use of resources on cloud computing infrastructure provided by different vendors in different global regions. Forward compatible: an open cloud architecture is forward compatible to match higher power computing resources as they are rolled out. Sharing: all point cloud datasets are secure in one place and accessible at any time from
Author: David Selviah
Last updated: 04/08/2020
Comments