Without innovative HD 3D maps, such as the ones recently introduced by Here Technologies, the autonomous car revolution would grind to a halt. Artificial intelligence needs the data from the maps to orient vehicles precisely enough to guarantee safety in complex, changing environments. Transport planners, too, could use the 3D maps to help design roads, update traffic infrastructure and improve the performance of roadside cameras. Meanwhile the greater detail in the 3D maps is improving the performance of ADAS functions.
Designing 3D maps is a radical departure for Here, which has been producing 2D standard definition (SD) digital maps for 30 years. SD maps were designed for human drivers to go from A to B and were simple to read. But in 2013 Here began to wrestle with the problem of how best to create accurate 3D representations for autonomous cars, in partnership with Mercedes, one of three members of the consortium that owns Here, along with Audi and BMW. The challenge was to produce a vastly more precise and detailed map using live data.
“The HD map and its derivative products can be used for road planning and traffic design,” says Brian Lent, Here’s CTO. “The precise detail and the real-time nature of the data will help planners. We are also providing consumers with a mobile app enabling them to report real-time road conditions with two taps. That information goes to our cloud and then gets sent back to drivers to give them a warning. We call it our Hyper Local Awareness Platform, an example of the use of real-time data that consumers can choose to participate in. The data can also be sent back at the same time to the local transport authorities.”
Lent also explains that the data they are collecting is relevant to connected vehicle services: “We get more efficient data for parking and traffic. We can do things at a higher definition than we have traditionally done with GPS probe data, because the data we are collecting from our automobile customers comes from a much richer sensor package array, which means it’s generally more accurate. So we can improve the accuracy of a lot of the existing packages we have as part of the same update.”
Triple layered
To achieve superior performance, HD Live Map is composed of three layers. The first is similar to an SD map. It provides a road map based on curves, elevation and coordinates. This layer allows the AI system to understand the topology of intersections and the basic attributes of the roads, including speeds. But it remains a generalized view, with nowhere near enough information for an autonomous vehicle.
With the addition of the second layer, Here was already getting much closer to its goal. The HD Lane Model provides live information that differentiates between lane types, such as shoulder, passing and express lanes, as well as lanes reserved for certain vehicles, such as HOVs and bikes. It sends data about the width and speeds of each lane, whether markers are striped or solid, and even their color. All the data is continually updated.
The final layer is the HD Localization Model, which allows the car to locate itself precisely on the map. “This layer provides information about traffic signs and infrastructure at the sides of the road. It also gets data about walls and guardrails, as well as polls and barriers, none of which exist in an SD map. With this added layer, a vehicle continually checks its orientation, which enables it to stay in its lane, stop in the right place at intersections, and make turns accurate to within 20cm [7.9in],” says Dietmar Rabel, Here’s director of product management for autonomous driving.
Making the map
The data to build the HD Live Map is captured from two main sources. The first is the fleet of Here’s True Cars, which drive around gathering 28TB of data every day. Each True Car features a 360° lidar system with rotating lasers that measure the world in 3D. There are six cameras beneath the lasers and a panoramic camera above them. Additional sensors measure speed and orientation, as well as the incline of a street and the degree of a turn. This information is accurate downto centimeters.
The second source consists of crowd-sourced vehicle sensor data about paths, lanes and pavement marking. To date, 20 OEMs are helping Here gather the information. “As the saying goes, ‘A rising tide floats all boats’,” says Lent. “The more data we get from manufacturers, the better the quality of information. Our customers are finding ways in which more advanced vehicles, with more advanced sensors, can update the map as they go. Having up-to-date maps will become even more important as we move from
level 2 to level 5 automation.”
The HD 3D maps have broad applications beyond autonomous driving, including some for road transport agencies, which can receive precise data about traffic flows and accidents, or be alerted about sudden changes on the roads. “There are lots of scenarios a modern car can detect with its sensors. It could be useful for a DOT to know that a section of guardrail has been knocked down, a tree has fallen on a road, a dead animal is on the highway, or a fresh pothole has appeared,” says Bernd Fastenrath, senior product manager at Here. “The cars can inform us and we can deliver that information, including the precise location, to highway maintenance companies.”
Fastenrath says because the maps are so detailed, a city DOT could use them to spot bits of road infrastructure that require updating. “For example, a city that wants to make its environment more inclusive could use the highly precise map to know where it has pavements that are not wheelchair-friendly,” he says.
Increasingly, camera-based sensors are also becoming adept at observing their environment. With roadside cameras, image-processing technology can be paired with HD mapping to help with lane-level vehicle counts. “You can get a more detailed understanding of traffic flow, or precise figures for people movements, such as pavement use by pedestrians, numbers using public transit stations, or cyclists using bike lanes,” Fastenrath says.
Even with consumer-grade lidar, he adds, there are interesting potential applications. Some cars are equipped with lidar that could generate mini-maps for places such as indoor parking garages, where Here does not drive its capture vehicles. Another use for the data is to update Here’s existing ADAS packages, such as cruise control, with more accurate information.
3D Slam
Here is not alone in innovating with 3D map technology for autonomous vehicles. Kudan, the UK-Japanese developer of computer vision, recently announced that its 3D SLAM (simultaneous localization and mapping) technology was ready for market. Kudan’s real-time 3D tracking and mapping system provides computers with a form of vision. Unlike Here, Kudan does not provide AI computing, but teams up with hardware manufacturers.
“We offer tracking technology to generate 3D maps that help cars understand their position,” says Daiu Ko, Kudan’s COO of strategy and business development. “But to understand the meaning of the data, we team up with machine-learning companies. Kudan SLAM is the ‘eye’ and machine learning is the ‘brain’ and they need to be closely integrated.”
Ko says that Kudan’s SLAM is a highly versatile algorithm that is suitable for all hardware, including drones and robots. “Previous SLAM technologies have been mainly in-house products, such as Microsoft’s HoloLens wearables. But we see the versatility of our product as crucial in selling the software. We’re already working with a major Japanese automotive OEM that wants to develop its own camera and sensing system.”
HD maps without lidar
It’s been almost impossible to think of live HD 3D maps without also thinking about lidar. A new startup is changing that
In July 2017 three former Tesla and iRobot engineers launched lvl5, a developer of crowdsourced HD 3D maps and computer vision software for self-driving vehicles. It has already secured US$2m in seed funding from investors.
The founders of lvl5 argue that lidar, while enabling cars to understand the world around them, has limitations, including an average price tag of US$80,000 and sub-par performance in inclement weather. With lvl5, there’s no need for lidar. The system uses a combination of crowdsourced HD 3D maps and sub-US$15 consumer-grade cameras.
The company has been crowdsourcing mapping data using a dashcam app, Payver, allowing anyone to earn US$0.01-US$0.05 per mile driven. lvl5 is already working with Uber and Lyft drivers across the USA who run the dashcam app throughout the day. As cars drive around, they are vacuuming up data, such as ‘a stop sign was just added to this intersection’, which is then sent to a central hub.
lvl5 has also developed a computer-vision algorithm that translates the footage into HDmaps that are accurate down to a 10cm threshold. In the three months leading up to its launch in July 2017, lvl5 claimed to have mapped 90% of all US highways using this crowdsourcing strategy, and will later build Payver’s dashcam into vehicles. The company has existing pilots with major OEMs who will pay an initial fee to install the system into vehicles. Because HD maps must be continuously maintained, lvl5 will also charge a monthly subscription fee per vehicle.
HD maps without lidar
An alternative way to build HD maps without the need for lidar cameras is being developed by a startup from California
San Francisco-based startup Mapper emerged out of stealth mode on October 11, 2017, after a year spent developing 3D maps for autonomous cars and other applications. Mapper takes a different approach to Uber, General Motors and Google’s Waymo, which all make their own detailed maps using vans equipped with expensive lidar technology. But Mapper doesn’t believe lidar is essential. Instead, it has created a small army of part-time workers to gather the data required to create a base map.
The company’s engineers have designed a small plastic device called an S1 that contains high-definition cameras and sensors. It fits on a car’s dashboard and gets power from the lighter socket. The S1 syncs to the driver’s smartphone wirelessly. The Mapper app on the phone directs the driver to take a particular route. Data about the route is sent back
to the company’s server.
Mapper expects to need about 10,000 mappers to maintain its base map, which could then be licensed to auto makers and technology companies. Other customers might include DOTs looking for detailed coverage of their roadways, insurance companies looking for hazards, and augmented reality ventures.
