Year: 2020

Apple isn’t the only one accused of kicking out competitive solutions from its App Store. Google did the same — for over a month at least — or so alleges parental control app maker Boomerang. The company’s product competes with Google’s own Family Link solution for controlling screen time and children’s use of mobile devices. The company claims Google repeatedly removed its application from the Play Store for a variety of issues, including violations of Google’s “Deceptive Behavior Policy” which relates to users’ inability to easily remove the application from their Android device.

The issue itself is complicated and an indication of how poor developer communication processes can make an existing problem worse, leading developers to complain of anti-competitive behaviors.

Like Apple, Google also has a set of rules developers have to agree to in order to publish apps on the Google Play store. The difficulty is that those rules are often haphazardly or unevenly enforced, requests for appeals are met with no replies or automated responses and, at the end of the day, there’s no way for a developer to reach a human and have a real discussion.

You may recall a similar situation involving screen time apps hit a group of screen time app makers last year. Apple then had suddenly removed a host of third-party screen time and parental control apps, shortly after introducing its own Screen Time solution within iOS 12. The company’s move was brought up during last week’s antitrust hearings in Congress, where Apple CEO Tim Cook insisted Apple’s decision was due to the risk to user privacy and security these apps caused.

The case with Boomerang is not that different. A developer gets kicked out of the Play Store and seems to have no way to escalate the appeal to an actual human to discuss the nuances of the situation further.

The Boomerang Ban

For starters, let’s acknowledge that it makes sense that the Play Store would have a policy against apps that are difficult to uninstall, as this would allow for a host of malware, spam and spyware applications to exist and torment users.

However, in the case of a parental control solution, the reality is that parents don’t want their kids to have the option to simply uninstall the program. In fact, Boomerang added the feature based on user feedback from parents.

Google itself puts its Family Link controls behind a parental PIN code and requires parents to sign into to their Google account to remove the child’s account from a device, for instance.

Boomerang’s app required a similar course of action. In “Parent Mode,” parents would toggle a switch that says “prevent app uninstallation” in the app’s Settings to make the protection on the child device non-removable.

Image Credits: Boomerang

But despite the obvious intended use case here, Boomerang’s app was repeatedly flagged for the same “can’t uninstall app” reason by the Play Store’s app review process when it submitted updates and bug fixes.

This began on May 8th, 2020 and took over a month to resolve. The developer, Justin Payeur, submitted the first appeal on May 11th to test if the ban had just been triggered by Google’s “app review robots.” On May 13th, the app was re-approved without any human response or feedback to the appeals message he had sent to Google.

But then on June 30th, Boomerang was again flagged for the same reason: “can’t uninstall app.” Payeur filed a second appeal, explaining the feature is not on by default — it’s there for parents to use if they choose.

On July 6th, Boomerang had to inform users of the problem, as they had become increasingly frustrated they couldn’t find the app on Google Play. In a customer email that didn’t mince words, Boomerang wrote: “Google has become evil.” Complaints from users said that if the app didn’t offer the “prevent uninstall” feature, it wouldn’t be worth using.

On July 8th, Boomerang received a reply from Google with more information, explaining that Google doesn’t allow apps that change the user’s device settings or features outside the app without user’s knowledge or consent. Specifically, it also cited the app’s use of the “Google Accessibility Services API” in a manner that’s  in violation with the Play Store terms. Google said the app wouldn’t be approved until it remove functionality that prevented a user from removing or uninstalling the app from their device.

This requirement, though rooted in user security, disadvantages parental control apps compared with Google’s own Family Link offering. As Google’s help documentation indicates, removing a child’s account from an Android device requires parents to input a passcode — it can’t simply be uninstalled by the end user (the child).

Boomerang later that day received a second violation notification after it changed the app to be explicitly clear to the end user (the child) that the Device Administrator (a parent) would have permission to control the device, mimicking other apps Boomerang said were still live on Google Play.

After two more days pass with no reply from the Appeals team, Boomerang requested a phone call to discuss. Google sent a brief email, saying it was merging the two active Appeals into one but no other information about the Appeal was provided.

On July 13th, Boomerang was informed Google was still examining the app. The company replied again to explain why a parental control app would have such a feature. The same day, Boomerang was alerted that older versions of its app in its internal testing area in the Play Console were being rejected. These versions were never published live, the company says. The rejections indicated Boomerang was “degrading device security” with its app.

The next day, Boomerang informed its user base that it may have to remove the feature they wanted and emailed Google again to again point out the app now has clear consent included.

Image Credits: Boomerang; Email complains of “material impact” to business 

Despite not having made any changes, Google informs Boomerang on July 16th it’s in violation of the “Elevated Privilege Abuse” section of the Google Play Malware policy. On July 19th, the company removed the additional app protection feature and on July 21st, Google again rejected the app for the same violation — over a feature that had now been removed.

Despite repeated emails, Boomerang didn’t receive any message from Google until an automated email arrived on July 24th. Again, Google sent no response to the emails where Payeur explains the violating feature had now been removed. Repeated emails through July 30th were also not responded to.

After hearing about Boomerang’s issues, TechCrunch asked Google on July 27th to explain its reasoning.

The company, after a few follow-ups, told TechCrunch on August 3rd that the issues with Boomerang — as later emails to Boomerang had said — were related to how the app implemented its features. Google does not allow apps to engage in “elevated privilege” abuse. And it doesn’t allow apps to abuse the Android Accessibility APIs to interfere with basic operations on a device.

Google also said it doesn’t allow any apps to use the same mechanism Boomerang does, including Google’s own. (Of course, Google’s own apps have the advantage of deep integrations with the Android OS. Developers can’t tap into some sort of “Family Link API,” for example, to gain a similar ability to control a child’s device.)

“We recognize the value of supervision apps in various contexts, and developers are free to create this experience with appropriate safeguards,” a Google spokesperson said.

More broadly, Boomerang’s experience is similar to what iOS parental control apps went through last year. Like those apps, Boomerang too bumped up against a security safeguard meant to protect an entire app store from abusive software. But the blanket rule leaves no wiggle room for exceptions. Google, meanwhile, argues its OS security is not meant to be “worked around” like this. But it has also at the same time offered no official means of interacting with its OS and own screen time/parental control features. Instead, alternative screen time apps have to figure out ways to basically hack the system to even exist in the first place, even though there’s clear consumer demand for their offerings.

Boomerang’s particular case also reveals the complexities involved with of having a business live or die by the whims of an app review process.

It’s easy enough to argue that the developer should have simply removed the feature and moved on, but the developer seemed to believe the feature would be fine — as evidenced by prior approvals and the approval received upon at least one of its appeals. Plus, the developer is incentivized to fight for the feature because it’s something users said they wanted — or rather, what they demanded, to make the app worth paying for.

Had someone from Google just picked up the phone and explained to Boomerang what’s wrong and what alternative methods would be permitted, the case may not have dragged on in such a manner. In the meantime, Boomerang likely lost user trust, and its removal definitely impacted its business in the near-term.

Reached for a follow-up, Payeur expressed continued frustration, despite the app now being re-approved for Play Store distribution.

“It took Google over a month to provide us with this feedback,” he said, referencing the forbidden API usage that was the real problem. “We are currently digesting this”  he said, adding how difficult it was to not be able to talk to Google’s teams to get proper communication and feedback over the past several weeks.

Boomerang has begun collecting the names of other similarly impacted apps, lile Filter Chrome, Minder Parental Control, and Netsanity. The company says other apps can reach out privately to discuss, if they prefer.

After more than three decades climbing Apple’s ranks, marketing chief Phil Schiller is taking a step back at the company, being replaced by a long-time product marketing leader inside Apple.

Schiller is taking on a new role as an Apple Fellow where he will continue to lead the App Store and the company’s events, a press release details. Schiller has been with Apple since 1987, serving on the executive team for more than two decades, and has been a frequent presence onstage at Apple events. Schiller will continue to report directly to CEO Tim Cook in his new role.

Replacing Schiller and taking over the bulk of his responsibilities is Greg Joswiak, an Apple product marketing veteran who has become a more public face for the company in recent years at events and in media presentations. Joswiak’s promotion to senior vice president of Worldwide Marketing comes after nearly 20 years at Apple.

This appears to be Schiller moving to a largely advisory role with Apple employing some of its own marketing flourishes on the transition. Schiller’s maintenance of App Store messaging is interesting, especially as the company continues to be at the forefront of conversations around anti-competitive behavior among American tech companies. The App Store has been criticized for its revenue share model on digital services and CEO Tim Cook recently Zoom-testified in front of the House Antitrust Committee alongside other Big Tech CEO’s where the bulk of critiques levied at him by government officials seemed to focus on his handling of the App Store.

“Phil has helped make Apple the company it is today and his contributions are broad, vast, and run deep. In this new role he will continue to provide the incredible thought partnership, and guidance that have defined his decades at Apple,” said Cook in a statement. “Joz’s many years of leadership in the Product Marketing organization make him perfectly suited to this new role and will ensure a seamless transition at a moment when the team is engaged in such important and exciting work. I’m thrilled that the whole executive team will benefit from his collaboration, ideas, and energy.”

We’ve aggregated many of the world’s best growth marketers into one community. Twice a month, we ask them to share their most effective growth tactics, and we compile them into this Growth Report.

This is how you stay up-to-date on growth marketing tactics — with advice that’s hard to find elsewhere.

Our community consists of startup founders and heads of growth. You can participate by joining Demand Curve’s marketing training program or its Slack group.

Without further ado, on to our community’s advice.

More thoughts on growing podcasts

Insights from Harry Morton of Lower Street.

Podcast growth is all about relationships. To increase your listenership, consider partnering with:

1. Other podcasters. Do an episode swap where you play an episode of your show on theirs, and vice versa. Make sure the two podcasts share similarly minded audiences.
2. Curators. Every podcast aggregator has someone responsible for curating their featured content. Look them up on LinkedIn. Reach out via email. Be their friend. Send them only your best stuff.
3. Subscribers. You rise in Apple’s podcast charts (which account for 60% of podcast listenership) by having a subscriber growth spurt in a concentrated period of time (24-48 hours). So, when you release an episode, immediately run your audience promotions aggressively and all at once.

Increasing referral incentives might not increase referrals

Qualified, a startup co-founded by former Salesforce executives Kraig Swensrud and Sean Whiteley, has raised $12 million in Series A funding.

Swensrud (Qualified’s CEO) said the startup is meant to solve a problem that he faced back when he was CMO at Salesforce. Apparently he’d complaining about being “blind,” because he knew so little about who was visiting the Salesforce website.

“There could be 10 or 100 or 100,000 people on my website right now, and I don’t know who they are, I don’t know what they’re interested in, my sales team has no idea that they’re even there,” he said.

Apparently, this is a big problem in business-to-business sales, where waiting five minutes after a lead leaves your website can result in a 10x decrease in the odds of making contact. But the solution currently adopted by many websites is just a chatbot that treats every visitor similarly.

Qualified, meanwhile, connects real-time website visitor information with a company’s Salesforce customer database. That means it can identify visitors from high-value accounts and route them to the correct salesperson while they’re still on the website, turning into a full-on sales meeting that can also include a phone call and screensharing.

Image Credits: Qualified

Of course, the amount of data Qualified has access to will differ from visitor to visitor. Some visitors may be purely incognito, while in other cases, the platform might simply know your city or what company you work for. In still others (say if you click on a link from marketing email), it can identify you individually.

That’s something I experienced myself, when I decided to take a look at the Qualified website this morning and was quickly greeted with a message that read, “ Welcome TechCrunch! We’re excited about our funding announcement…” It was a little creepy, but also much more effective than my visits to other marketing technology websites, where someone usually sends me a generic sales message.

Swensrud acknowledged that using Qualified represents “a change to people’s selling processes,” since it requires sales to respond in real-time to website visitors (as a last resort, Qualified can also use chatbots and schedule future calls), but he argued that it’s a necessary change.

“If you email them later, some percentage of those people, they ghost you, they get bored, they moved on to the competition,” he said. “This real-time approach, it forces organizations to think differently in terms of their process.”

And it’s an approach that seems to be working. Among Qualified’s customers, the company says ThoughtSpot increased conversations with its target accounts by 10x, Bitly grew its enterprise sales pipeline by 6x and Gamma drove over $2.5 million in new business pipeline.

The Series A brings Qualified’s total funding to $17 million. It was led by Norwest Venture Partners, with participation from existing investors including Redpoint Ventures and Salesforce Ventures. Norwest’s Scott Beechuk is joining Qualified’s board of directors.

“The conversational model is simply a better way to connect with new customers,” Beechuk said in a statement. “Buyers love the real-time engagement, sellers love the instant connections, and marketers have the confidence that every dollar spent on demand generation is maximized. The multi-billion-dollar market for Salesforce automation software is going to adopt this new model, and Qualified is perfectly positioned to capture that demand.”

A 17-year-old Florida teenager is accused of perpetrating one of the year’s biggest and most high-profile hacks: Twitter.

A federal 30-count indictment filed in Tampa said Graham Ivan Clark used a phone spearphishing attack to pivot through multiple layers of Twitter’s security and bypassed its two-factor authentication to gain access to an internal “admin” tool that let the hacker take over any account. With two accomplices named in a separate federal indictment, Clark — who went by the online handle “Kirk” — allegedly used the tool to hijack the accounts of dozens of celebrities and public figures, including Bill Gates, Elon Musk and former president Barack Obama, to post a cryptocurrency scam netting over $100,000 in bitcoin in just a few hours.

It was, by all accounts, a sophisticated attack that required technical skills and an ability to trick and deceive to pull off the scam. Some security professionals were impressed, comparing the attack to one that had the finesse and professionalism of a well-resourced nation-state attacker.

But a profile in The New York Times describes Clark was an “adept scammer with an explosive temper.”

In the teenager’s defense, the attack could have been much worse. Instead of pushing a scam that promised to “double your money,” Clark and his compatriots could have wreaked havoc. In 2013, hackers hijacked the Associated Press’ Twitter account and tweeted a fake bomb attack on the White House, sending the markets plummeting — only to quickly recover after the all-clear was given.

But with control of some of the world’s most popular Twitter accounts, Clark was for a few hours in July one of the most powerful people in the world. If found guilty, the teenager could spend his better years behind bars.

Here’s more from the past week.

THE BIG PICTURE

Garmin hobbles back after ransomware attack, but questions remain

Rocket Lab has managed to engineer a significant payload capacity bump into its existing Electron space launch vehicle, the company revealed today. Electron can now fly as much as 300 kg (660 lbs) to low Earth orbit (or around 440 lbs to a higher, sun synchronous orbit), and that’s mostly due to battery technology advances, according to Rocket Lab.

Electron is not battery-powered, of course – but the electric pumps that help feed its Rutherford engines are. That’s where they’re getting the boost, along with some other optimizations, increasing the total payload capacity by a full third. That’s a lot of additional capacity in the small satellite launch market, where a CubeSat can weigh as little as 3 lbs or less.

Rocket Lab notes that this means customers who are useing their Photon spacecraft as a satellite bus (essentially the basic satellite platform upon which a company can build their specific instrumentation needs) will now have nearly 400 lbs available to them for their equipment, which should make possible a whole range of potential new applications.

The company announced last week that it was aiming to return to active launch status as early as this month, after an issue caused the early termination and failure of a mission in early July. It said it was able to quickly identify the problem and is already implementing a fix, and now it clearly wants to remind potential customers of its unique offerings and capabilities int he small satellite market.

Microsoft’s venture capital fund, M12 Ventures has led a slew of strategic corporate investors backing a new chip developer out of Southern California called Syntiant, which makes semiconductors for voice recognition and speech-based applications.

“We started out to build a new type of processor for machine learning, and voice is our first application,” says Syntiant chief executive Kurt Busch. “We decided to build a chip for alwyas-on battery powered devices.”

Those chips need a different kind of processor than traditional chipsets, says Busch. Traditional compute is about logic, and deep learning is about memory access, traditional microchip designs also don’t perform as well when it comes to parallel processing of information.

Syntiant claims that its chips are two orders of magnitude more efficient, thanks to its data flow architecture that was built for deep learning, according to Busch.

It’s that efficiency that attracted investors including M12, Microsoft Corp.’s venture fund; the Amazon Alexa Fund; Applied Ventures, the investment arm of Applied Materials; Intel Capital, Motorola Solutions Venture Capital and Robert Bosch Venture Capital.

These investment firms represent some of the technology industry’s top chip makers and software developers, and they’re pooling their resources to support Syntiant’s Irvine, Calif.-based operations.

Image Credits: Bryce Durbin / TechCrunch

“Syntiant aligns perfectly with our mission to support companies that fuel voice technology innovation,” said Paul Bernard, director of the Alexa Fund at Amazon. “Its technology has enormous potential to drive continued adoption of voice services like Alexa, especially in mobile scenarios that require devices to balance low power with continuous, high-accuracy voice recognition. We look forward to working with Syntiant to extend its speech technology to new devices and environments.” 

Syntiant’s first device measures 1.4 by 1.8 millimeters and draws 140 microwatts of power. In some applications, Syntiant’s chips can run for a year on a single coin cell.

“Syntiant’s neural network technology and its memory-centric architecture fits well with Applied Materials’ core expertise in materials engineering as we enable radical leaps in device performance and novel materials-enabled memory technologies,” said Michael Stewart, principal at Applied Ventures, the venture capital arm of Applied Materials, Inc. “Syntiant’s ultra-low-power neural decision processors have the potential to create growth in the chip marketplace and provide an effective solution for today’s demanding voice and video applications.” 

So far, 80 customers are working with Syntiant to integrate the company’s chips into their products. There are a few dozen companies in the design stage and the company has already notched design wins for products ranging from cell phones and smart speakers to remote controls, hearing aids, laptops and monitors. Already the company has shipped its first million units.  

“We expect to scale that by 10 x by the end of this year,” says Busch. 

Syntiant’s chipsets are designed specifically to handle wakes and commands, which means that users can add voice recognition features and commands unique to their particular voice, Busch says.

Initially backed by venture firms including Atlantic Bridge, Miramar and Alpha Edison, Syntiant raised its first round of funding in October 2017. The company has raised a total of $65 million to date, according to Busch.

“Syntiant’s architecture is well suited for the computational patterns and inherent parallelism of deep neural networks,” said Samir Kumar, an investor with M12 and new director on the Syntiant board. “We see great potential in its ability to enable breakthroughs in power performance for AI processing in IoT [Internet of things].” 

Rigetti Computing, the quantum computing startup that is challenging industry heavyweights like IBM, Microsoft and D-Wave, today announced that it has closed a $79 million Series C funding round. The round was led by Bessemer Venture Partners, with participation from Franklin Templeton, Alumni Ventures Group, DCVC, EDBI, Morpheus Ventures, and Northgate Capital.

Bessemer’s Tomer Diari will join the company’s board of directors and the company, together with Veritas Software’s former CEO Mark Leslie.

Earlier this year, TechCrunch reported that Rigetti was looking to raise about $71 million in what — at least at the time — looked to be a down round. A Rigetti spokesperson declined to share any details about the company’s valuation in this round.

“This round of financing brings us one step closer to delivering quantum advantage to the market,” said Rigetti founder and CEO Chad Rigetti. “The company is dually focused on building scalable, error-corrected quantum computers and supporting high-performance access to current systems over the cloud. Rigetti offers a distinctive hybrid computing access model designed for practical applications.”

Rigetti currently offers its own cloud-based service for access to its machines, as well as through AWS’ Braket service, which is currently in preview. The company also recently won an $8.6 million DARPA award to build a quantum computer that outperforms classical computers.

“It’s hard to find an area where quantum computing won’t be tremendously valuable once quantum advantage is achieved,” said Jonathan Curtis, vice president and portfolio manager at Franklin Equity Group. “We believe that Rigetti is one of a select few leaders in this important emerging market with a strong combination of leading technology, an accomplished and focused team, and important commercial, government, and go to market relationships.”

While quantum computing has long held a lot of promise, it’s actually starting to make real strides in the last few years, with various companies building working systems that aren’t quite powerful enough for most real-world use cases yet, but that show a lot of promise. Rigetti, maybe more so than others, has focused on these real-world use cases.

“Quantum computing will drive a paradigm shift in high-performance computers as we continue pushing the boundaries of science deeper into the realms of science fiction,” said Tomer Diari, a leading deep-tech investor from Bessemer Venture Partners who has joined the company’s board. “Quantum technology has the potential to unlock significant advancements in biology, chemistry, logistics and material science, and we believe that Rigetti provides the most immediate and clear path to a production-grade system in the market.”

In a typical security team, engineers write one-off scripts to track a particular problem on a cloud vendor such as an unauthorized user on your GitHub account, and while engineers are capable of writing such scripts, it’s not exactly an efficient or scalable way to deal with the range of security problems these pros need to track.

Vectrix, a member of the Y Combinator Summer 2020 cohort started by three security veterans, wants to fix that problem. It has created a security marketplace where fellow security pros write modules to automate these kinds of fixes, and other security pros can take advantage without reinventing the script writing wheel every time.

Alex Dunbrack, company co-founder and COO, says that he and fellow co-founders, CTO Matthew Lewis and CEO Corey Mahan saw this problem first-hand in their previous jobs at PlanGrid, Vimeo and Uber. So like so many YC company founders, they decided to build a solution.

“It’s a marketplace of automated security tools that monitor tech and have response capabilities for any security issues that a company may have within their cloud vendors,” Dunbrack explained. He says this could be on GitHub, AWS, G Suite, potentially any cloud service.

The idea is to have security professionals build these modules, then give them a “royalty” and bragging rights for coming up with a viable solution. Dunbrack says it’s not unlike the HackerOne model, which provides a financial incentive and community recognition to find vulnerabilities in code.

Users don’t actually download anything. They simply select a module, enter their cloud service credentials and provide an output like Slack or Jira for any alerts the module generates.

Image Credits: Vectrix

The startup vets the modules and the developers before allowing them in the marketplace. While this is a manual process at the moment, he says they are working on bringing more automation to it. For now, each person that wants to contribute modules, they do an interview, a reference check, employment background check and similar types of investigation.

Once they pass this, and the security pro writes the module, it has to pass further scrutiny. “We basically scope exactly what they’re going to build and the kinds of alerts that will come out of it. And then from there, we have an extremely templated logic scheme on the code side where they’re just writing the logic to go do the scan,” he said.

Module writers can’t see any user information on the service, and Vectrix makes sure there are no issues like outbound requests for data. Presently they have 10 modules with plans to add several more soon. While they are working on the pricing model, today customers pay a flat fee for access to the entire marketplace, rather than paying per module.

The company is currently just the three co-founders, but they hope to expand, and when they do they have already given a lot of thought about how to build a diverse and inclusive company. He says, for starters, they are not swayed by the Silicon Valley network effect.

“A lot of people will say ‘we simply want the best people,’ but our interpretation of the best people is really a collective of differing thoughts and experiences that really make someone’s perspective unique. That comes from diversity in the way that we see it, so in a lot of senses bringing the best people on is bringing the widest range of thinking processes, and that comes with diversity and being inclusive, and kind of taking all of those factors into account,” he said.

As for the YC experience, Dunbrack says he was mostly looking forward to learning from the network of companies that came before him, and he says that even virtually the company has succeeded in giving him that experience.

So far, the company has bootstrapped and used the money from Y Combinator, but it intends to do a fundraising round soon. “We’re cognizant of what we’re bringing to the industry and the value there. So bringing on strategic partners is really how we’re going to be approaching this,” he said.

At the end of every calendar year, the complaints from autonomous vehicle companies start piling up. This annual tradition is the result of a requirement by the California Department of Motor Vehicles that AV companies deliver “disengagement reports” by January 1 of each year showing the number of times an AV operator had to disengage the vehicle’s autonomous driving function while testing the vehicle.

However, all disengagement reports have one thing in common: their usefulness is ubiquitously criticized by those who have to submit them. The CEO and founder of a San Francisco-based self-driving car company publicly stated that disengagement reporting is “woefully inadequate … to give a meaningful signal about whether an AV is ready for commercial deployment.” The CEO of a self-driving technology startup called the metrics “misguided.” Waymo stated in a tweet that the metric “does not provide relevant insights” into its self-driving technology or “distinguish its performance from others in the self-driving space.”

Why do AV companies object so strongly to California’s disengagement reports? They argue the metric is misleading based on lack of context due to the AV companies’ varied testing strategies. I would argue that a lack of guidance regarding the language used to describe the disengagements also makes the data misleading. Furthermore, the metric incentivizes testing in less difficult circumstances and favors real-world testing over more insightful virtual testing.

Understanding California reporting metrics

To test an autonomous vehicle on public roads in California, an AV company must obtain an AV Testing Permit. As of June 22, 2020, there were 66 Autonomous Vehicle Testing Permit holders in California and 36 of those companies reported autonomous vehicle testing in California in 2019. Only five of those companies have permits to transport passengers.

To operate on California public roads, each permitted company must report any collision that results in property damage, bodily injury, or death within 10 days of the incident.

There have been 24 autonomous vehicle collision reports in 2020 thus far. However, though the majority of those incidents occurred in autonomous mode, accidents were almost exclusively the result of the autonomous vehicle being rear-ended. In California, rear-end collisions are almost always deemed the fault of the rear-ending driver.

The usefulness of collision data is evident — consumers and regulators are most concerned with the safety of autonomous vehicles for pedestrians and passengers. If an AV company reports even one accident resulting in substantial damage to the vehicle or harm to a pedestrian or passenger while the vehicle operates in autonomous mode, the implications and repercussions for the company (and potentially the entire AV industry) are substantial.

However, the usefulness of disengagement reporting data is much more questionable. The California DMV requires AV operators to report the number and details of disengagements while testing on California public roads by January 1 of each year. The DMV defines this as “how often their vehicles disengaged from autonomous mode during tests (whether because of technical failure or situations requiring the test driver/operator to take manual control of the vehicle to operate safely).”

Operators must also track how often their vehicles disengaged from autonomous mode, and whether that disengagement was the result of software malfunction, human error, or at the option of the vehicle operator.

AV companies have kept a tight lid on measurable metrics, often only sharing limited footage of demonstrations performed under controlled settings and very little data, if any. Some companies have shared the occasional “annual safety report,” which reads more like a promotional deck than a source of data on AV performance. Furthermore, there are almost no reporting requirements for companies doing public testing in any other state. California’s disengagement reports are the exception.

This AV information desert means that disengagement reporting in California has often been treated as our only source of information on AVs. The public is forced to judge AV readiness and relative performance based on this disengagement data, which is incomplete at best and misleading at worst.

Disengagement reporting data offers no context

Most AV companies claim that disengagement reporting data is a poor metric for judging advancement in the AV industry due to a lack of context for the numbers: knowing where those miles were driven and the purpose of those trips is essential to understanding the data in disengagement reports.

Some in the AV industry have complained that miles driven in sparsely populated areas with arid climates and few intersections are miles dissimilar from miles driven in a city like San Francisco, Pittsburgh, or Atlanta. As a result, the number of disengagements reported by companies that test in the former versus the latter geography are incomparable.

It’s also important to understand that disengagement reporting requirements influence AV companies’ decisions on where and how to test. A test that requires substantial disengagements, even while safe, would be discouraged, as it would make the company look less ready for commercial deployment than its competitors. In reality, such testing may result in the most commercially ready vehicle. Indeed, some in the AV industry have accused competitors of manipulating disengagement reporting metrics by easing the difficulty of miles driven over time to look like real progress.

Furthermore, while data can look particularly good when manipulated by easy drives and clear roads, data can look particularly bad when it’s being used strategically to improve AV software.

Let’s consider an example provided by Jack Stewart, a reporter for NPR’s Marketplace covering transportation:

“Say a company rolls out a brand-new build of their software, and they’re testing that in California because it’s near their headquarters. That software could be extra buggy at the beginning, and you could see a bunch of disengagements, but that same company could be running a commercial service somewhere like Arizona, where they don’t have to collect these reports.

That service could be running super smoothly. You don’t really get a picture of a company’s overall performance just by looking at this one really tight little metric. It was a nice idea of California some years ago to start collecting some information, but it’s not really doing what it was originally intended to do nowadays.”

Disengagement reports lack prescriptive language

The disengagement reports are also misleading due to a lack of guidance and uniformity in the language used to describe the disengagements. For example, while AV companies used a variety of language, “perception discrepancies” was the most common term used to describe the reason for a disengagement — however, it’s not clear that the term “perception discrepancies” has a set meaning.

Several operators used the phrase “perception discrepancy” to describe a failure to detect an object correctly. Valeo North America described a similar error as “false detection of object.” Toyota Research Institute almost exclusively described their disengagements vaguely as “Safety Driver proactive disengagement,” the meaning of which is “any kind of disengagement.” Whereas, Pony.ai described each instance of disengagement with particularity.

Many other operators reported disengagements that were “planned testing disengagements” or that were described with such insufficient particularity as to be virtually meaningless.

For example, “planned disengagements” could mean the testing of intentionally created malfunctions, or it could simply mean the software is so nascent and unsophisticated that the company expected the disengagement. Similarly, “perception discrepancy” could mean anything from precautionary disengagements to disengagements due to extremely hazardous software malfunctions. “Perception discrepancy,” “planned disengagement” or any number of other vague descriptions of disengagements make comparisons across AV operators virtually impossible.

So, for example, while it appears that a San Francisco-based AV company’s disengagements were exclusively precautionary, the lack of guidance on how to describe disengagements and the many vague descriptions provided by AV companies have cast a shadow over disengagement descriptions, calling them all into question.

Regulations discourage virtual testing

Today, the software of AV companies is the real product. The hardware and physical components — lidar, sensors, etc. — of AV vehicles have become so uniform, they’re practically off-the-shelf. The real component that is being tested is software. It’s well known that software bugs are best found by running the software as often as possible; road testing simply can’t reach the sheer numbers necessary to find all the bugs. What can reach those numbers is virtual testing.

However, the regulations discourage virtual testing as the lower reported road miles would seem to imply that a company is not road-ready.

Jack Stewart of NPR’s Marketplace expressed a similar point of view:

“There are things that can be relatively bought off the shelf and, more so these days, there are just a few companies that you can go to and pick up the hardware that you need. It’s the software, and it’s how many miles that software has driven both in simulation and on the real roads without any incident.”

So, where can we find the real data we need to compare AV companies? One company runs over 30,000 instances daily through its end-to-end, three-dimensional simulation environment. Another company runs millions of off-road tests a day through its internal simulation tool, running driving models that include scenarios that it can’t test on roads involving pedestrians, lane merging, and parked cars. Waymo drives 20 million miles a day in its Carcraft simulation platform — the equivalent of over 100 years of real-world driving on public roads.

One CEO estimated that a single virtual mile can be just as insightful as 1,000 miles collected on the open road.

Jonathan Karmel, Waymo’s product lead for simulation and automation, similarly explained that Carcraft provides “the most interesting miles and useful information.”

Where we go from here

Clearly there are issues with disengagement reports — both in relying on the data therein and in the negative incentives they create for AV companies. However, there are voluntary steps that the AV industry can take to combat some of these issues:

1. Prioritize and invest in virtual testing. Developing and operating a robust system of virtual testing may present a high expense to AV companies, but it also presents the opportunity to dramatically shorten the pathway to commercial deployment through the ability to test more complex, higher risk, and higher number scenarios.
2. Share data from virtual testing. Voluntary disclosure of virtual testing data will reduce reliance on disengagement reports by the public. Commercial readiness will be pointless unless AV companies have provided the public with reliable data on AV readiness for a sustained period.
3. Seek the greatest value from on-road miles. AV companies should continue using on-road testing in California, but they should use those miles to fill in the gaps from virtual testing. They should seek the greatest value possible out of those slower miles, accept the higher percentage of disengagements they will be required to report, and when reporting on those miles, describe their context in particularity.

With these steps, AV companies can lessen the pain of California’s disengagement reporting data and advance more quickly to an AV-ready future.