Although these conversations are supposedly anonymised, this is, in itself, intrusive. As this breach indicated, the conversations themselves may contain sensitive personal data that identifies users along with locations. Also, it confirmed assistants can switch on and record in error.
This also happens with Alexa. Users can playback transcripts of Alexa conversations and this recording error happens with some frequency. This glitch happens when the “wakeup” word process goes wrong. Sometimes this can be triggered by random conversation, or some background noise, that the machine “hears” as a wakeup word. In effect, a digital assistant includes a microphone that is always on and listening. The programme is supposed to start recording only when the wakeup is used. But the mike itself is always on. Even if devices offer a push switch to activate, users tend to keep the mike on by default because that is inherent to functionality.
That programme can thus, be started by accident and it isn’t under the user’s control. Those recordings are sent back to the mother servers for analysis, including human analysis. This helps to improve functionality and it is critical to developing the technology that makes assistants useful.
But this process is insecure. Contractors hired to transcribe and analyse such recordings may choose to break confidentiality. For that matter, a server hack could put such data into the wrong hands.
Anonymising can also lead to poorer functionality. One reason why some users claim Siri lags Google Assistant and Alexa in the things it can do, could be related to anonymising. Siri anonymises to greater extent. Users can’t playback and human analysis is only done with a random tag attached to identify specific conversations. But this also means that Siri doesn’t learn the individual quirks and tastes of specific users.
Amazon and Google monetise their assistants’ interactions in ways, which makes them reluctant to anonymise. They do big data analysis and meta-data analysis to discover patterns of searches, online purchases, entertainment choices, appointments, medical conditions, phone calls and text messages etc. It is even possible for assistants to accurately gauge the user’s mood and mental health from tonality.
This analysis builds more complete profiles that can be used for targeted marketing and personalised advertising initiatives. It can seem like magic when Amazon offers you a choice of TV shows, music and books that you really like, or Google throws up ads about destinations you’re interested in visiting. This is actually enabled by this sort of analysis of your online usage patterns and your interactions with assistants.
That convenience comes with a cost. Users need to find a balance in the trade off between loss of privacy and convenience. This would mean a far greater degree of transparency, and user control over the data being harvested. Unfortunately, most jurisdictions don’t have legislation that enables this. The European Union’s general data protection regulation (GDPR) is pioneering legislation in this area but most nations don’t have privacy legislation of this strength. That in effect means that the corporate service provider decides how much privacy it will allow.
There have been thought experiments and speculation about what the service provider should do, if an assistant picks up a conversation about criminal or terrorist activity, or hears an incident of domestic violence. That’s a grey area but it’s likely this has happened. The vast majority of recordings would be about the mundane but they would also provide lots of private information. In this version of dystopia, it isn’t only Big Brother that’s watching.