Automated OSINT can be described as the collection of publicly available data with the aid of specialist software or web applications (‘tools’). Open Source Intelligence (OSINT) is often viewed as a non-intrusive type of surveillance, since the data is available to anyone by using search engines such as Google or by purchase of data.
However, traditional OSINT has evolved into a professional and intrusive practice. With the use of tools hundreds of online sources can be queried simultaneously. These sources are diverse and can range from publicly available data on social media services, to location data generated by advertisements on apps on mobile phones, to leaked user data. Automated OSINT interferes with the right to privacy and the right to the protection of personal data more seriously than before. The Dutch Review Committee on the Intelligence and Security Services (CTIVD) recently published a report about automated OSINT which illustrates this development and addresses the legal questions that arise from it (the summary is available in English).
In this blog, I will briefly describe automated OSINT and explain how it interferes with the right to privacy. To make clear why states should think about (regulating) automated OSINT, I will briefly discuss the Dutch regulations and the outcome of the review by the CTIVD.
Privacy and automated OSINT
OSINT no longer consists of ‘checking telephone directories’ or ‘searching for data on the internet using a search engine’. Instead, dedicated software can be used that queries hundreds of sources simultaneously, including data on social media services, location data generated by advertisements on apps, on mobile phones and leaked user data.
Well known tools for automated OSINT, developed by private companies, are for example ‘SpiderFoot’, ‘Recon-ng’, ‘Maltego’ and ‘Spyse’. For example, SpiderFoot markets itself on their website as a “a reconnaissance tool that automatically queries over 100 public data sources (OSINT) to gather intelligence on IP addresses, domain names, e-mail addresses, names and more”.
In its report, the CTIVD considers the processing of location data and leaked data as more intrusive than – for example – processing publicly available data from a website. Legal arguments for this standpoint with regard to location data can be found in cases of the European Court of Justice, such as La Quadrature du Net (2020, para. 117). The fact that the possession or distribution of leaked data is criminalised in many countries, is also indicative for the seriousness of the privacy interference when this data is processed by governmental authorities. Due to this serious privacy interference, the risks to the right to privacy and right to data protection need to be identified and remedied with accompanying safeguards.
The CTIVD reviewed to which extent the practice complied with data protection regulations in the Dutch Intelligence and Security Services Act 2017 (‘Wiv 2017’). The outcome was that the Dutch services should first make an assessment on what data is processed and how it is processed before automated OSINT tools are employed. The CTIVD made clear that after the assessment, measures should be taken to minimise the risk of processing personal data unlawfully – such as detailed and obligatory logging when these tools are used. Then, based on experience and detailed in working instructions, the intelligence and services can decide whether the use of these tools and copying results to operational reports is necessary and proportionate.
In the Netherlands, OSINT by intelligence and security services is actually specifically regulated.
An investigative power (regulated in article 38 Wiv 2017) must be applied when ‘publicly available information is gathered and subsequently stored in a file and when it is expected beforehand that a serious privacy interference will take place’. Prior to storing the data, it must be explained why and for what purpose the data is collected and why it is proportionate (among other criteria). There is no independent prior authorisation required to use this investigative power, but the application and results must be registered.
A public debate about automated OSINT
The companies behind these automated OSINT tools make money by processing and making publicly available data available after payment. These companies also combine data and offer it as a searchable dataset, which may contain up to billions of records (according to the CTIVD report). Other businesses and governmental institutions, such as law enforcement agencies and intelligence and security services, obtain licenses to make use of these tools and access these datasets made available by these tools.
In the United States, a public debate about the use of OSINT tools and the processing location data by intelligence agencies has been taking place for some time now. See, for example, this article: ‘Intelligence Analysts Use U.S. Smartphone Location Data Without Warrants, Memo Says’. Some senators argue that a ‘warrant’ (a judicial authorization of a judge) is required when location data of American citizens or residents in the United States are analysed. Also, an oversight authority has announced that it will further investigate OSINT conducted by the FBI.
Within Europe, there is no debate about the question whether and under which conditions intelligence and security services can make use of these tools and access datasets. A quick scan shows that many states still view Open Source Intelligence as a non-intrusive surveillance power and do not regulate OSINT in statutory law.
Intelligence and security services should be able to access and make use of open source information and state of the art tools, within the boundaries of the law and with respect to data protection principles. I don’t think it is desirable to ban the use of these tools or require a warrant (or other prior authorization by an independent authority) for each search with these kinds of tools. However, it does require that adequate safeguards are in place, fit to deal with the described evolution and practice of automated OSINT.
Please let me know in the comments below if your country has specific regulations for OSINT in national security legislation or these are desirable.