INTRODUCTION: “WHAT CAN OPEN SOURCE INTELLIGENCE DO FOR ME?”
…is probably the main question that you had in mind when you picked up this book.
After spending more than a decade working within the field of intelligence and security, and five years teaching open source intelligence (OSINT) techniques to hundreds of professionals within the military, police, counter fraud, academia, non-governmental organisation (NGO) and government sectors, I have been asked that question a thousand times and I can honestly say that I am no closer to giving an accurate answer now than when I was originally asked the question.
Although modern OSINT is many things, it’s perhaps easier to fully conceptualise its capability by defining what OSINT is not. OSINT is not the silver bullet that will blow your stalled investigation wide open, and neither is it just another faddy ‘must have’ capability; it is certainly not a frightening and soulless computer-based technology that is here to take your job. Instead, OSINT is just another arrow within the quiver of the investigative analyst, just like techniques such as interviewing, surveillance, fingerprinting and any number of others open to the skilled professional investigator or analyst.
Wikipedia defines the term as:
“Open-source intelligence (OSINT) is intelligence collected from publicly available sources. In the intelligence community (IC), the term “open” refers to overt, publicly available sources (as opposed to covert or clandestine sources);”1
Based even on the loosest interpretation of the preceding definition, it is obvious that OSINT is not a new capability, given how accessible and easy to use it is. What has changed the game for OSINT is the arrival of the Internet.
Until usage of the Internet became widespread, OSINT was confined to primarily paper-based resources such as libraries and other common paper media such as newspapers, industry publications, fliers and propaganda. This ‘old school’ form of OSINT is of course useful for analysis and investigation; indeed anyone who has seen legendary Central Intelligence Agency analyst Sherman Kent’s notebooks will testify that they are filled with newspaper clippings. However, older OSINT research was limited by both the coverage of its information and the ability of the researcher to focus the capability on a specific subject, be it a person, location or topic. In plain terms, with paper-based Open Source: what you saw was what you got and that was that.
What has changed this status quo is the arrival of the Internet, and particularly the explosion in the use of social media technology circa 2000. The rise of these two technologies created a multilingual, geographically distributed, completely unregulated publishing platform to which any user could also become an author and a publisher. The effect of this was to vastly expand both the coverage of the topics covered by OSINT and to increase the ability of the researcher to focus OSINT capabilities to predefined information requirements. By increasing the coverage and focus of OSINT the Internet effectively promoted OSINT from a supporting role to finally sit alongside other more clandestine and less accessible investigative capabilities5.
Talking point: the Internet and cyberspace
Have you ever wondered what the difference (if any) is between the term ‘cyberspace’ and ‘the Internet’?The term ‘cyberspace’ was popularised in the 1980s by the science fiction author William Gibson, in his seminal cyberpunk novels Burning Chrome2 and Neuromancer3. Gibson’s intent behind using the term ‘cyberspace’ was to encapsulate a number of fictional communication technologies as a foil for his books’ plotlines. Later interpretations of what cyberspace actually was became more defined, with the Oxford English Dictionary currently defining the term as “The notional environment in which communication over computer networks occurs.” In contrast to the coldness of the term ‘cyberspace’, the term ‘the Internet’, which came into common usage about the 1970s, came to represent the networking technologies that fostered communication between users and in so doing placed the importance of human users at its core. The important point in all this is to understand that cyberspace is much more than just the Internet; it is conceptually an artificial, virtual space that may or may not be connected to the wider Internet. For example, many industrial control systems that run Critical Nation Infrastructure are made up of networks of computers that are not connected to the Internet; however, this is still cyberspace. It’s an interesting quirk of cyber culture that Gibson ultimately rejected the term cyberspace, commenting in 2000: “All I knew about the word ‘cyberspace’ when I coined it, was that it seemed like an effective buzzword. It seemed evocative and essentially meaningless. It was suggestive of something, but had no real semantic meaning, even for me…4”
With the amazing statistic that based on current trends, four out of five people globally will have a Facebook account by 2020, and with companies such as Google conducting research into using radical technologies such as weather balloons and aquatic buoys to spread Internet access to mountains and oceans, the importance of OSINT for investigators is only set to grow.
You probably picked up this book because you had become aware either from a colleague, conference, job advert or other source that OSINT was gaining prominence within the investigative community and you felt that you needed some kind of guide to get you up to speed with the current state of the art. Helping you on this journey is what this book is all about.
An effective OSINT capability is not simply a process list of things to do at certain points within an investigation, nor is it a ‘black box’ capability. It is in fact a combination of technical expertise, domain-specific knowledge and old-fashioned investigative smarts. As Richards J. Heuer, Jr, a giant in the intelligence analysis field, said, an effective intelligence capability is a blend of an art, craft and a science6 and OSINT is no exception.
With regard to the approach that this work takes to the tools used for OSINT research, it’s worth referencing another quotation from cyberpunk author William Gibson, who famously said “…the street finds its own uses for things7.” This statement, which encapsulates the sentiment that users will find uses for technology that the original designers never envisaged, is a core theme that pervades this book’s approach to the tools that can be applied to OSINT research. You will be introduced to tools designed primarily for salesmen, web designers, computer hackers and the casual web user that although never intended for investigative use, have unique features that can be rallied to the investigator’s cause.
The objective of this book is not to provide an exhaustive list of tools (Michael Bazzell’s excellent work8 does a far better job at that). Instead, it seeks to instruct you on a philosophical approach to effectively using OSINT within professional investigative work. Although tools and services are examined within the book (more than 30 in fact), they are only looked at to underline a higher-level investigative concept and to demonstrate the function of that specific class of tool. The reasoning behind this approach is that the OSINT toolbox is ever changing as old tools wither and die and new tools rise to replace them; however, the underlying principles (i.e. an understanding of the principles that underline a successful OSINT investigation) will equip the knowledgeable practitioner for years to come.
And on that note…
Four key concepts
This book is built around the following four foundations that underpin the entire field of OSINT as a professional practice:
1. Multilayered – there are three layers to the Internet: the Surface, Deep and Dark Web. They are commonly misunderstood; a working conceptualisation within the mind of the investigator is essential within effective OSINT professional practice. The beginnings of definitions of these terms are given below and are expanded upon through the remainder of this book.
a. Surface Web – is the part of the Internet that is accessible via mainstream web browsers such as Google or Bing. Typically, sites within this layer of the Internet are designed to be listed on mainstream search engines that are intended to be found easily by the casual web user. Much of the information on the Surface Web is common knowledge that is not sensitive.
b. Deep Web – is the part of the Internet that is not listed (or to use the more technical term, ‘indexed’) by the main search engines. The reason for this is that the technology used in Deep Web publishing platforms such as Facebook, Twitter and LinkedIn is not designed to be read and understood by the technology that drives searches engines such as Google. To be specific, Deep websites are not designed to be hidden by the publisher, it’s just that the contents of Deep websites cannot be read by conventional search engines. Most of the information on the Surface Web has been placed there by the subject of the information via social media technologies such as Facebook. As such, a lot of the time most of the information that an investigator is seeking on an individual within the context of an investigation is located within the Deep Web; the main challenge for the investigator is getting access to this data.
c. Dark Web – is possibly the most mysterious and misreported of the three layers of the Internet. The Dark Web can only be accessed via the use of specialist pieces of anonymising software such as The Onion Router (TOR). For the most part, websites and services placed onto the Dark Web are meant to be hidden from all but the most informed and technically savvy web user and often contain deeply criminal content (drugs, guns and child pornography are rife) vended on illicit online marketplaces. As it is the most overtly criminal section of the Internet, the Dark Web is a place only to go to if you have a specific remit to investigate crime within this space.
2. Cyber geography – just like physical geography, the Internet has its own unique ‘regions’ based upon linguistic divisions. For example, Russophone Internet users publish to the Internet in Russian and not English. Additionally, different linguistic groups publish different volumes of content to the Internet. As Anglophones have been connected to the Internet for the longest period, by far the dominating language (by volume) of the Internet is English. However, that is not to say that there are not many hundreds of thousands of pages published in other languages as well. Although this point may seem obvious with hindsight, cyber geography is a property of the Internet that is often overlooked by the novice and expert investigator alike. The impact of not fully appreciating this point is that if an investigator limits his or her search to just one linguistic sphere within one specific cyber geography then they are vastly diminishing their chance of success.
3. Mixed medium – at the layer of the user experience, the Internet (Surface, Deep or Dark) is not made up of one single technology. Instead, it is a complex mixture of searching and display technologies that all have their own unique set of rules that require specific techniques for the experienced investigator to penetrate. Exhaustively enumerating all the different technologies and approaches required by the OSINT practitioner is not the objective of this book. What this book does seek to do is equip the investigator with the skills necessary to fully assess the risks and operational requirements needed to penetrate a new technological space on the Internet.
4. Tangibility – often the Internet is actively distinguished from ‘the real world’. By using this term, it is implied that the Internet is somehow less tangible and hence less important to the human experience than events that occur within a physical space. Although this may have been the case 20 years ago, the Internet has become so deeply intertwined with our daily lives that for many users (particularly within Western economies) access to the Internet is as important as the provision of a stable electricity grid or clean water supply. Recent world events such as the Arab Spring and the ongoing unrest in Ukraine have demonstrated how integral the Internet has become to people’s lives.
To credit myself with the observation of the preceding points would be wrong; instead, I merely advocated their centrality to OSINT as a professional practice.
What to expect from the rest of this book
The remainder of this book is built around the different tools and investigative approaches that are required when conducting research within the Surface, Deep and Dark Webs. Although there is more to OSINT than simply examining these three divisions of the Internet, this is an appropriate approach to OSINT due to the context that the focus on each web layer provides. This book will serve as more than merely a list of tools and instead facilitate the growth of an individual’s ability to integrate OSINT into their existing investigative skill set. It is intended to be an explanation of theory via the demonstration of tools.
Although the value of the knowledge presented within the next chapters will vary depending upon the objective of the individual investigator, each concept should be examined in the order that they are presented. The professional engaged in due diligence will no doubt find the Deep Web chapter more immediately useful than the professional engaged in national security, but the threat landscape changes quickly and previously ‘nice to know’ knowledge quickly becomes the key to unlocking a future investigation9. Additionally, concepts are introduced early within this book and revisited in increasing detail through the remaining chapters.
1 http://en.wikipedia.org/wiki/Open-source_intelligence#cite_note-1
2 Gibson, W. (1995). Burning Chrome. Harper Voyager; New Ed edition (27 Nov 1995)
3 Gibson, W. (1995). Neuromancer. Harper voyager; New Ed Edition (27 Nov 1995)
4 Neale, M. (2000). No Maps for These Territories. Documentary Distributed by Docurama
5 Although purely paper-based OSINT is still very much in existence, and will always play a part within the investigative process, within this book the term ‘OSINT’ refers to purely cyberspace-based research activities.
6 Heur, R. (1999). Psychology of Intelligence Analysis. Center for the study of intelligence. Available from www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/
7 Gibson, W. (1995). Burning Chrome. Harper Voyager; New Ed edition (27 Nov 1995). Page 215
8 Bazzell, M. (2014). Open Source Intelligence Techniques: Resources for Searching and Analyzing Online Information. CreateSpace Independent Publishing Platform; 3rd edition (1 Jan 2014)
9 A prime example being how quickly Twitter aggregation tools, previously primarily only used for marketing, were quickly pressed into the service of nation security during the London riots of 2011.