Security Tools

An Introduction to Automating Open Source Intelligence Using SpiderFoot

SpiderFoot - An Open Source Intelligence Tool

What Is OSINT?

Open Source Intelligence (OSINT) is a methodology for collecting, analyzing, and decision-making using publicly available sources of data. According the Wikipedia, OSINT sources can be devided into te following categories:

  • Media, print newspapers, magazines, radio, television
  • Internet, online publications, blogs, discussion groups, citizen media
  • Public government data, public government reports, budgets, hearings, telephone directories, press conferences, websites, speaches
  • Professional and academic publications, information acquired from journals, conferences, symposia, academic papers, dissertations, theses
  • Commercial data, commercial imagery, financial and industrial assessments, and databases
  • Grey literature, technical reports, preprints, patents, working papers, business documents, unpublished works, newsletters

The purpose of OSINT is to create a tailored level of knowledge (or intelligence) for supporting individuals and groups in making decisions.

A vast amount of information is available publicly. OSINT Framework provides a hierarchical view of hundreds of OSINT resources broken down by a variety of indicators.

What Is SpiderFoot?

SpiderFoot is an open source tool, built in Python, that can query a large number of data sources (over 100 according the website) to gather information on a number of different targets including ip addresses, domain names, and even bitcoin addresses.

SpiderFoot Scan Target Panel

The power of SpiderFoot comes from Modules. Modules are how SpiderFoot organizes data into containers. Some Modules like those that integrate with Shodan, AlienVault OTX, and HaveIBeenPwned, required an API key from those individual services. API Keys can be imported/exported as needed. Approximately 60 services that require API’s are available via SpiderFoot.

Scanning in SpiderFoot is as simple as giving the scan a title, a target, and then by selecting the Use Case, Required Data, or Modules that you’d like to use. Scans can be as detailed or as broad as you’d like.

SpiderFoot Scan Settings Panel

Results are available via several dashboards including the Summary visual below. You can also browse the data in a table, and exclude duplicates, as well as view the data in a graph showing you the connections between data points.

Spiderfoot Scan Summary Panel

In summary, SpiderFoot is a web-based tool for collecting, analyzing and storing OSINT data, and is completely open source. It has its limits, like only being able to complete one scan at a time. However, it’s so easy to setup and can be virtualized using Python Virtual Environments, that analysts can easily have their own instances.

CyberChef Cyber “Swiss-Army” Knife – Free Tool For Security Professionals

Cyber Chef Logo

CyberChef is a web-based tool that assists with carrying out a number of complex operations such as compressing or decompressing data, encrypting data, creating binary and hex dumps, extracting metadata, matching YARA rules, and so much more. It was created by the GCHQ or Government Communications Headquarters, which is an intelligence service in the United Kingdom. It’s completely open source and is available for anyone and everyone on GitHub.

CyberChef is an absolute must for any security operations analyst or security professional, and can save you time in almost every use case. That is why we at Rogue Security are proud to introduce the Rogue Security hosted, CyberChef instance!

Direct Link: https://cyberchef.roguesecurity.ca/

Disclaimer: Rogue Security will not take responsibility for any data that you use within the tool. Please make sure that you never place secret of confidential data into a tool that you do not have full trust on.

We are making CyberChef available to the general public because we believe that access to tools and resources is critical to the growth of Information Security and Cybersecurity.

CyberChef Interface

Cyberchef Interface
Cyberchef Interface

The CyberChef interface is made up of 4 areas:

1. Input — The Input field provides an area to enter or paste your text or file input.

2. Output — The Output field provides the outcome of your recipe.

3. Operations — The Operations menu provides you wish access to both simple and complex operations, in which will be performed against the input.

4. Recipe — In the Recipe field, you will use any number of Operations that will determine how your input will be processed.

We can use the Input Menu to upload folders and files. The plus ( + ) sign allows us to create multiple tabs for inputs. The Input window will also allow you to drag and drop files, or paste text directly.

Operations list in CyberChef
Operations list in CyberChef

The Operations menu gives you a list of both simple and complex operations that can be performed against

The screenshot to the right provides a list of operations categories that can be performed. I won’t go through the list because it’s extensive, but if you can think about a data manipulation technique then it’s probably here.

Operations can be hovered over to see additional details on what they do, and can be dragged into the recipe window for use against your input. Once your drag an operation into the recipe window, and as long as you have an input and the Auto Bake feature turned on, an output will automatically be generated in the Output window.

CyberChef Resources