Parsing NMAP XML output

Last blog looked at the DB Schema to be used to store the data from NMAP Scans and mentioned the next step is to look at Parsing the NMAP XML output into something that can be stored in MySQL.

Building the code to do this operation from scratch seemed like too much work so I borrow some code I found posted by Yunshu on google code. 

He has written his on Class and broken the various facets of the file into modules that he imports into his main Parser class. As Nmap already had a 'Parser' module I changed the naming convention to 'NmapParse' in all of the files. The code also uses the xml.dom.minidom module to assist in the reading of the file format.

So now the Nmap scan can:

• Detect it's hosts IP
• Run a scan based on this to enumerate other hosts on the network
• Publish the results into an XML file
• Read the XML file into a series of local variables

The next steps are to push the local variables into the MySQL DB schema that has been created. Once in the MySQL tables the data can be transformed into useful information such as.

• Show when a host joined and left the network.
• Show when a certain port became available.
• Show when a HOSTs configuration changes.

The next blog will look at how to push the data to the tables and setup the SQL queries and produce reports on the data.

DB Schema for NMAP in MySQL

I now have two scripts one that connects to the remotely hosted MySQL DB and the other that starts NMAP and runs a scan with XML output.

The next step is to create some code that will read the XML output file and import it into the DB. Once the scans are in the DB all kinds of queries can be executed on the data.

DB-SCHEMA

The NMAP manual has this to say about importing XML:
For projects large enough to require a database, I recommend deciding on an optimal DB schema first, then writing a simple program or script to import Nmap XML data appropriately. Such scripts often take only minutes, thanks to the wide availability of XML parsers and database access modules. Perl often makes a good choice, as it offers a powerful database abstraction layer and also custom Nmap XML support. the section called “Manipulating XML Output with Perl” shows how easily Perl scripts can make use of Nmap XML data.

The links in the NMAP documentation where all out of date and the newer versions of NMAP have introduced new functionality which means the DB Schema needs to be extended. I found a good article on the BLOG at Redspin a penetration testing company. I will use their schema with some refinements to the DB structure to use less space as they have been very generous in there field allocations.

TABLE nmap (

    sid INTEGER PRIMARY KEY AUTOINCREMENT,

    version TINYTEXT,

    xmlversion TINYTEXT,

    args TEXT,

    types TEXT,

    starttime DATETIME,

    startstr TEXT,

    endtime DATETIME,

    endstr TEXT,

    numservices INTEGER) 

TABLE hosts (

    sid INTEGER,

    hid INTEGER PRIMARY KEY AUTOINCREMENT,

    ip4 VARCHAR(15),

    ip4num INTEGER,

    hostname TEXT,

    status TEXT,

    tcpcount INTEGER,

    udpcount INTEGER,

    mac CHAR(12),

    vendor TEXT,

    ip6 TEXT,

    distance INTEGER,

    uptime TEXT,

    upstr TEXT) 

TABLE sequencing (

    hid INTEGER,

    tcpclass TEXT,

    tcpindex TEXT,

    tcpvalues TEXT,

    ipclass TEXT,

    ipvalues TEXT,

    tcptclass TEXT,

    tcptvalues TEXT) 

TABLE ports (

    hid INTEGER,

    port INTEGER,

    type TEXT,

    state TEXT,

    name TEXT,

    tunnel TEXT,

    product TEXT,

    version TEXT,

    extra TEXT,

    confidence INTEGER,

    method TEXT,

    proto TEXT,

    owner TEXT,

    rpcnum TEXT,

    fingerprint TEXT) 

TABLE os (

    hid INTEGER,

    name TEXT,

    family TEXT,

    generation TEXT,

    type TEXT,

    vendor TEXT,

    accuracy INTEGER)

Next stage will be to write the scripting to create an instance of the DB and the parsing engine to read the NMAP XML output so that the information can be read into the correct fields. 

Establishing Connection to remote mySQL Server

After mucking around getting mysql and MySQLdb libs to compile and install properly with python on OS-Lion I spent a couple of hours mucking around testing connecting to a remote db hosted on GoDaddy.

I came up with this simple test harness to perform a couple of simple tests on the link to make sure that it can be raised and is working correctly.

#!/usr/bin/python

import sys, re

import os, glob, MySQLdb, _mysql

import socket, subprocess

def main():

    

    DB = 'adminsecscan'

    DB_HOST = '50.63.244.10'

    DB_USER = 'adminsecscan'

    DB_PASSWORD = 'Password'

    conn = MySQLdb.Connection(db=DB, host=DB_HOST,port=3306, user=DB_USER,passwd=DB_PASSWORD)

    

    print conn.get_server_info()

    

    myCursor = conn.cursor()

    myCursor.execute("show databases;")

    rows = myCursor.fetchall()

    for row in rows:

        print row

    conn.close

if __name__ == '__main__':

    main()

Next steps will be to develop a schema and parse the XML output and add the scans to the DB, also enumerate the devices connected to the network and store them in a table that can be used to show connection to the network overtime. 

The other idea is to use one of the packet capture libraries like pcapy to sniff the RDP or DHCP traffic and grab to IP and MAC Addresses of new devices as they join the network, so a more comprehensive scan can be performed.

Network Snoop - NMAP scan with python

Upon deployment within a network the first task will be to undertake a scan of the network, as the tool will be targeted towards small networks it will only look at the Class-C range.

First task for the tool will be to find out what the Hosts internal IP address is;

    ipAddr = ([ip for ip in socket.gethostbyname_ex(socket.gethostname())[2] if not ip.startswith("127.")][:1])

The next step is to feed that into an nmap scan, I found the following nmap scan to be good at doing a quick scout around the network:

NMAP -v --open -T4 -oX $NMAPOUT.xml 10.0.0.0/24

The IP address we have needs to be cut down to a class C address and have '/24' appended:

    CuripAddr = ipAddr[0]
    NMAPOutputFile = 'scan-%T-%D.xml'
    
    #Nmap scan of class-C subnet: NMAP -v --open -T4 -oX $NMAPOUT.xml 10.0.0.0/24
    ClassCAddr = CuripAddr.rsplit('.',1)[0] + '.0'
    NmapCommand = 'nmap -v --open -T4 -oX \'scan-%T-%D.xml ' + ClassCAddr + '/24'

Once the command is formed it can be run as a subprocess:

    from subprocess import Popen, PIPE    
    runningNmap = subprocess.Popen(NmapCommand, stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True, bufsize=1)

This creates an XML file which can then be parsed into a DB or used for further scans.

Network Snoop

A program written in Python that performs network analysis/enumeration and reports back to a remote server.

• Perform scan of network using nmap
• Monitor DHCP traffic and enumerate and scan any device connecting to the network
• Ph home and report back using port 80 or 443 to avoid tripping FW
• Deployable on USB thumbdrive 

Found a good post on capture libraries on 'everything about nothing' blog and will need to further investigate into the possibility of using Pcapy & Impacket libraries for python. The device would mainly be used for monitoring networks in small businesses using one central Wifi router to monitor devices connected and usage of bandwidth etc.

Construction needs to be broken down into several steps:

- Run initial Nmap scan of the network using the commandshell version.

- Parse output and save each device as a dictionary entry

- Save dictionary to disk

- Sniff traffic

- Inspect DHCP traffic and enumerate new devices requesting an IP address

- Add new devices to dictionary

- Connect to home server via port 80 or 443 and transmit contents of dictionary file

Running Nmap from Python

First task is to work out how to drive Nmap to scan, looking around it seems to run it as an external process and use subproccess to manage interactions.

Found a good blogpost on running a nightly nmap scan using a cron job etc. Something like this would be good put without the cronjob and shell scripting

to be continued..