https-scan

This tool is a command-line client designed for automated and/or bulk testing of domains with the SSL Labs API and other APIs. The tool is based on Qualys' ssllabs-scan which is available here. The scan results can automatically be saved into SQL-Database, if needed.

The following APIs are included at the moment:

• Qualys SSL Labs API (https://www.ssllabs.com/about/terms.html)
• HTTP Observatory API by mozilla (https://observatory.mozilla.org/terms.html)
• Securityheaders.io

Additionally a crawler was added to check the redirects of a domain.

Requirements

• Tested with go 10.3
• A running MSSQL-database with the tables as specified below

Installtion

Run the commands:

    # Download project
    git clone https://github.com/smarthouse/https-scan
    
    cd https-scan
    
    # Download dependencies
    go get github.com/denisenkom/go-mssqldb
    go get github.com/fatih/structs
    
    # Create Exe
    go build

Copy the file sql_config.json.example to sql_config.json. It must have the same location as the exe. Example content:

{
    "SQLServer": "localhost\\SQLEXPRESS",
    "SQLUserID": "myUser",
    "SQLPassword": "MyPw",
    "SQLDatabase": "myDb",
    "SQLEncryption": "disable"
}

Befor running create the table in this order:

1. backend\tables\Scans.sql

2. backend\tables\Domains.sql

3. backend\tables\Customers.sql

4. backend\tables\Project.sql

5. backend\tables\Domain_Project.sql

6. backend\tables\Unreachable.sql

7. apis\crawler\Crawler.sql

8. apis\observatory\Observatory.sql

9. apis\securityheaders\SecuriyHeaders.sql

10. apis\ssllabs\Certificates.sql

11. apis\ssllabs\SSLLabs.sql

Or just run backend\install.sql.

Usage

SYNOPSIS

    https-scan [options]

Adding Domains from a file to a List

    https-scan -list string -file file -add

Adding Domains from a file to a List

    https-scan -list string -file file -add

Adding Domains from a file to a List auto-overwriting the current List-Value

    https-scan -list string -file file -add -force

Adding a single Domain to a List

    https-scan -list string -domain string -add

Removing Domains from a List

    https-scan -list string -file file -remove

Setting Domains inactive

    https-scan -domain string -file file -inactive

Setting Domains active

    https-scan -domain string -file file -inactive

Starting a scan

    https-scan -scan

Starting a scan with custom configuration

    https-scan -scan -config file

Starting a scan with domains from a file

    https-scan -scan -file file

Starting a scan with domains from a project id

    https-scan -scan -project projectId

GENERAL OPTIONS

Option	Default value	Description
-active	false	Set the given domains to active (only active domains are scanned)
-add	false	Add the given domains to the specified ListID
-continue	false	Continue last scan
-domain		Field to specify a single domain
-file		Field to specify a file containing multiple domains (separated by linebreak)
-force	false	Force overwrite, if there are conflicting adds
-inactive	false	Set the given domains to inactive (only active domains are scanned)
-list		Field to specify the domains belonging to a ListID
-project		Field to specify the domains belonging to a project (project id)
-remove	false	Remove the given domains from the specified ListID
-scan	false	Scan the given domains
-verbosity	info	Configure log verbosity: error, notice, info, debug, or trace
-config	file	File to read API-Options from
-no-crawler	false	Don't use the redirect crawler
-no-obs	false	Don't use the Observatory-Scan
-no-sechead	false	Don't use the SecurityHeaders-Scan
-no-ssllabs	false	Don't use the SSLLabs-Scan

The APIs can be additionally configured via a config-file in the json-Format. Such a file with the default values set, can be found here. The configuration is explained in the README of each API respectively.

All results will be saved in a database. The database as well as the login credentials have to be stored in a file sql_config.json. An empty file can be found here. The sql_user needs read and write access to the used tables.

Also the logs of the last three calls to the function are stored in the logs-folder.

SQL-Database

The sql-database consists of:

• a table containing the scan settings for each scan,
• a table containing all domains and their current status,
• a table to log unreachable domains,
• and one table per scan-api (two in case of the ssllabs-scan).

The meaning of the entries for each table column can be found in the README for each api. A more in depth explanation can be found here.

Structure

After parsing the options and creating an entry in the Scans-table, the https-scanner gets the domains that are in the next scan from the Domains-table. For these domains a connectivity test is done to port 80 (http) and port 443 (https). Domains that are reachable are added to the scan-tables and the rest is stored in the Unreachable-table. Now a thread for each scan-api is created. These threads check the domains to be scanned and start scanning them based on the domain connectivity. The scan-apis handle multiple scan at once by starting a thread for each domain, that is currently scanned. The number of parallel scans is limited. If a scan is finished, the results are returned to the master-thread for the respective api and are saved to the table. In case of an error the api starts the scan of a domain again if the retries number isn't surpassed. The apis send the original thread status reports every 4 seconds. If an api doesn't send a status message in 20 seconds, it is assumed dead and the scan is terminated.

Adding a new API

A short instruction on how to add your own API to the https-scan can be found here.