Skip to content

Commit

Permalink
Documentation update.
Browse files Browse the repository at this point in the history
  • Loading branch information
@elsif2
elsif2 committed Nov 13, 2023
1 parent a3a3aee commit 307386d
Showing 1 changed file with 61 additions and 113 deletions.
174 changes: 61 additions & 113 deletions docs/user/bots.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -929,11 +929,6 @@ The resulting reports contain the following special field:

**Parameters (also expects [feed parameters](#feed-parameters) and [cache parameters](#cache-parameters)):**

**`country`**

(required, string) **Deprecated:** The country you want to download the reports for. Will be removed in IntelMQ version
4.0.0, use *reports* instead.

**`apikey`**

(required, string) Your Shadowserver API key.
Expand All @@ -948,7 +943,27 @@ The resulting reports contain the following special field:

**`types`**

(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names given in the section Supported Reports of the [Shadowserver parser](#intelmq.bots.parsers.shadowserver.parser_json).
(optional, string/array of strings) An array of strings (or a list of comma-separated values) with the names of report types you want to process. If you leave this empty, all the available reports will be downloaded and processed (i.e. 'scan', 'drones', 'intel', 'sandbox_connection', 'sinkhole_combined'). The possible report types are equivalent to the file names given in the section Supported Reports of the [Shadowserver parser](#intelmq.bots.parsers.shadowserver.parser).

**Sample configuration**

```yaml

shadowserver-collector:
description: Our bot responsible for getting reports from Shadowserver
enabled: true
group: Collector
module: intelmq.bots.collectors.shadowserver.collector_reports_api
name: Shadowserver_Collector
parameters:
destination_queues:
_default: [shadowserver-parser-queue]
file_format: csv
api_key: "$API_KEY_received_from_the_shadowserver_foundation"
secret: "$SECRET_received_from_the_shadowserver_foundation"
run_mode: continuous

```

---

Expand Down Expand Up @@ -2071,12 +2086,10 @@ No additional parameters.

---

### Shadowserver <div id="intelmq.bots.parsers.shadowserver.parser" /> <div id="intelmq.bots.parsers.shadowserver.parser_json" />
### Shadowserver <div id="intelmq.bots.parsers.shadowserver.parser" />

Parses various reports from Shadowserver.
The Shadowserver parser operates on CSV formatted data.

There are two Shadowserver parsers, one for data in `CSV` format and one for data in `JSON` format. The latter was added
in IntelMQ 2.3 and is meant to be used together with the Shadowserver API collector.

**How this bot works?**

Expand Down Expand Up @@ -2105,8 +2118,7 @@ correct mapping of the columns:

**Module:**

`intelmq.bots.parsers.shadowserver.parser` (for CSV data)
`intelmq.bots.parsers.shadowserver.parser_json` (for JSON data)
`intelmq.bots.parsers.shadowserver.parser`

**Parameters:**

Expand All @@ -2120,108 +2132,44 @@ correct mapping of the columns:

**Supported reports:**

These are the supported report types and their corresponding file name for automatic detection:

| Report Type (`feedname`) | File Name |
|-----------|-----------|
| Accessible-ADB | `scan_adb` |
| Accessible-AFP | `scan_afp` |
| Accessible-AMQP | `scan_amqp` |
| Accessible-ARD | `scan_ard` |
| Accessible-Cisco-Smart-Install | `cisco_smart_install` |
| Accessible-CoAP | `scan_coap` |
| Accessible-CWMP | `scan_cwmp` |
| Accessible-MS-RDPEUDP | `scan_msrdpeudp` |
| Accessible-FTP | `scan_ftp` |
| Accessible-Hadoop | `scan_hadoop` |
| Accessible-HTTP | `scan_http` |
| Accessible-Radmin | `scan_radmin` |
| Accessible-RDP | `scan_rdp` |
| Accessible-Rsync | `scan_rsync` |
| Accessible-SMB | `scan_smb` |
| Accessible-Telnet | `scan_telnet` |
| Accessible-Ubiquiti-Discovery-Service | `scan_ubiquiti` |
| Accessible-VNC | `scan_vnc` |
| Blacklisted-IP (deprecated) | `blacklist` |
| Blocklist | `blocklist` |
| Compromised-Website| `compromised_website` |
| Device-Identification-IPv4 | `device_id` |
| Device-Identification-IPv6 | `device_id6` |
| DNS-Open-Resolvers | `scan_dns` |
| Honeypot-Amplification-DDoS-Events | `event4_honeypot_ddos_amp` |
| Honeypot-Brute-Force-Events | `event4_honeypot_brute_force` |
| Honeypot-Darknet | `event4_honeypot_darknet` |
| Honeypot-HTTP-Scan | `event4_honeypot_http_scan` |
| HTTP-Scanners | `hp_http_scan` |
| ICS-Scanners | `hp_ics_scan` |
| IP-Spoofer-Events | `event4_ip_spoofer` |
| Microsoft-Sinkhole-Events-IPv4 | `event4_microsoft_sinkhole` |
| Microsoft-Sinkhole-Events-HTTP | `event4_microsoft_sinkhole_http` |
| NTP-Monitor | `scan_ntpmonitor` |
| NTP-Version | `scan_ntp` |
| Open-Chargen | `scan_chargen` |
| Open-DB2-Discovery-Service | `scan_db2` |
| Open-Elasticsearch | `scan_elasticsearch` |
| Open-IPMI| `scan_ipmi` |
| Open-IPP | `scan_ipp` |
| Open-LDAP | `scan_ldap` |
| Open-LDAP-TCP | `scan_ldap_tcp` |
| Open-mDNS | `scan_mdns` |
| Open-Memcached | `scan_memcached` |
| Open-MongoDB | `scan_mongodb` |
| Open-MQTT | `scan_mqtt` |
| Open-MSSQL | `scan_mssql` |
| Open-NATPMP | `scan_nat_pmp` |
| Open-NetBIOS-Nameservice | `scan_netbios` |
| Open-Netis | `netis_router` |
| Open-Portmapper | `scan_portmapper` |
| Open-QOTD | `scan_qotd` |
| Open-Redis | `scan_redis` |
| Open-SNMP | `scan_snmp` |
| Open-SSDP | `scan_ssdp` |
| Open-TFTP | `scan_tftp` |
| Open-XDMCP | `scan_xdmcp` |
| Outdated-DNSSEC-Key| `outdated_dnssec_key` |
| Outdated-DNSSEC-Key-IPv6 | `outdated_dnssec_key_v6` |
| Sandbox-URL | `cwsandbox_url` |
| Sinkhole-DNS | `sinkhole_dns` |
| Sinkhole-Events | `event4_sinkhole` |
| Sinkhole-Events IPv4 | `event4_sinkhole` |
| Sinkhole-Events IPv6 | `event6_sinkhole` |
| Sinkhole-HTTP-Events | `event4_sinkhole_http`/`event6_sinkhole_http` |
| Sinkhole-HTTP-Events IPv4 | `event4_sinkhole_http` |
| Sinkhole-HTTP-Events IPv6 | `event6_sinkhole_http` |
| Sinkhole-Events-HTTP-Referer| `event4_sinkhole_http_referer`/`event6_sinkhole_http_referer` |
| Sinkhole-Events-HTTP-Referer IPv4 | `event4_sinkhole_http_referer` |
| Sinkhole-Events-HTTP-Referer IPv6 | `event6_sinkhole_http_referer` |
| Spam-URL | `spam_url` |
| SSL-FREAK-Vulnerable-Servers | `scan_ssl_freak` |
| SSL-POODLE-Vulnerable-Servers | `scan_ssl_poodle`/`scan6_ssl_poodle` |
| Vulnerable-Exchange-Server* | `scan_exchange` |
| Vulnerable-ISAKMP | `scan_isakmp` |
| Vulnerable-HTTP | `scan_http` |
| Vulnerable-SMTP | `scan_smtp_vulnerable` |

\* This report can also contain data on active webshells (column `tag` is `exchange;webshell`), and are therefore not
only vulnerable but also actively infected.

In addition, the following legacy reports are supported:

| Legacy Report Type | Successor Report Type | File Name |
|--------------------|-----------------------|-----------|
| Amplification-DDoS-Victim | Honeypot-Amplification-DDoS-Events | `ddos_amplification` |
| CAIDA-IP-Spoofer | IP-Spoofer-Events | `caida_ip_spoofer` |
| Darknet | Honeypot-Darknet | `darknet` |
| Drone | Sinkhole-Events | `botnet_drone` |
| Drone-Brute-Force | Honeypot-Brute-Force-Events, Sinkhole-HTTP-Events | `drone_brute_force` |
| Microsoft-Sinkhole | Sinkhole-HTTP-Events | `microsoft_sinkhole` |
| Sinkhole-HTTP-Drone | Sinkhole-HTTP-Events | `sinkhole_http_drone` |
| IPv6-Sinkhole-HTTP-Drone | Sinkhole-HTTP-Events | `sinkhole6_http` |

More information on these legacy reports can be found
in [Changes in Sinkhole and Honeypot Report Types and Formats](https://www.shadowserver.org/news/changes-in-sinkhole-and-honeypot-report-types-and-formats/)
.
The report configuration is stored in a `shadowserver-schema.json` file downloaded from https://interchange.shadowserver.org/intelmq/v1/schema.

The parser will attempt to download a schema update on startup when the *auto_update* option is enabled.

Schema downloads can also be scheduled as a cron job for the `intelmq` user:

```bash
02 01 * * * intelmq.bots.parsers.shadowserver.parser --update-schema
```

For air-gapped systems automation will be required to download and copy the file to VAR_STATE_PATH/shadowserver-schema.json.

The parser will automatically reload the configuration when the file changes.


**Schema contract**

Once set in the schema, the `classification.identifier`, `classification.taxonomy`, and `classification.type` fields will remain static for a specific report.

The schema revision history is maintained at https://github.com/The-Shadowserver-Foundation/report_schema/.


**Sample configuration**

```yaml
shadowserver-parser:
bot_id: shadowserver-parser
name: Shadowserver Parser
enabled: true
group: Parser
groupname: parsers
module: intelmq.bots.parsers.shadowserver.parser
parameters:
destination_queues:
_default: [file-output-queue]
auto_update: true
run_mode: continuous
```
---

### Shodan <div id="intelmq.bots.parsers.shodan.parser" />
Expand Down

0 comments on commit 307386d

Please sign in to comment.