This tool has been written to address the need to query MineMeld for a specific IP address to know if it matches a MineMeld list. The agent is also able to handle custom lists of IPs/FQDNs

This tool has been used in conjunction with an Application Firewall with HTTP Callout capabilities. The AppFirewall queries the tool to know if the Source Public IP of a specific call is one of the Microsoft IP in the list provided by Minemeld. This is useful in cases where you cannot filter a NAT/Firewall Rules because it holds different services

• Loader: this loads a urls.json file containing a list of URL to fetch
• Fetcher: this takes URLs from Loader and fetch lists
• Validator: this validate response from various endpoints
• Parser: this parse URL List into a golang object
• Checker: this checks if a given IP/FQDN is in a MineMeld list
• WebServer: this exposes routines through API

GET /api/v1/check-ipv4/{ipv4-address}
> Query used to know if an ipv4 address is in one of the urls.json file lists

GET /api/v1/check-ipv6/{ipv6-address}
> Query used to know if an ipv6 address is in one of the urls.json file lists

GET /api/v1/check-fqdn/{fqdn-address}
> Query used to know if a fqdn address is in one of the urls.json file lists

GET /api/v1/check-ipv4/{ipv4-address/{anchor}
> Query used to know if an ipv4 address is in one of the urls.json file lists with the given anchor

GET /api/v1/check-ipv6/{ipv6-address}/{anchor}
> Query used to know if an ipv6 address is in one of the urls.json file lists with the given anchor

GET /api/v1/check-fqdn/{fqdn-address}/{anchor}
> Query used to know if a fqdn address is in one of the urls.json file lists with the given anchor

GET /api/v1/get-anchor-list
> Query used to get a list of all the anchor defined in the urls.json file

To get the tool you need golang

root@localhost:> go version
go version go1.12.5 darwin/amd64

You can download MineMeld-Agent with the command

root@localhost:> go get github.com/areYouLazy/minemeld-agent

Go inside MineMeld-Agent folder and compile it

root@localhost:> go build

And you're ready to go!

MineMeld-Agent can run on any machine capable of running golang, just make sure URLs in the urls.json file are resolvable (if you're using FQDN) and to use the -fetch-insecure flag if the endpoint does not provide a valid certificate.

By default MineMeld-Agent logs to stdout so you can check that everything is working file. You can than redirect logs to your preferred file with the -log-output flag

Endpoint must be written in the urls.json (or in a custom JSON file) in the format of

[
    {
        "type": "ipv4",
        "endpoint": "https://minemeld.example.org/feeds/office365_IPv4s",
        "description": "MineMeld IPv4 feed for Office365",
        "anchor": "microsoft"
    },
    {
        "type": "ipv6",
        "endpoint": "https://minemeld.example.org/feeds/office365_IPv6s",
        "description": "MineMeld IPv6 feed for Office365",
        "anchor": "microsoft"
    }
]

type and endpoint are required fields

Valid Entpoint types are:

• ipv4
• ipv6
• fqdn

Any other type will throw an error

By design fetch is done for every Endpoint, every 10 seconds

You can add your own lists to the agent. To add a list compile the urls.json file with the endpoint url

{
  "type": "ipv4",
  "endpoint": "http://my.custome.endpoint/ip-list.html",
  "description": "Custom List",
  "anchor": "custom-ipv4-list"
}

To add a network range you can use the following syntax:

• 192.168.1.1-192.168.1.50
• 192.168.1.0/24

To add a single IP you can use the following syntax:

• 192.168.1.1-192.168.1.1
• 192.168.1.1/32

FQDN supports wildcards, also note that FQDN comparsion is done by regulare expression

• example.org
• .example.org
• *.example.org
• minemeld.example.org
• minemeld.example.org/data

Anchor is used to create a label for all the IP/FQDN in a specific list You can query MineMeld-Agent for a specific Anchor

If a list is defined with an anchor like microsoft you can query like this

root@localhost:> curl http://minemeld.example.org/api/v1/check-ipv4/192.168.1.1/microsoft
Address 192.168.1.1 is not in microsoft list

If you query without an anchor, the match is done against any list of that type

root@localhost:> curl http://minemeld.example.org/api/v1/check-ipv4/192.168.1.1
Address 192.168.1.1 is not in list

The GET call to /api/v1/get-anchor-list can be used to obtain a list of all the anchor currently loaded from the urls.json file

root@localhost:> curl http://minemeld.example.org/api/v1/get-anchor-list
["microsoft", "custom-ipv4-list"]

Usage of minemeld-agent:
  -fetch-insecure
        Set to true to ignore certificate errors while fetching MineMeld URLs
  -log-colors
        Set to false to turn off colored log output (default true)
  -log-debug
        Set true to print debug message
  -log-output string
        Set the output interface for log
  -url-file string
        PATH of the JSON file containing urls. (default "urls.json")
  -webserver-port int
        Specify port for WebServer (default 9000)

• IPv6 Address check is not implemented yet