generate_esgfconf.sh is a script file from the esgf_scanner repo which is used to generate as output, a configuration file for use with the CVEChecker tool.
The idea is to be able to auto-generate a manifest for each release, and use that an input to scan for known vulnerabilities. When a reported vulnerability is studied and deemed to be addressed, it can then be muted, to prevent repeated notifications for the same issue.

• Clone this repo with the --recursive flag, as it checks out the CVEChecker repository as a submodule.

• cog_packages, pub_packages, jarlist, solr_webappjars, solr_serverjars, esgf_manual, esgf_excludes
  • cog_packages and pub_packages are the output of a pip freeze, obtained from the CoG python, and the esgfpub environment.
  • jarlist is the output of a find command, looking for jar files in the tomcat webapps directory (/usr/local/tomcat/webapps).
  • solr_webappjars is the output of a find command, looking for jar files in the solr webapps directory (/usr/local/solr/server/solr-webapps/WEB-INF/lib).
  • solr_serverjars is the output of a find command, looking for jar files in the solr server directory (/usr/local/solr/server/lib').
  • lasjars is the output of a find command, looking for jar files in the las installation directory (/usr/local/las-esgf//las-esgf-/WebContent/WEB-INF/lib).
  • esgf_manual and esgf_excludes contains packages which are manually specified, in addition to what is automatically added from the jar files and python package lists.

• For testing, simply execute firstuse.sh, which creates sample input files from the template files.
• The template files are the actual manifests for latest available production release of ESGF (2.7.1)
• Modify the input files as needed, to test. If you wish to add packages to be checked, add them to esgf_manual, and if you wish to add keywords, add them to esgf_keywords.
• When ready, run generate_esgfconf.sh, to generate the esgf.conf file, which can then be used as the configuration file with CVEChecker.

• A detailed README for CVEChecker exists; you can see it here, and it's recommended that you read it completely, before proceeding.

• Start by executing bash firstuse.sh; this sets up an empty vulnerability store, and an empty checksum list.
• Initialize CVEChecker by executing python3 cvechecker.py -u.
• Copy the exportedmutes file from the esgf_scanner repository: cp ../exportedmutes ..
• Import the muted entry file: python3 cvechecker.py -i exportedmutes. Repeat this operation regularly, with the latest exportedmutes file from the esgf_scanner repo.
• Copy the esgf.conf file generated by the generate_esgfconf.sh script: cp ../esgf.conf ..
• Generate a report: python3 cvechecker.py -r esgf.conf >esgf.report.
• Use the matchstats.sh script to print out statistics about the matches.

pchengi@thebeast:~/cvechecker$ python3 cvechecker.py -r esgf.conf >esgf.report
pchengi@thebeast:~/cvechecker$ bash matchstats.sh esgf.report 
Total number of matching CVEs: 1240
annotations:1
bleach:1
cdm:1
cryptography:1
cxf-core:1
dom4j:1
findbugs:1
future:1
guava:1
junit:1
log4j:1
pyOpenSSL:1
requests:1
slf4j-api:1
solr:1
standard:1
tqdm:1
urllib3:1
xalan:1
commons-fileupload:2
hibernate-validator:2
html5lib:2
jdk:2
json:2
mysql-connector-java:2
org.restlet:2
httpclient:3
numpy:3
opensaml:3
xmltooling:3
zookeeper:3
xmlsec:4
django:5
forms:5
jackson-databind:5
struts:14
Apache Commons:31
Django:43
spring:48
postgresql:67
tomcat:90
java:880

• If you have a very large number of results, you can select an individual package, and generate a report for that, inspect it and whitelist entries as necessary.
• When you generate a report for an individual package (as opposed to from the configuration file), the numbers may be different as there might be some results which are excluded due to the packages being present in the esgf_excludes file.

• Product and keyword matches are case-sensitive.
• Keyword lookups match words in CVE descriptions whereas product lookups look at the actual affected products, as listed by the CVEs.
• Product lookups use 'startswith' and not an exact match, so a lookup for 'postgresql' returns hits for postgresql-<some version>, postgresql-libs-<some version> etc.
  • This has the undesired side-effect, particularly in the case of very short product names, such as ant, which would trigger matches for antivirus, antipathy, and so on.
  • In such cases, instead of doing a product lookup on ant, a keyword search for 'Apache Ant' can be done, and it would result in highly relevant matches.
• Keyword lookups are both case-sensitive and 'space' sensitive. A keyword match for Intel would pick up a description that says "Vulnerability found in Intel chips..." but not "Vulnerability found in the IntelliSense package of ...'.
• Excludes are done on exact matches, and not 'startswith', to prevent false-negatives. Note though that a single CVE can list multiple packages, and if you've asked to exclude rh-postgresql-7.3.9-2, you'll not see the alert for the same CVE which also lists postgresql-7.4.7-2.RHEL4.1 as an affected package. Use the --exclude option with care.

• While an instance of CVEChecker can be setup work with automatic muting, to issue new alerts if any come up, for the regular security scanning work, we need to use manual whitelisting, and let open issues remain thus, so we know they are yet to be addressed.
• The -m option controls muting: -m on turns muting on and -m off turns muting off. Muting can be done on a single CVE, or to a whole set of CVEs that match a defined set of parameters. Use with caution.
• For example, python3 cvechecker.py -p kernel -m on mutes all known CVEs related to the package 'kernel'.
• You can display muted CVEs by using the -d option, with or without other search parameters.
• You can use the -l flag with -m on to log comments about why you are muting the CVE. This information is exported when you use the -e option, to export the list of CVEs you have muted.

pchengi@thebeast:~/cvechecker$ python3 cvechecker.py -c CVE-2017-7546 -m on -l
Product name?
postgresql
Reason for muting?
Issue fixed in postgresql-8.4.20-8.el6_9 released in October 2017
pchengi@thebeast:~/cvechecker$ python3 cvechecker.py -e exportedmutes

pchengi@thebeast:~/cvechecker$ cat exportedmutes 
CVE-2017-7546|postgresql|2018-10-02 09:28|Issue fixed in postgresql-8.4.20-8.el6_9 released in October 2017

• You can use the exported file containing the muting information to import it onto a fresh system, to get all the CVEs muted instantly.
• While importing muting information from a file, the muting timestamp is inspected; if a CVE has been modified since the last time it was muted, it won't be muted while importing it.

• If you've generated a report for a certain product or combination of search parameters, you can use the whitelist-helper, to quickly select CVEs for whitelisting (muting). To launch the helper, simply use the same search parameters as you used for the report generation, but include the -w flag.
• The whitelist-helper prompts your response for every listed CVE for the selected search-parameters; you can go through the report in a different terminal while you run the whitelist-helper.
• The default response is Y, which selects the CVE for subsequent muting.
• You can press Ctrl-C at any point, and you won't lose responses made till that point.
• The cves selected by you for whitelisting will be written out to whitelist_out.
• The whitelist-helper only generates a list of CVEs, and doesn't actually mute/whitelist anything. The output file can be used as input for a subsequent manual muting operation.

pchengi@thebeast:~/cvechecker$ python3 cvechecker.py -p struts -w
Whitelist entry CVE-2011-1772?(Y/n)
Whitelist entry CVE-2011-2087?(Y/n)
Whitelist entry CVE-2011-2088?(Y/n)n
Whitelist entry CVE-2013-1965?(Y/n)n
Whitelist entry CVE-2013-1966?(Y/n)^Cbye
pchengi@thebeast:~/cvechecker$ cat whitelist_out 
CVE-2011-1772,CVE-2011-2087

pchengi@thebeast:~/cvechecker$ python3 cvechecker.py --cve --file whitelist_out -m on -l
Product name?
Apache Struts
Reason for muting?
CVEs against older versions of Struts.
pchengi@thebeast:~/cvechecker$ python3 cvechecker.py -e exportedmutes 

pchengi@thebeast:~/cvechecker$ cat exportedmutes 
CVE-2011-1772|Apache Struts|2018-10-02 09:46|CVEs against older versions of Struts.
CVE-2011-2087|Apache Struts|2018-10-02 09:46|CVEs against older versions of Struts.
CVE-2017-7546|postgresql|2018-10-02 09:28|Issue fixed in postgresql-8.4.20-8.el6_9 released in October 2017