nuwcdivnpt / stigman-watcher Goto Github PK

Current behavior:
preflight request failure - leaves a log entry, does not retry on failure, watcher stops.
subsequent token failure has no effect on other processing (but will show lots of errors in the log)

subsequent token acquisition failure modes:

• Identity provider unavailable (timeout):
  • throw error, stop processing, retry
• client credentials no longer working
  • throw error, stop processing, retry

retry behavior:

• scan mode: suspend scanning and file parsing while retrying
• event mode: continue capturing file add/remove events, suspend file parsing
• retry every 60 seconds
  • Log initial failure
  • Log every (hour? day? thereafter)
  • Log success
  • log: every attempt at debug level
  • continue trying indefinitely

Add additional log statements, perhaps at the "silly" level, for logging each attempted request, whether a response was received, etc.

Several places are overly optimistic and could benefit from more logging or tweaks to error handling. (try/catch block starting at line 74 in authenticateSignedJwt [auth.js] assumes a response object)

This line in cargo.js expects this.parsedResults to be an Array. However, if the cargo size is 1, then this.parsedResults is an Object and the attempt to iterate it fails.

The latest fast-xml-parser (currently 4.0.6) supports the parsing of XML comments, which will be useful for later enhancements.

Current behavior:
preflight auth failure - leaves a log entry, does not retry on failure, watcher stops.
subsequent auth failures - keeps processing, adds log entries

subsequent auth failure modes:

• user does not have required grant for collection

  • throw error, stop processing, retry

• user can't create assets or assign stigs, when Watcher is configured to do so.

  • throw error, stop processing, retry

• user can't approve reviews, when watcher is configured to do so.

  • note: double-check API behavior when given status that does not conform with Collection grants/settings
  • log error, continue processing, mark file complete?

retry behavior:

• scan mode: suspend scanning and file parsing while retrying
• event mode: continue capturing file add/remove events, suspend file parsing
• retry every 60 seconds
  • Log initial failure
  • Log every (hour? day? thereafter)
  • Log success
  • log: every attempt at debug level
  • continue trying indefinitely

Hello. We've got STIG Man Watcher working and importing CKL files placed in the watched directory, however, XCCDF imports do not work. I've increased the log file logging level to 'silly' and don't even see the XCCDF file being discovered by Watcher. Any thoughts?

History file could/should have any entries that are no longer present in the scanned directory removed.

Include:
Comprehensive typedefs for parameters and return objects
Description of process flow
expected error responses

Most of the jsdoc notation needed was moved to stig-manager-client-modules

Include test cases for:

• all import setting combinations
• files with invalid xml
• files with 0 reviews
• files with multiple STIGs
• files with duplicate Review Ids
• "normal" files (expected # of reviews, etc.)

Breakout Watcher auth functions into their own re-usable libraries, which could become part of a resource repo providing a scaffold for building other clients.

• Remove keycloak-specific elements
• Add support for Device Code flow.

Possible someone on the STIGMan team may add this info to the op/configuration endpoint.

STIG Manager instance behind nginx rev proxy with CAC enabled. DoD certificate issued to system correctly. STIGMan Watcher errors with "..."level":"error","component":"api","message":"unable to verify the first certificate","request":{"method":"GET","url":"https://..."

Watcher should pay attention to the new Review Field settings added to the API

Build a workflow to build sign then upload binaries.

Current behavior:
preflight request failure - leaves a log entry, does not retry on failure, watcher stops.
Subsequent failures - keeps trying, adds log entries.

API unavailable after preflight (timeout):

• throw error, stop processing, retry

retry behavior:

• scan mode: suspend scanning and file parsing while retrying
• event mode: continue capturing file add/remove events, suspend file parsing
• retry every 60 seconds
  • Log initial failure
  • Log every (hour? day? thereafter)
  • Log success
  • log: every attempt at debug level
  • continue trying indefinitely

STIGMan API will soon be able to return Collection Import setting preferences that are specified at the Collection level.
Watcher should pay attention to these settings, and post reviews in accordance with them, in order to minimize rejected reviews.

Watcher will also need access to the /user endpoint (and thus the stig-manager:user:read scope), to determine if it is allowed to Accept reviews.

This is complicated by the cargo queue implementation, which means different components of a given .ckl file may be chunked into different cargos.

Take into consideration #61

Latest versions of got only available as ESM, rather than CommonJS modules.

Consider:
converting Watcher project as a whole to ESM, rather than CommonJS
or
Switch to build-in Node fetch

Issue is the same as NUWCDIVNPT/stig-manager#880.

This should include:

Hello,
Been using stigman-watcher for almost a year now and we still love it. Kudos to everyone involed. When I first set it up I forgot and left logging on and by defualt debug mode was enabled so I happen to notice a 7GB on our file server today..lol So I wanted to reach out and see if there could be a way to incorporate a log rotation feature in the future releases. I am working on possibly injesting our logs into Graylog and alerting on specific systems when the STIGs are pulled into STIG Manager. This way our assigned teams will know when to log in and review them.

Hello,
I'm trying to deploy STIGMan watcher in our enviornment but it seems I am stuck with using only the command line options. When I try to run the stigman-watcher-win.exe I get the following: error: required option '--api ' not specified

 I have specified all of the required variables here:

 And also I edited the .env sample provided with the source code within the same directory. I had to name the file env.env since Windows does not allow me to rename a file with just an extension.
 I would personally love to just deploy this on Linux where the STIG Manager runs natively but we are not allowed SAMBA in our environment so the Linux servers cannot talk to the file share. 
 
 What I want to do is just setup the stigman-watcher-win.exe as a service to monitor the folders. I can run the command with all the cli options and get it to connect and successully pull in the .ckl. However, I have to use the --prompt command and provide the key each time since it is not actually looking for the variables. Im not sure if i am doing something wrong or if STIGMan Watcher just doesn't support environmental variables on Windows. Thanks!!

The utility is unreliable in Windows when watching more than 64 directories. The 64 directory limit is hard coded into Windows and according to MSDN can only be overcome by multi-threaded processes. Node.js is single-threaded and cannot do this without add-ons we do not wish to require.

Recommend adding an alternative algorithm that scans trees instead of watching them.

Clients should request the OAS definition from API, and constrain text sizes in accordance with the specified max string length specified for Review Detail, Comment, and metadata fields.

Requires:

• NUWCDIVNPT/stig-manager#681

The Wiki and README mention Watcher needing a Collection grant of "Manage", but there is no description on how to accomplish that. The REQUIREMENTS portion in the Wiki should have a "STIG Manager" section that explains this.

Hi there!

I have been using STIGMAN watcher for a few months now. I have noticed one interesting behavior.

Sometimes watcher will reupload a file to STIGMAN, even though the file has not been changed, modified, or updated at all. Is there a reason this could be happening?

See picture below:

Any help would be appreciated.

Thanks!

There is currently an environment variable to tell watcher to ignore specified directories (nice!).

Evaluate-STIG recently added a new file in the top level folder, named SummaryReport.xml. This file is being parsed and produces the following message:

{"timestamp":"2022-05-26T13:29:39.559Z","level":"warn","component":"parser","message":"No Benchmark element","file":"<my-output-path>/<hostname>/SummaryReport.xml"}

Additionally, this file is logged in the history.txt file which is unnecessary.

Requesting a new option to specify file names to ignore in the watched directory.

Thanks!

Possible avenue:
Develop a github actions workflow that can verify Watcher functions with latest version of the STIGMan API.
Create sets of .ckls (and/or xccdf) that, on import, will pass an appropriate subset of the Newman tests from the STIGMan repo.

When running in events mode, the following error stops the process:

{"timestamp":"2022-07-20T13:02:34.506Z","level":"error","component":"main","message":"ignored is not defined","error":{"name":"ReferenceError","message":"ignored is not defined","stack":"ReferenceError: ignored is not defined\n    at Object.module.exports.startFsEventWatcher (/home/csmig/dev/stigman-watcher/lib/events.js:11:25)\n    at run (/home/csmig/dev/stigman-watcher/index.js:28:15)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)"}}

We have several instances of watcher used by different groups posting results to a shared collection. It would be handy to be able to apply a label to all assets posted by a specific watcher; This would help sorting a large collection by using labels to identify the originating department/division/etc.

Using STIG Manager repo as a model, add these files:
CONTRIBUTORS.md
CONTRIBUTING.md
INTENT.md

refactor:
LICENSE.md
to reference company name instead of individual contributors (which should be added to CONTRIBUTORS.md)

I believe I have my .env file configured correctly but I am seeing the subject message when running the standalone executable in linux. I didn't see any option to ignore the certificate. My first presumption is that uploads aren't happening because of this error. Can you confirm this behavior? We are using our own CA, not Dod signed certs.

publish to npm

Right now Watcher presumes /protocol/openid-connect/token instead of referring to /.well-known/openid-configuration for the token endpoint.

Likely needed for non-keycloak OIDC providers.