Enable Kibana Diagnostic to be Space Aware for Rules about support-diagnostics HOT 13 OPEN

I say "ideally", but honestly not sure - there may be some reason it would be preferable to get the data from Kibana. I think the ES versions will likely have "more stuff" than the Kibana versions, which seems like a good thing. @mikecote ?

In general, the diagnostic has avoided reading from data indexes -- even system indexes -- to avoid scraping user data accidentally. Having an API in Kibana would be ideal for the diagnostic because then the diagnostic would not need to be changed for any unanticipated changes to the data structure (or indexes for that matter), and Kibana can be responsible for removing content we shouldn't have.

from support-diagnostics.

I don't think we had an issue open for this already, so opened a new one, which should get linked back to this issue: elastic/kibana#137352

from support-diagnostics.

It feels like PR #580 moved us a little closer, now we have a list of the spaces. So, now it feels like we should use that list, and iterate through each space, collect the rules/connectors like we currently do. We'll have to figure out some convention to name the files, maybe an index of the space in the spaces list, so would be like kibana_alerts_1_1.json for the first set of connectors in the first space listed in kibana_spaces.json?

from support-diagnostics.

Heh, was just looking at this yesterday.

Ideally, we could get the alerting connectors / rules directly from ES, and not Kibana. But I think the problem with that approach is the diagnostic tool either points to ES when getting ES diags, or Kibana when getting Kibana diags, so when running the Kibana diags, the ES url is not available.

I say "ideally", but honestly not sure - there may be some reason it would be preferable to get the data from Kibana. I think the ES versions will likely have "more stuff" than the Kibana versions, which seems like a good thing. @mikecote ?

If we want to continue down the path of just using the Kibana URL, we'll need to get all the spaces, then for each space, do what's currently being done with the default space.

At some point, connectors / rules will be shareable across spaces, so I guess if we continue down the path of using the Kibana APIs, we'd end up having dups of rules/connectors that are shared. Not a huge problem, but one we won't have if we query the rules right from ES.

from support-diagnostics.

From @pickypg's update, I believe we're confirming going with @pmuellr update

If we want to continue down the path of just using the Kibana URL, we'll need to get all the spaces, then for each space, do what's currently being done with the default space.

As mini-step-forward, I've filed #580 to pull Kibana Spaces which'll at least allow introspection that non-default-space data's missing. I'm guessing some variant of @CamiloSierraH's #514 PR, we'll want to grab spaces and then iterate pulling per space:

• Find Saved Objects: Index Pattern
• Actions/Connectors per current
• Rules per current

from support-diagnostics.

I say "ideally", but honestly not sure - there may be some reason it would be preferable to get the data from Kibana. I think the ES versions will likely have "more stuff" than the Kibana versions, which seems like a good thing. @mikecote ?

If we want to continue down the path of just using the Kibana URL, we'll need to get all the spaces, then for each space, do what's currently being done with the default space.

@pmuellr

The data returned from Elasticsearch directly would contain encrypted attributes like API keys for alerting rules and configuration / passwords for connectors (if we export connectors). Going through Kibana will remove those encrypted attributes when using the APIs.

In Kibana, there is such a way when not using authorization (by using the saved object repository) to get a list of saved-objects across all the spaces. This is how we handle the alerting health API => https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/health/get_health.ts#L37

from support-diagnostics.

The data returned from Elasticsearch directly would contain encrypted attributes like API keys for alerting rules and configuration / passwords for connectors (if we export connectors). Going through Kibana will remove those encrypted attributes when using the APIs.

But they are encrypted :-).

Ya, probably best to make sure these aren't exported, so if we did use ES, we'd have to filter those away.

In Kibana, there is such a way when not using authorization (by using the saved object repository) to get a list of saved-objects across all the spaces. This is how we handle the alerting health API => https://github.com/elastic/kibana/blob/main/x-pack/plugins/alerting/server/health/get_health.ts#L37

The diagnostic tool uses HTTP endpoints to get it's data, so I don't think this will help. I'm guessing someday we'll have a role of "alerting admin" or such, that would be kinda super-user for alerting things only, and then we could make some endpoints available only for those roles or something. Guessing that will be a while though.

from support-diagnostics.

The diagnostic tool uses HTTP endpoints to get it's data, so I don't think this will help. I'm guessing someday we'll have a role of "alerting admin" or such, that would be kinda super-user for alerting things only, and then we could make some endpoints available only for those roles or something. Guessing that will be a while though.

I think that path makes sense as a starting point, even if starts with diagnostics purposes. We could make the API internal, protected behind a certain privilege that would give space agnostic information.

from support-diagnostics.

@mikecote @pmuellr will you link the Kibana-side Github to create an internal API for space-agnostic data pulls so I can follow? 🙏🏼

from support-diagnostics.

👋🏼 @mikecote @pmuellr may we review this again? Cross-posted elastic/kibana#137352 is now closed Jan3. And user-facing elastic/kibana#140449 is also complete but not (currently/yet/?) in Kibana diagnostics.

The current workaround, is to have the user supplement diagnostic by pulling

> GET .kibana*/_search?q=alert.enabled:true&size=2000 > rules.json
$ cat rules.json | jq '.hits.total.value'
# check if need to re-pull with higher-up-to-10000 size
$ cat rules.json | jq '.hits.hits[]|._source.alert.executionStatus.status as $s|select($s!="ok" and $s!="active")'
# shows erring Rules

It may be possible for Alerting & DetectionEngine statuses to disagree (e.g. https://github.com/elastic/sdh-security-team/issues/584) so for List Detection Rules, may also consider polling the following

> GET .kibana*/_search?q=type:"siem-detection-engine-rule-execution-info"&size=2000

from support-diagnostics.

cc @XavierM for the request to list all the rules across all the spaces in the diagnostic bundle.

from support-diagnostics.

I suppose looking to elastic/kibana#140449
a request against:
/internal/alerting/_global_execution_logs?date_start=<start>&date_end=<end>&per_page=<num>&page=<num>&namespaces=[<list from spaces>]
should fulfill the request taken here:
#578 (comment)

However given the API has prefix /internal I suppose this one has not been tested to be valid against Infosec issues like exposing credentials to the error messages?

from support-diagnostics.

However given the API has prefix /internal ...

I don't think that's an appropriate API to use for this - at least I'm not aware of anyone asking for execution data like this.

It should be possible to use public APIs to iterate through all the Kibana spaces, and then do the same processing we're doing for the default space, in those spaces.

from support-diagnostics.