Support for Mongo 3.6+ DNS seedlists mongodb+srv://

The globalsign/mbo driver doesn't support mongodb+srv:// seedlists. Related - request for srv support in the driver:
globalsign/mgo#112

Would there be an issue with switching from globalsign/mbo to the driver published by MongoDB, which already has support for the new connection string format?

Description of the problem

When the keys from the top command are not of the form <database>.<collection>, the integration fails because it attempts to split on . and take indices 0 and 1. We should handle this more gracefully, either logging and skipping or fililng in the database and collection with dummy values.

src/entities/collect.go

174   for key, collectionStats := range topMetrics.Totals {
175     splitKey := strings.SplitN(key, ".", 2)
176     databaseName := splitKey[0]
177     collectionName := splitKey[1]
178
179     ms := e.NewMetricSet("MongodTopSample",
180       metric.Attribute{Key: "displayName", Value: e.Metadata.Name},
181       metric.Attribute{Key: "entityName", Value: fmt.Sprintf("%s:%s", e.Metadata.Namespace, e.Metadata.Name)},
182       metric.Attribute{Key: "database", Value: databaseName},
183       metric.Attribute{Key: "collection", Value: collectionName},
184     )
185
186     logError(ms.MarshalMetrics(collectionStats), "Marshal metrics on top failed: %v")
187
188   }

OS

• All of them
• Amazon Linux, all versions
• CentOS, version 6 or higher
• Debian, version 7 ("Wheezy") or higher
• Red Hat Enterprise Linux (RHEL), version 6 or higher
• Ubuntu, versions 12.04, 14.04, and 16.04 (LTS versions)
• Windows Server, 2008 and 2012 and their service packs

Description of the problem

Currently, the integration litters the logs with errors when a standalone instance is monitored rather than a cluster. Additionally, documentation doesn't clearly describe the differences in metric collection when monitoring a standalone instance.

OS

• All of them
• Amazon Linux, all versions
• CentOS, version 6 or higher
• Debian, version 7 ("Wheezy") or higher
• Red Hat Enterprise Linux (RHEL), version 6 or higher
• Ubuntu, versions 12.04, 14.04, and 16.04 (LTS versions)
• Windows Server, 2008 and 2012 and their service packs

When monitoring a sharded clusters in K8s the integration fails to discover the mongos endpoint and no MongosSample is generated.

Description

Mongos metrics collection endpoints are discovered in the integration but in K8s the Id on collections.config.mongos contains the Pod name which cannot be resolved by the integration.

Expected Behavior

Support native mongos autodiscovery for K8s environments

Steps to Reproduce

Using bitnami mongo-sharded chart and configured the integration with discovery to the mongos pod

nri-mongodb:
        discovery:
          command:
            exec: /var/db/newrelic-infra/nri-discovery-kubernetes --port 10250 --tls
            match:
              label.app.kubernetes.io/component: mongos
        integrations:
          - name: nri-mongodb
            env:
              HOST: ${discovery.ip}
              ...

The integrations prints the error like:
[ERR] Failed to connect to mongos server <mongos-pod-name>:27017: no reachable servers

Your Environment

Additional context

I workaround this by using creating a service in the Agent namespace that points to the mongos instance and which have the same name of the mongos pod (using StatefulSet kind to deploy mongos to have predictable pod names). But this was just for the propose of validate the issue.

The infra agent now supports:

• Amazon Linux 2022 (Public Preview, GA expected by ~July)
• RHEL 9 (GA)
• Ubuntu 22.04 (GA)

I get following error in logs
Failed to create session: server returned error on SASL authentication step: BSON field 'saslContinue.mechanism' is an unknown field."

I think the problem is with old drivers.

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ readme-starts-with-community-plus-header
  • ✅ readme-contains-link-to-security-policy
  • ✅ readme-contains-forum-topic
  • ✅ code-of-conduct-should-not-exist-here
  • ✅ third-party-notices-file-exists

Passed #

Description

When implementing the MongoDB integration for New Relic, I get an authentication failure when using any credentials using SCRAM. The same credentials works both in the MongoDB service and in the individual pods. I'm able to get all the databases and collections needed based on the role and user to add for New Relic to work properly.

Expected Behavior

The authentication should be working in the integration if it is working both on the service and the individual replicas.

NR Diag

time="2022-03-11T19:11:15Z" level=warning msg="integration exited with error state" component=integrations.runner.Runner env=azure error="exit status 1" integration_name=nri-mongodb stderr="[INFO] Collecting all collections for database 'test'\n[ERR] Failed to create session: server returned error on SASL authentication step: Authentication failed."

Steps to Reproduce

1. Install the MongoDB community operator then a replica set on K8S.
2. Create the NR user along with the necessary roles.
3. Install the nri-bundle helm chart and ensure those values are included:
   - name: mongodb-config.yml data: discovery: command: exec: /var/db/newrelic-infra/nri-discovery-kubernetes --namespaces mongodb --port 10250 --tls match: label.app: mongodb-replica-svc integrations: - name: nri-mongodb env: HOST: ${discovery.ip} PORT: 27017 USERNAME: newrelic PASSWORD: *** AUTH_SOURCE: admin MONGODB_CLUSTER_NAME: abc FILTERS: '{"test":null}' labels: env: dev

This fails, but mongodb://newrelic:***@localhost:27017/admin?authSource=admin&readPreference=primary&appname=MongoDB%20Compass&directConnection=true&ssl=false works without any issues. The password is the same.

Your Environment

I'm using the latest helm chart version of the helm nri-bundle helm chart as well as the mongodb community operator. Kubernetes version is 1.21.

Is your feature request related to a problem? Please describe.

Integration fails due to no mongo 4+ SCRAM-SHA-256 authentication support. I've found that the driver you all use has yet to enable support for SCRAM-SHA-256 authentication as well. There's an existing pr (stale - 2+ years old) in the driver repo you all use. I understand that this will need to be done first in order to enable here, so I'm curious and asking if the mgo feature is completed would you all be willing to support it.

Feature Description

Enable mongo 4+ SCRAM-SHA-256 authentication mechanism

Describe Alternatives

n/a

Additional context

Existing mgo pr seeking support for completion

Priority

Blocker

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

• Fail
  • ❌ readme-starts-with-community-plus-header
  • ❌ code-of-conduct-should-not-exist-here
• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ readme-contains-link-to-security-policy
  • ✅ readme-contains-discuss-topic
  • ✅ third-party-notices-file-exists

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

• README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
  • 🔨 Suggested Fix: prepend [![Community Plus header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Plus.png)](https://opensource.newrelic.com/oss-category/#community-plus) to file

❌ code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Found files. Below is a list of files or patterns that failed:

• CODE_OF_CONDUCT.md
  • 🔨 Suggested Fix: Remove file

Passed #

Create mapping on the metadata backend service infra backend

Is your feature request related to a problem? Please describe.

CPU and Memory Usage for nri-mongodb seems very high due to collectTop / MongoTopSample

Feature Description

Add flag to disable collectTop / MongoTopSample

Additional context

We have added the nri-mongodb integration on our Servers. CPU and Memory Usage for nri-mongodb seems very high and the amount of data being sent to New Relic is also high.

Is there currently a way to disable collectTop that generates MongodTopSample? Or something someone can add easily? Or is this something we should look into adding and submitting a pull request for?

We are already excluding ingest of this data but dont want to keep collecting them on the server side as it seems to use alot of CPU and Mem resources.

We really only need Cluster Health related metrics, the rest we can debug manually on servers. Its nice to have on New Relic side but ingest amount seems extremely high and CPY and Mem usage also seem high.

Priority

Really Want

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ readme-starts-with-community-plus-header
  • ✅ readme-contains-link-to-security-policy
  • ✅ readme-contains-discuss-topic
  • ✅ code-of-conduct-should-not-exist-here
  • ✅ third-party-notices-file-exists

Passed #

Working with a customer and they had to re-configure their Atlas implementation to allow a public endpoint connection. The New Relic MongoDB integration evidently will not work with the DNS seed config where SRV: becomes part of the connection string for a private endpoint.

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

• Fail
  • ❌ readme-starts-with-community-plus-header
• Passed
  • ✅ license-file-exists
  • ✅ readme-file-exists
  • ✅ readme-contains-link-to-security-policy
  • ✅ readme-contains-discuss-topic
  • ✅ code-of-conduct-should-not-exist-here
  • ✅ third-party-notices-file-exists

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

• README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
  • 🔨 Suggested Fix: prepend [![Community Plus header](https://github.com/newrelic/opensource-website/raw/master/src/images/categories/Community_Plus.png)](https://opensource.newrelic.com/oss-category/#community-plus) to file

Passed #

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update golang version to v1.21.5 (go, golang)
• chore(deps): update aquasecurity/trivy-action action to v0.16.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/github-script action to v7
• chore(deps): update actions/setup-go action to v5
• chore(deps): update docker.io/bitnami/mongodb-sharded docker tag to v7
• chore(deps): update docker/login-action action to v3
• chore(deps): update github/codeql-action action to v3
• chore(deps): update newrelic/coreint-automation action to v2
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

When make a new release, generate build for any linux distribution for easy install.

I'm try to integrate MR and MongoDB atlas by using this repo. I have renamed mongodb-atlas-config.yml and added required configurations. But getting below error when try to execute.

[ERR] Invalid arguments: Must supply a cluster name to identify this Mongodb cluster. Use MongodbClusterName config parameter

Could you please share guidance to configure

Thanks
Tharindu

Currently we are using a third party mongodb library , we could move to the official one https://github.com/mongodb/mongo-go-driver

Improve static analysis with Semgrep adding the newrelic-infra-checkers actions to get the semgrep and golangci-lint configurations.

Description of the problem

The MongodTopSample is missing from documentation.

OS

• All of them
• Amazon Linux, all versions
• CentOS, version 6 or higher
• Debian, version 7 ("Wheezy") or higher
• Red Hat Enterprise Linux (RHEL), version 6 or higher
• Ubuntu, versions 12.04, 14.04, and 16.04 (LTS versions)
• Windows Server, 2008 and 2012 and their service packs

Description

The default dashboard for MongoDB in NROne has several widgets querying against MongoSample, which results in empty/blank charts.

Steps to Reproduce

Navigate to Infrastructure >> Third-party services >> MongoDB Dashboard in NROne

Our Documentation says: MongoDB Atlas is supported for tiers M10 and above.

However, it has been reported that in general we do not have enough documentation for that.

We should test the whole solution again, verify any limitation and improve the documentation.

Update documentation to point that mem.residentInBytes and mem.virtual are in MiB not Bytes.

Is your feature request related to a problem? Please describe.

On multi-instance cluster the integration needs to point to one instance and metrics from all the instances are collected. This is a good feature but if the configured instance is down there will be no metrics collected.

Feature Description

Add a flag like MasterOnly that if true will only collect metrics if the current instance is the Master instance. With this flag, multiple instance of the integration can be configured to point all instances of the cluster.
Similar approach as nri-elasticsearch.

Describe Alternatives

A clear and concise description of any alternative solutions or features you've considered. Are there examples you could link us to?

Additional context

Add any other context here.

Priority

Please help us better understand this feature request by choosing a priority from the following options:
[Nice to Have, Really Want, Must Have, Blocker]