Acceptance Criteria

• Start to define the technical work for Epic 2 of O-Packs
• break out tickets for epic 2
• identify any open questions

Justin confirmed Authors is required in the main config.yml as of right now that set of rules have already been schema stitched.. We should add a check to ensure contributors know to add an author and if they don't we communicate that in the PR workflow.

Acceptance Criteria

• add a check that an author is provided

Details

The o11y pack repo needs to be setup to facilitate contributions from internal relics for Reli-Hack so we'll want to put in some basic github actions and ensure documentation is clear to facilitate contributions.

Acceptance Criteria

• determine with @kidk what basics checks may be required.
• provide feedback on the documentation and developer exp to @kidk.
• look for basic automation opportunities in the process and implement those checks.
• look for advanced automation opportunities int the process and make a new ticket to implement those checks in a future sprint.
• Define JSON schema for dashboards
• Define JSON schema for alerts
• Define JSON schema for synthetics
• Define overall JSON schema for o11y packs
• Add PR template
• Update contributing guide and include guidance on conventional commits
• Add check for conventional commits on push
• Add check for schema compliance on push

Acceptance Criteria:

• add .prettierrc.js & ignore file

Description

We will need a new repository to store our code for the Observability Pack details nerdlet (and the installed pack nerdlet). This work should be done in Github Enterprise, but we will track the efforts in this repo (in public Github).

The repository can be set up using a nos template. Refer to the one-core docs (hint: /guides/development/#1-project-setup---creating-a-nerdpack).

Acceptance Criteria

• Initialize a new repository in Github Enterprise (use main as the mainline branch) (handled by template)
• Ensure that the Developer Enablement engineers have access
• Enable branch protection
• Set up an empty Nerdpack (handled by template)
• Connect to internal deployment platform (handled by template)

Acceptance Criteria:

• Dashboards and alerts are imported with a single command via CLI
• We use the following command: npm run import --accountId [accountId] --apiToken [token] [pathToPack]

Code Stream offers some cool features, how could we use it to better manage all of our opensource repos?

https://www.codestream.com/

Acceptance Criteria

• JV reviews this and reports back cool findings
• Follow up with codestream team for a demo
• The team plays around with it to see if it offers value
• We decide if we want to use it.

Summary

The JSON schema used to validate observability packs only checks that a few fields exist. That schema needs to be updated to support all fields in the template and enforce required fields.

Helpful links

• Pack template
• Pack schemas
• JSON schema
  • FYI we are using draft-07
• Justin is compiling a doc on pack fields, here

Acceptance Criteria

• Fill out the main_config schema (for the top level config.yaml)
• Fill out the alert_config schema
• Ensure documentation lines up with the schemas
• Exclude the UUID and Slug for now as these are still being discussed
• schema should match the file found here

The schema doc is using terminology that is different from what in a pack config.yml we should align these terms.

Issues

1. UUID is mentioned in the schema doc but not in the config.yml
2. config.yml template has thelevel set to New Relic, but we should set the example to Community as this is how all packs will start until approved to New Relic
   3.slugis mentioned in the schema doc but not in the config.yml
3. in the config.yml description visibility is spelled wrong
4. title is mentioned in the schema doc but not in the config.yml
5. short-description is mentioned in the schema doc but not in the config.yml
6. full-description is mentioned in the schema doc but not in the config.yml
7. tags are mentioned in the schema doc but not in the config.yml
8. screenshots are mentioned in the schema doc but not in the config.yml but we seem to use files that are placed in /dashboards within a pack?
9. thumbnails are mentioned in the schema doc but not in the config.yml
10. children are mentioned in the schema doc but not in the config.yml
11. contributors are mentioned in the schema doc but not in the config.yml

Summary

We had some great contributions coming out of Reli Hack, but the contributions need to be cleaned up or finished before we can merge them in.

Acceptance Criteria

• Add description that describes the value of the pack
• Add screenshots, icon, logo for dashboards (add files to repo, don't reference external files)
• Ensure the description also describes the steps needed for the instrumentation (if it's more complex than just infra agent + ohi, for example using OpenTelemetry)
• Fix typos
• Update authors:

authors: 
- New Relic
- <Your Name>

• Have a newline at the end of your json or config files

We'll need a GH project that will organize all the PRs and issues for Olly Packs in a single view for contributors to interact with.

Acceptance Criteria

• Follow a similar pattern as the boards for [3rd Party apps]https://github.com/orgs/newrelic/projects/44) and DTK projects
• Setup a project on github under new relic org
• Name the project NR1 Quick Start Community

We have some packs that have the permissions field in the JSON export, which doesn't work for the o11y pack service (permissions are hardcoded to PUBLIC_READ_WRITE when the service creates a dashboard).

Desired remedy:

• Fix these packs: Golden Signals for Web Servers, LAMP stack, PHP
• Add some schema validation in our PRs to avoid this issue in the future

Programming language: NodeJS + Typescript

Use template project https://github.com/actions/typescript-action

Related issue: #8

let’s create a Getting Started guide based on the new format and repository as part of our sprint. The goal is to share this with Reli Hack participants by Tue May 18th, so would be good to prioritize this early in the sprint, and a good exercise to get familiar with the format.

Location of Guide: https://github.com/newrelic-experimental/futurehack-2021
Format: Readme file

Acceptance Criteria

• Work with @alexronquillo on the creation of this guide to explain the following:
• How to create an observability pack
• How to add a dashboard and alert
• How to use the newrelic cli to install a pack (@svandamme is working on this)
• How to contribute to the repo
• Ideally whatever we document can be easily turned into a guide on the developer site in the future.

Can we make it so --dashboardPack is not needed, but you just give it the path as argument: npm run import --accountId 123 --key my-super-secret-key [name of pack]

Before
npm run import --accountId 123 --key my-super-secret-key --dashboardPack [name of pack]
After
npm run import --accountId 123 --key my-super-secret-key [name of pack]

Summary

Observability pack slugs are unique. We need a job that verifies this at PR time.

Acceptance Criteria

• Add a job that checks if a pack's slug name is unique
• check should ignore case.
• check should ignore punctuation.
• Add a job that checks if a pack's UUID is unique
• Failure output is human readable and points the contributor to a fix
• Workflow documentation is updated

In order to make it easier on contributors and maintainers lets set some limits on the API for the pack meta data. These limits will ensure the process to add meta data such as descriptions, icons, logos, etc is simple and will allow us to put in some checks to prevent abuse or mistakes

Name: required

• 100 characters
• Should be a unique identifier for a pack
• Will be used in the URL such as: developer.newrelic.com/observability-packs/[name-of-pack]/

Description: optional

• 2000 characters
• Can we support Markdown as well as plain text?

Summary: auto-generated

relates to JIRA UIWRITE-1210

• 500 characters
• Auto generated based on this template:

For a Landing page:

“This {Name} observability pack is an extension to New Relic. This pack contains {x} dashboards and {y} alerts and allows you to easily setup monitoring and observability within New Relic.

For CTA Components & Google Snippets:

“Use the {Name} observability pack, as an extension to New Relic. This pack contains {x} dashboards and {y} alerts and allows you to easily setup monitoring and observability within New Relic.

Level: optional

support level

• New Relic
• Verified
• Community
• Enterprise
• Local

Icon: optional

• .png or .jpeg or .svg format
• Max 1
• Recommended size: 48px (width) x 48px (height)

Logo: optional

• .png or .jpeg or .svg format
• Max 1
• Recommended size: Aspect ratio: 1:1
• Recommended size: 250px (width) x 100px (height)

Website: optional

• Valid url only (https://www.newrelic.com)

Authors: required

• No limit to authors

Images: optional

• Max 6
• .png or jpeg or svg format
• Recommended size
• Aspect ratio: 3:2
• 800 px (width)
• 1600 px (height)

Decisions to make

1. we won't be auto-scaling images initially, so there is still a question to discuss if we should try to enforce some level of image size and scale. It’s inconvenient for the end user to worry about sizing images, but if we don’t enforce something that worry is passed onto the maintainers.
2. It's not clear to me if the enforcement of these limits will be at the API level, or are we relying on Github actions to check for these things.
3. How do we auto generate the summary text? Where is that done?

Details

Review Draft Spec of O-Pack API and provide feedback as it relates to the needs of UI design

Once we have all the repo automation checks in place let's document this thoroughly in the contributing guide.

Acceptance Criteria

• update the contributing guide to encourage contributions and explain pack limits.

Requirements:

• Dashboard are created automatically in the account with public permissions

npm run import --accountId [accountId] --apiToken [token] [pathToPack]

API to use: https://api.newrelic.com/graphiql & https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph/

Example GraphQL query for importing dashboards:

mutation($accountId: Int!, $dashboard: DashboardInput!) {
        dashboardCreate(accountId: $accountId, dashboard: $dashboardJson) {
            errors {
                description
                type
            }
            entityResult {
                guid
            }
        }
    }

In case of nonexistent package or an error returned from NR API it should be handled

We have several open questions in the Observability Packs Epic: Public Library Doc that we need to confirm this sprint.

There a lot of non-engineering checks) to be made on a pack, like support level, or the pack content and meta data is correct, we need a process to define who is responsible to do this. By setting up a review team upfront, we have multiple teams with skin in the game which should help avoid forgetting about duties.

Working doc

G-Drive O11y Pack Contributor Review Process

Acceptance Criteria

• establish a 'review team' and ask the Technical Architect team to join, (for example Zack Mutchler, to be confirmed) to help us validate the business value of new contributions. That team has lots of experience with many different technologies and use cases

• Given we will default all packs to community. During the PR review the addition of Level can be made to the pack config and then it can be validated accordingly. This puts some responsibility on an active maintainer community defined in this process.

• Could we automate this process of assigning 1 engineer as reviewer who looks at the technical aspects, and 1 field person who looks at business value? OR could we create a slack channel for this, assign a hero each week and post a message from new PRs in that channel to be picked up by the hero. Is that over-engineering it and would we rather just look at a board?

• Define the SLA on contribution responde time frame on cross functional review process. Bill asked for a 24 hr first responder message. So let's look at create a bot that can respond to PRs & Isssue in a timeline manner and also drop a comment on PR that is sitting idle. See Gatsby for ideas. :D

• obtain review from our team

We need to make it easy as possible for users to use the import tool. To avoid hem having to install node, npm and other tools we need to pack the application in a way users can easily run it. Most New Relic employees should have Docker installed, so let's package the code as a container, and provide a bash script to use it.

1. Create a docker image.
2. Write a guide for user to help them create a config file that contains all the credentials we need.

For example: import-config.sh (make sure it's gitignored)

#!/usr/bin/env bash

export O11y_ACCOUNT_ID = ".."
export O11y_API_TOKEN = ".."

3. Create a bash script import.sh that loads in the import-config.sh file, and takes 1 argument which is the pack name. The bash script should then run the docker container with all the right parameters.

In the end the user should just have to run import.sh [pack name] to import and test out a pack in their environment.

Acceptance Criteria:

• NR API types for alerts are implemented in the tool
• NR API types for policy creation are implemented in the tool

Summary

The support levels, "New Relic", "Verified", and "Community" will determine how a pack is reviewed and merged. We will be using Github actions and labels to denote which PRs need maintainer attention.

Flow chart of the PR process

AC:

• PRs containing a new o11y pack are automatically labeled with new-pack
• PRs containing code changes to a pack with the "New Relic" support level are labeled with support-level:newrelic
• PRs containing code changes to a pack with the "Verified" support level are labeled with support-level:verified
• PRs containing code changes to a pack with the "Community" support level or no support level are labeled with support-level:community
• Workflow documentation is updated
• This workflow should conditionally run after validation has occurred. if validation fails, this doesn't run.

Prompt the user "We found ${number} of alerts/dashboards with the same name. Would you like to delete them? Y(es)/N(no)" when duplicate items are found.

Summary

We do not want to require node and yarn as development dependencies, move the package.json and yarn.lock files under the utils/ directory. We will also need to replace the Git hook that verifies that commit messages are conventional with a bash or python version.

Helpful links

• Conventional commit regex
• Commit-msg hook output example

AC:

• Move node/yarn files under the utils/ directory
• Update the validate packs workflow to account for the new package.json location
• Replace husky and the conventional commit hook with shell

Change to 'Name of the pack you want to import'

Overview

Below are a list of questions we'll want to answer before sprint 4 ends.

• Do we need to worry about keeping things in private for QuickStarts? We'll be working in a public repo, for the Developer Site, and previously with the Docs site we did everything in the open. Is there a reason we should consider making a private repo of the developer site and doing this work secretively?

no. we can work in the open.

• Ensure we've finalized our pack structure approach
• Ensure we've finalized our product naming conventions
• I'd like to answer if this a "big splash" project, where we must hold everything back until we are done, or if we can deliver incremental features.

we'll target an MVP release towards the end of our development cycle, and if we can release some things a little early we'll try to do that

• Define: Basic Users What can they do with Quickstarts?
• Define: Basic Users What can’t they do with Quickstarts?
• Confirm support approach, GTS vs Community
• When are designs going to be finalized and can we begin to breakdown Epic 2 in detail?

Usability testing with users on the wireframes that we already have (mpastor is conducting these beginning this week).
We are beginning addressing the use cases that Bill highlighted (users creating new dashboards and new alerts). @kidk as a great lead on how we can handle the alerts use case, and we've been meeting with ppascual (the designer for dashboards) this week on the dashboards front. We’re working on the users flows and wireframes and will update you sam and stijn as soon as we have some artifact ready to show.
Once we have results back from our usability tests, Miriam will analyze them and provide a report on the results which we can act on. We’ll go through and improve on our wireframes and if we’ve landed at a satisfactory solution then we can get to work on the high-fidelity mocks that (in addition to conversations between myself and the engineers) will act as a tool to communicate implementation details.

• Confirm what is the shape of the data we'll be dealing with in design?
• Schedule time with Samuel and team to finalize the repo format & API design, we need an API spec to work from
• Is there an architectural review required for this project? - Who would that be?

miguel and vince, meeting scheduled

• How much bandwidth does @danielgolden have to assist us with design throughout the lifecycle of this project?

~40% of his time each week

• How much will the DevEx team do in Epic 4, for the in-product catalog?
• How we will leverage Entity Synthesis in this flow?

JF is meeting with GPE PMs to discuss soon

• Is entity synthesis solving the same problem as o11y packs? If you can visualize data from entity synthesis already, what is the purpose of the o11y packs at this point?

JF is meeting with GPE PMs to discuss soon

Summary

Add a job to the validate-packs workflow that parses the top level config.yml and checks to make sure that files referenced in the icon and logo fields exist.

AC:

• Workflow validate-packs fails when the icon field does not reference an existing file
• Workflow validate-packs fails when the logo field does not reference an existing file
• Ensure the output clearly directs the contributor to the problem
• Workflow documentation is updated

Acceptance Criteria

• Our docs team will review the pack meta data and provide improvements in the form of PRs.
• all summaries have been added.
• PRs will be reviewed and code review / merged
• pack meta data will be improved

Summary

We need to verify that there aren't any bad words getting into the repository. To do this we will be adding job to the validate-packs workflow to parse the packs for any bad language.

Acceptance Criteria

• There is a new job that checks the packs for bad words and fails if any are found
• Name this job offensive-language
• Failure output is informative enough that contributors can easily find and fix any problematic language
• Workflow documentation is updated
• Check all files in the pack directory for bad words. (for now)

References

https://github.com/newrelic-experimental/futurehack-2021

Issue

to carve out time for the DevEn Team to participate and support the Future Hack Relic event on May 25th.

Acceptance Criteria

• The DevEn team can participate in a future hack contribution for the Relic track.
• One idea would to be build out an O11y pack and contribute to this repo
• Another idea is to work on a problem that interest you around the 3rd Party App Catalog, Gatsby Sites or Demotron.
• Or maybe you have your own idea? 🚀
• The entire team will support any issues from the field or from internal relics during this day.

Description

We want to ensure that, unless someone has the feature flag enabled, users will not be able to see the details for a Observability Pack.

The Developer Experience team might be setting this up.

Acceptance Criteria

• Create a staging feature flag for viewing Observability Pack details
• Create a staging feature flag for installing an Observability Pack (displaying the button)
• Create a production feature flag for viewing Observability Pack details
• Create a production feature flag for installing an Observability Pack (displaying the button)
• Create an EU production feature flag for viewing Observability Pack details
• Create an EU production feature flag for installing an Observability Pack (displaying the button)
• Add the Developer Enablement engineers to all feature flags

Requirements:

• Alerts are created automatically in a new Alert Policy with name "[pack name] policy"
  • 1st create the policy
  • 2nd create the alerts and link them to the policy

npm run import --accountId [accountId] --apiToken [token] [pathToPack]

API to use: https://api.newrelic.com/graphiql & https://docs.newrelic.com/docs/apis/nerdgraph/get-started/introduction-new-relic-nerdgraph/

Mutations to use for alerts:

• alertsNrqlConditionBaselineCreate
• alertsNrqlConditionOutlierCreate
• alertsNrqlConditionStaticCreate
• alertsPolicyCreate

Acceptance Criteria:

• User is able to import packs as following -> https://github.com/newrelic/quickstarts-dashboard-library/tree/main/library/browser

Dev Notes:

• We should avoid recreated graphQL instance with every call

Justin confirmed Level is required in the main config.yml as of right now that set of rules have already been schema stitched.. We should add a check to ensure contributors know to add a level and if they don't we communicate that in the PR workflow.

Acceptance Criteria

• add a check that a level is provided
• I believe all packs are defaulting to community, so we can just ensure that during the PR review process, no one has removed this field and left if blank. OR maybe we decided not to that? 🤔

Summary

To ensure that all screenshots will be visible and load in a reasonable amount of time on all the catalogs we need to restrict image type, size, and amount.

AC:

• A workflow job should check that screenshots are limited to JPEG and PNG filetypes
• A workflow job should check that all images are 4MB or less
• A workflow job should check that there are no more than 6 screenshots per pack component
• Workflow documentation is updated

Acceptance Criteria:

• Every alert file is imported and placed in an array
• The array is parsed and filled with missing data

The NGINX icon is currently in webp format which is failing the image validation

Summary

It would be really helpful to be able to reference variables / secrets for an entity / account when configuring components of an observability packs. A great example would be the URL for a browser entity (for synthetics scripts).

OR

Have a variable substitution when creating/installing a quickstart? For example if I wanted to add a WHERE appId = <value> to some of my queries, is there a mechanism for the user to specify <value> during installation.

Without this feature, we had to close #39

Other variable uses cases

1 We've identified several use cases where being able to add a variable to a quickstart to allow that to be set by a user on installation would be really nice.

One example is the when using the netlify quickstart

• 1. Supply just an app name and a New Relic user key
• 2. Supply the required ids/keys (the current method)

2 Another is variables required in synthetic checks

3 And there are likely many others. like setting 3rd party API keys, env variables, or license keys for access to services that a quickstart could integrate with.

Desired Behavior

Some mechanism for referencing variables that are set at an entity (or account) level. I believe there is already support for something like this in the platform, but I'm not sure on the level of effort needed to implement it.

Possible Solution

We could perhaps use variables with some sort of interpolation syntax like ${{some_var}}

name: Failed Signups
type: BASELINE
nrql:
  query: "SELECT count(*) FROM JavaScriptError WHERE pageUrl = "${{env.signup_page_url}}"

Acceptance Criteria

• Iterate over alerts' array to sent via GraphQL to NR API
• Alerts are created in NR1
• Alerts are created automatically in a new Alert Policy with name "[pack name] policy"
  -- 1st create the policy
  -- 2nd create the alerts and link them to the policy

Dev Notes:
Mutations to use for alerts:

• alertsNrqlConditionBaselineCreate
• alertsNrqlConditionOutlierCreate
• alertsNrqlConditionStaticCreate
• alertsPolicyCreate

For a pull request, our validation workflow should check for the existence of logo and icon images if they're present in the config.yml.

Example:

# config.yml

# if specified (as these are optional fields) these should exist in the incoming pack for the PR to be considered valid
icon: myIcon.{png|jpg|jpeg|gif|svg}
logo: myLogo.{png|jpg|jpeg|gif|svg}

Summary

The output generated by validate_packs.js script is not very informative when an enum does not match one of the allowed values.

The AJV library that we use to validate packs against these schemas provides the needed information when an error occurs, we are just outputting the message field.

AC:

• Error output from the validate-packs.js script should include the related field and the allowable values in the output message
• if any field fails validation, provide useful feedback.
• add unit tests
• Workflow documentation is updated contributing md file

Summary

Observability packs need a UUID, but we don't want to ask contributors to handle creating their own. Let's create a job that parses the main config file, checks for a UUID, and adds one if there isn't one present.

Acceptance Criteria

• The uuid field is added to Nerdgraph (can we use iD?) > Virtuoso
• The uuid is added to packs automatically when a PR is opened
• The uuid is added the schema
• The uuid field is not changed for packs that already have one
• Workflow documentation has been updated
• All existing Packs should have a uuid added to their config.

Justin confirmed Description field is required in the main config.yml as of right now that set of rules have already been schema stitched.. We should add a check to ensure contributors know to add a description and if they don't we communicate that in the PR workflow.

Acceptance Criteria

• Add a check that ensure description is present and is <= 2000 characters and is plain text

Example: https://github.com/newrelic/open-install-library/tree/main/docs

Copy the open-install-library model

Validator: https://github.com/newrelic/open-install-library/tree/main/validator