Table of Contents
- Table of Contents
- Metrics Accelerator
Metrics Accelerator
The metrics accelerator (formerly metrics framework) is a Mule application intended to collect, aggregate and load platform metrics into different visualization systems; providing out of the box integrations and visualization options, including useful dashboards and charts. In addition to the platform metrics, the accelerator also extends the capabilities to integrate with external applications like Jira, Confluence, Jenkins, Bitbucket and Splunk to gather SDLC metrics. This is an UNLICENSED software, please review the considerations. If you need assistance for extending this, contact MuleSoft Professional Services
Features
- Compact Mule application (1 single application)
- Provides more than 100 metrics from 3 complementary domains:
- Platform Operational Metrics: collected and calculated automatically based on multiple products from Anypoint Platform: Exchange, Design Center, Runtime Manager, Access Management; covering metrics from applications deployed on-prem (Standalone), RTF and CloudHub.
- Platform Benefits: require manual input to calculate final metrics, crossing information from the "Platform Operational" domain
- External SDLC Metrics: collected and calculated automatically based on multiple external applications: Jira, Confluence, Jenkins, Bitbucket and Splunk;
Modes
-
Poller (Push mode)
- Collects, transforms and loads metrics in a defined visualization system
- Configurable - frequency (cron expression and timezone ) and status (enabled/disabled).
-
API to manage the asset:
- API endpoint to obtain metrics on-demand (Pull mode)
- API endpoint for triggering a specific loader to push data to a visualization system
- API endpoint for loading platform benefits - manual input required
Loader options
- CSV
- JSON
- Plain Log: in case you forward logs to external systems (e.g using Splunk forwarder)
- Splunk: Including an out of the box dashboard with more than 100 charts
- ELK: Including out of the box, basic, Kibana dashboards
- Anypoint Monitoring: Requires Titanium subscription, dashboard is not provided
- Embedded dashboard: Including an out of the box basic embedded dashboard accessed by running the application offering an UI with a number of metrics obtained
- Tableau: (Not available yet)
Available Metrics
Platform Metrics
Product | Metric | Dimensions | - |
---|---|---|---|
Access Management | Users Total | BG | |
Access Management | Active Users | BG | |
Access Management | Inactive Users | BG | |
Access Management | Active Users Last 60 days | BG | |
Access Management | Active Users Last 30 days | BG | |
Access Management | Environments Total | BG | |
Access Management | Environments Production Total | BG | |
Access Management | Environments Sandbox Total | BG | |
Access Management | API Client Applications Total | BG | |
Design Center | Assets Total | BG | |
Design Center | API Specs Total | BG | |
Design Center | Fragments Total | BG | |
Design Center | Flow Designer Apps Total | BG | |
Exchange | Assets Total | BG | |
Exchange | API Specs Total | BG | |
Exchange | Mule 3 Connectors Total | BG | |
Exchange | SOAP APIs Total | BG | |
Exchange | Fragments Total | BG | |
Exchange | HTTP Proxies Total | BG | |
Exchange | Policies Total | BG | |
Exchange | Extensions Total | BG | |
Exchange | Custom Assets Total | BG | |
Exchange | Overall Satisfaction | BG | |
Exchange | API Fragments reuse | BG | |
Exchange | API Spec reuse | BG | |
Exchange | Extensions reuse | BG | |
Exchange | API Specs managed in API Manager reuse | BG, Environment | |
Exchange | Policies applied in API Manager reuse | BG, Environment | |
API Manager | API Specs Managed Total | BG, Environment | |
API Manager | API Instances Total | BG, Environment | |
API Manager | API Instances Active Total | BG, Environment | |
API Manager | API Instances Inactive Total | BG, Environment | |
API Manager | API Instances Versions Total | BG, Environment | |
API Manager | API Instances With Policies Total | BG, Environment | |
API Manager | API Instances Without Policies Total | BG, Environment | |
API Manager | API Instances With Security Policies Total | BG, Environment | |
API Manager | API Instances Without Security Policies Total | BG, Environment | |
API Manager | API Instances With Contracts Total | BG, Environment | |
API Manager | API Instances Without Contracts Total | BG, Environment | |
API Manager | API Instances With More Than One Consumer Total | BG, Environment | |
API Manager | API Instances With One or More Consumers Total | BG, Environment | |
API Manager | API Contracts Total | BG, Environment | |
API Manager | Policies Used | BG, Environment | |
API Manager | Policies Used Total | BG, Environment | |
API Manager | Automated Policies Used | BG, Environment | |
API Manager | Automated Policies Used Total | BG, Environment | |
API Analytics | Transactions Last 30 days Total | BG, Environment | |
RuntimeManager - CloudHub - Networking | VPCs Total | BG | |
RuntimeManager - CloudHub - Networking | VPCs Available Total | BG | |
RuntimeManager - CloudHub - Networking | VPCs Used Total | BG | |
RuntimeManager - CloudHub - Networking | VPNs Total | BG | |
RuntimeManager - CloudHub - Networking | VPNs Available Total | BG | |
RuntimeManager - CloudHub - Networking | VPNs Used Total | BG | |
RuntimeManager - CloudHub - Networking | DLBs Total | BG | |
RuntimeManager - CloudHub - Networking | DLBs Available Total | BG | |
RuntimeManager - CloudHub - Networking | DLBs Used Total | BG | |
RuntimeManager - CloudHub - Networking | Static IPs Total | BG | |
RuntimeManager - CloudHub - Networking | Static IPs Available Total | BG | |
RuntimeManager - CloudHub - Networking | Static IPs Used Total | BG | |
RuntimeManager - CloudHub - Applications | vCores Total | BG, Environment | |
RuntimeManager - CloudHub - Applications | vCores Available Total | BG, Environment | |
RuntimeManager - CloudHub - Applications | vCores Used Total | BG, Environment | |
RuntimeManager - CloudHub - Applications | Applications Total | BG, Environment | |
RuntimeManager - CloudHub - Applications | Applications Started Total | BG, Environment | |
RuntimeManager - CloudHub - Applications | Applications Stopped Total | BG, Environment | |
RuntimeManager - CloudHub - Applications | Runtime Versions Used | BG, Environment | |
RuntimeManager - CloudHub - Applications | Runtime Versions Used Total | BG, Environment | |
RuntimeManager - RTF - Capacity | Fabrics Total | BG | |
RuntimeManager - RTF - Capacity | Workers Total | BG | |
RuntimeManager - RTF - Capacity | Controllers Total | BG | |
RuntimeManager - RTF - Capacity | Cores Allocated Total | BG | |
RuntimeManager - RTF - Capacity | Memory Allocated Total | BG | |
RuntimeManager - RTF - Capacity | Cores Allocated Per Fabric Average | BG | |
RuntimeManager - RTF - Capacity | Memory Allocated Per Fabric Average | BG | |
RuntimeManager - RTF - Applications | Cores Allocated Total | BG, Environment | |
RuntimeManager - RTF - Applications | Memory Allocated Total | BG, Environment | |
RuntimeManager - RTF - Applications | Applications Total | BG, Environment | |
RuntimeManager - RTF - Applications | Applications Started Total | BG, Environment | |
RuntimeManager - RTF - Applications | Applications Stopped Total | BG, Environment | |
RuntimeManager - RTF - Applications | Runtime Versions Used | BG, Environment | |
RuntimeManager - RTF - Applications | Runtime Versions Used Total | BG, Environment | |
RuntimeManager - Standalone | Mule Servers Total | BG, Environment | |
RuntimeManager - Standalone | Mule Clusters Total | BG, Environment | |
RuntimeManager - Standalone | Mule Server Groups Total | BG, Environment | |
RuntimeManager - Standalone | Mule Applications Total | BG, Environment | |
RuntimeManager - Standalone | Mule Applications Started Total | BG, Environment | |
RuntimeManager - Standalone | Mule Applications Stopped Total | BG, Environment | |
RuntimeManager - Standalone | Mule Runtime Versions | BG, Environment | |
RuntimeManager - Standalone | Mule Runtime Versions Total | BG, Environment | |
MQ | Queues total | BG, Environment, Region | |
MQ | FIFO Queues total | BG, Environment, Region | |
MQ | Queues In Flight Messages total | BG, Environment, Region | |
MQ | Queues Received Messages total | BG, Environment, Region | |
MQ | Queues Sent Messages total | BG, Environment, Region | |
MQ | Queues ACK Messages total | BG, Environment, Region | |
MQ | Exchanges total | BG, Environment, Region | |
MQ | Exchanges Published Messages total | BG, Environment, Region | |
MQ | Exchanges Delivered Messages total | BG, Environment, Region |
Platform Benefits
These metrics require manual inputs
Name | Dimensions |
---|---|
Developer Productivity | BG |
Platform Benefits | BG |
Savings From API Reuse | BG |
Savings From Maintenance Productivity | BG |
Savings From Reuse in Maintenance | BG |
Total Savings | BG |
SDLC Metrics
These metrics are optional and can be cherry picked as per your requirement
Name | Metric |
---|---|
BitBucket | Total Number of BitBucket Repositories |
Confluence | Total Number of Confluence pages |
Confluence | Total Number of Confluence pages created in the last 30 days |
Confluence | Total Number of Confluence pages updated in the last 30 days |
Confluence | Top Contributors in the last 30 days and associated number of pages created |
Jenkins | Total Number of Jenkins jobs |
Jenkins | Total Number of failed Jenkins jobs |
Jenkins | Total Number of successful Jenkins jobs |
Jenkins | Total Number of unexecuted Jenkins jobs |
Jira | Total Number of Jira stories in the backlog |
Jira | Total Number of Jira stories in the current sprint |
Jira | Jira stories categorized by type and associated count in the current sprint |
Jira | Jira stories categorized by status and associated count in the current sprint |
Splunk | Total Number of Splunk dashboards |
Installation
Requirements
- Mule Runtime 4.2.1 or above
- All deployments models are supported: CloudHub, OnPrem hosted Runtimes, Runtime Fabric
- Anypoint Platform credentials, that can be:
- Anypoint Platform user with the Organization Administrator role in the master organization and all Sub Orgs you want to collect data
- A Connected App (client credentials) with the following scopes (make sure to include all Sub Orgs and all environments you want to collect data):
- Design Center
- Design Center Developer
- Exchange
- Exchange Viewer
- Runtime Manager
- Cloudhub Network Viewer
- Read Alerts
- Read Applications
- Read Servers
- Runtime Fabric
- Manage Runtime Fabrics
- API Manager
- View APIs Configuration
- View Contracts
- View Policies
- General
- Profile
- View Environment
- View Organization
- Design Center
- (Optional for SDLC metrics) Authorized user with API access to any of the applications: Jira, Confluence, Jenkins, Bitbucket and Splunk for which you want to gather data.
Steps
-
Clone or download the project from GitHub
git clone [email protected]:mulesoft-catalyst/metrics-accelerator.git
-
Adjust the properties, run the project and test it - go to your browser and open
http://localhost:8081/console/
-
Use the postman collection provided (/postman) to understand the API. The postman collection contains the following requests:
-
Platform Metrics:
- GET Platform Metrics: retrieves plaform metrics
- POST Platform Metrics - Load - Splunk Strategy: used to load platform metrics to Splunk. For more information, see Splunk steps
- POST Platform Metrics - Load - CSV Strategy: returns platform metrics in CSV format.
- POST Platform Metrics - Load - JSON Strategy: returns business metrics in JSON format.
-
Business Metrics:
- GET Benefits: retrieves business metrics showing the benefits of using the platform
- POST Benefits - Load - Splunk Strategy: used to load business metrics to Splunk. For more information, see Splunk steps
- POST Benefits - Load - JSON Strategy: returns business metrics in JSON format.
- If you want to run the application using the poller mode, you have to configure some properties
Properties configurations
- Default configurations defined in
/src/main/resources/properties/app-{env}.yaml
: - Make sure to encrypt all sensitive data using the Secure Properties Module: https://docs.mulesoft.com/mule-runtime/4.2/secure-configuration-properties.
- Default secure config file defined in
/src/main/resources/properties/secure/app-{env}.yaml
- Example
mule.key
used and configured as a Global Property underglobal.xml
file
Name | Description | Default Value |
---|---|---|
http.port | The port for exposing the metrics-accelerator API | 8081 |
poller.enabled | Property to enable or disable the poller to collect and load metrics in external systems | false |
poller.frequency.cron | Defines the exact frequency (using cron-expressions) to trigger the execution: Recommended to collect metrics once a day | 0 0 0 * * ? * |
poller.frequency.timezone | Defines the time zone in which the cron-expression will be efective | GMT-3 |
aggregation.raw | Flag to define the format of the final response False: Won’t provide the raw data but final metrics True: Will provide raw data to be aggregated outside this asset | false |
collectors | Comma separated set of collectors that should be executed. Default value: all. Possible values available for all deployment models: core (Core Services) ap (Automated Policies) apc (API Clients) apm (API Manager) arm (Standalone Runtimes) dc (Design Center) ex (Exchange). The following collectors are not available for PCE: amq (Anypoint MQ) apma (API Manager Analytics) ch (Cloudhub) rtf (Runtime Fabric) | all |
loader.strategy | In the case of using the poller, this property defines the strategy for loading data in external systems, the options are: csv, json, logger, splunk, am, elk, tableau | logger |
anypoint.platform.host | Anypoint Platform Host. Change to eu1.anypoint.mulesoft.com if using the EU Control Plane or to a private host if using PCE | anypoint.mulesoft.com |
auth.mode | Authentication mode. Valid options are: platform-credentials or connected-app-credentials | platform-credentials |
auth.username | Anypoint Platform username. Used when auth.mode is platform-credentials | |
auth.password | Anypoint Platform password. Used when auth.mode is platform-credentials | |
auth.clientId | Anypoint Platform Connected App Client Id. Used when auth.mode is connected-app-credentials | |
auth.clientSecret | Anypoint Platform Connected App Client Secret. Used when auth.mode is connected-app-credentials | |
auth.orgId | Anypoint Platform master org Id | |
ignoreLists.organizations | An array (comma-separated values) of Anypoint Platform sub-organizations id that will be ignored while retrieving metrics e.g "cdfa4e7d-47cd-n1h1-8f39-6a73fbb9ffcb, cdfa4e7d-47cd-n2h2-8f39-6a73fbb9ffcb" |
SDLC metrics collectors properties configuration
Name | Description | Default Value |
---|---|---|
sdlc.confluence.enabled | Property to enable or disable application to collect metrics from Confluence | false |
sdlc.confluence.host | Confluence server host | |
sdlc.confluence.port | Confluence server port | |
sdlc.confluence.path | Context url of the Confluence REST API | |
sdlc.confluence.user | Authorized Confluence user to access REST APIs | |
sdlc.confluence.token | User token to access REST APIs | |
sdlc.bitbucket.enabled | Property to enable or disable application to collect metrics from Bitbucket | false |
sdlc.bitbucket.host | Bitbucket server host | |
sdlc.bitbucket.port | Bitbucket server port | |
sdlc.bitbucket.path | Context url of the Bitbucket REST API | |
sdlc.bitbucket.user | Authorized Bitbucket user to access REST APIs | |
sdlc.bitbucket.token | User token to access REST APIs | |
sdlc.jira.enabled | Property to enable or disable application to collect metrics from Jira | false |
sdlc.jira.host | Jira server host | |
sdlc.jira.port | Jira server port | |
sdlc.jira.path | Context url of the Jira REST API | |
sdlc.jira.backlogPath | Context url of the Jira REST API to fetch stories from backlog | |
sdlc.jira.user | Authorized Jira user to access REST APIs | |
sdlc.jira.token | User token to access REST APIs | |
sdlc.jenkins.enabled | Property to enable or disable application to collect metrics from Jenkins | false |
sdlc.jenkins.host | Jenkins server host | |
sdlc.jenkins.port | Jenkins server port | |
sdlc.jenkins.path | Context url of the Jenkins REST API | |
sdlc.jenkins.user | Authorized Jenkins user to access REST APIs | |
sdlc.jenkins.token | User token to access REST APIs | |
sdlc.splunk.enabled | Property to enable or disable application to collect metrics from Splunk | false |
sdlc.splunk.host | Splunk server host | |
sdlc.splunk.port | Splunk server port | |
sdlc.splunk.path | Context url of the Splunk REST API | |
sdlc.splunk.user | Authorized Splunk user to access REST APIs | |
sdlc.splunk.password | Password to access REST APIs |
Splunk steps
- Create 2 indexes: metrics and platform_benefits (of type Events)
- In the Splunk instance configure an HTTP Event Collector (HEC) associated to these 2 indexes, format _json
- The token obtained will be used as part of the properties of the Mule application
- Create a new application
- Load the dashboards, simply copy the xmls provided under
/dashboards/splunk
to{SPLUNK_HOME}/etc/apps/{APP_NAME}/local/data/ui/views
- Restart the Splunk instance
- If you can't copy the dashboard xmls, you can use the UI to create them and using the "Source" option, you can copy & paste the content of the xmls provided
Follow official Splunk documentation: https://docs.splunk.com/Documentation/Splunk/
Properties specific for Splunk
Name | Description | Default Value |
---|---|---|
splunk.host | HTTP Event Collector (HEC) host | |
splunk.port | HEC port | 8088 |
splunk.protocol | HEC endpoint protocol: HTTPS or HTTP | HTTP |
splunk.token | HEC token | |
splunk.source | HEC source | metrics-source |
splunk.source.type | Source Type | _json |
splunk.index.metrics | Index for storing Platform operational metrics | metrics |
splunk.index.benefits | Index for storing Platform benefits | platform_benefits |
ELK steps
NOTE: Dashboards were created and tested with Kibana 7.6.2, adjustments may be necessary for other versions
- The accelerator will load data into the
metrics
andplatformbenefits
indexes. Once data is loaded create an index pattern on Kibana for these indexes - Set the loader strategy to
elk
on theapp-{env}.yaml
file, along with theelk.user
andelk.password
parameters in the secureapp-{env}.yaml
file - To load the dashboards, replace the
<YOUR-INDEX-PATTERN-ID>
occurrences on all of the dashboards provided under/dashboards/elk
with your index pattern ID. The index pattern ID can be obtained on Kibana underManagement>>Index Patterns
- Log into your Kibana instance, and on the
Management>>Saved Objects
menu, click on import for each dashboard. This will import all dashboards and visualizations
Follow official ELK documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html https://www.elastic.co/guide/en/kibana/current/index.html
Properties specific for ELK
Name | Description | Default Value |
---|---|---|
elk.host | Elasticsearch host | |
elk.port | Elasticsearch port | 9200 |
elk.protocol | Elasticsearch protocol: HTTP | HTTP |
elk.user | Elasticsearch username | |
elk.password | Elasticsearch password | |
elk.index.metrics | Index for storing Platform operational metrics | metrics |
elk.index.benefits | Index for storing Platform benefits | platformbenefits |
Embedded Dashboard steps
- Enable the dashboard by changing the embedded.dashboard.enabled property to "true"
- Deploy & Run the application
- Use a web browser to access the applications base URL (e.g. if deployed locally, use http://localhost:8081)
- Use the "Login" page to enter your Anypoint platform username, password and organization ID
- Wait for the dashboard to run the metrics request and once done, navigate through the different metrics taken using the UI
Considerations
- This application can be deployed in any Mule Runtime (OnPrem, CloudHub, RTF)
- The metrics collection will depend on the features available in each account; e.g if the account has the API Manager add-on, the accelerator will collect and aggregate the metrics related to API Manager, otherwise the values will appear as zeroes; if using PCE, there won't be information about API Analytics
- In order to enable or disable specific collectors, you have to change the property "collectors" if using the poller or add a query parameter "collectors" if using the API, including a CSV string as explained in the properties section
Limitations
- Access Management metrics:
- Not supported on GovCloud
- Exchange reuse metrics:
- Not supported for Private Cloud Edition (PCE)
- API Manager metrics:
- API Manager metrics available only for accounts with the API Manager and Analytics add-on
- Runtime Manager (CloudHub) application metrics:
- CloudHub is not supported on Private Cloud Edition (PCE)
- Runtime Manager (CloudHub) networking metrics - VPCs, VPNs, DLBs and static IPs usage:
- Not supported when authenticating with Connected Apps
- Not supported on GovCloud
- Runtime Manager (RuntimeFabric) metrics:
- Runtime Fabric is not supported on GovCloud
- Runtime Fabric is not supported on Private Cloud Edition (PCE)
- Runtime Manager (Standalone) metrics:
- Runtime Manager (Standalone Runtimes) not supported on GovCloud
- API Platform Client Applications metrics:
- Not supported when authenticating with Connected Apps
- Not supported on GovCloud
- Analytics metrics:
- Not supported on GovCloud
- Not supported on Private Cloud Edition (PCE)
- Not supported when authenticating with Connected Apps
- Anypoint MQ metrics:
- Not supported on Private Cloud Edition (PCE)
- Not supported when authenticating with Connected Apps
- Not supported on GovCloud
Some Theory around the Accelerator
The accelerator is intended to cover the main areas to define and implement metrics using Mule.
Business Needs
There are several business needs around the definition of metrics. The principal goal is to provide visibility on 4 main areas: People/Teams, Projects, Processes and Technology. By having visibility on these areas, the involved stakeholders can make decisions, anticipate technical and non-technical needs and optimize time and resources
Data Sources
Any system containing raw data
Measurements
- Basic measurements:
Examples:
- Number of projects
- Number of incidents
- Complex/Composed measurements
Examples:
- Average daily-TPS per application/project
- vCores usage per application/project - monthly
- Indexes/KPIs
Examples:
- Growth
- Tendencies
- Efficiency / Performance
Problem Space
How to link business needs, measurements and data sources?
Approach - Conceptual
-
Event: Canonical object representing a metric, based on dimensions and facts
-
Collectors: Processes to Collect raw data from different data sources
-
Aggregators/Transformers: Transform (calculate, compose, enrich, build metric/event normalized/denormalized objects
-
Loaders: Processes to send/load events into external storages/systems
-
Visualization: Third party systems for showing charts and dashboards based on metrics-events (e.g Microstrategy, Tableau, Splunk dashboards, ELK, custom dashboards, BI software, etc)
Approach
Conceptual
Detailed
Final Notes
Enjoy and provide feedback / contribute :)