Giter VIP home page Giter VIP logo

mssql-monitoring's Introduction

What is this project?

This is a project to get users started on monitoring Microsoft SQL Server on Linux using collectd, InfluxDB and Grafana. Please see How the SQLCAT Customer Lab is Monitoring SQL on Linux for more details on the project.

What do I need?

There are a few things that you will need bofore you can implement this monitoring environment.

  • Access to docker.io and Github for pulling Docker images and this repository.
  • A SQL machine or VM that you would like to monitor.
  • 1-2 machines for running InfluxDB and Grafana, depending on how large your deployment is.
  • InfluxDB opened ports: 25826 (inbound data to InfluxDB), 8086 (outbound queries from Grafana)
  • Grafana opened port: 3000 (default web port for inbound connections)

Setting up InfluxDb.

For sizing InfluxDB, refer to their documentation. Also note that it is also recommended that you provision SSD volumes for the InfluxDB data and wal directories. Not a requirement if you are doing a small (monitoring a few machines) scale setup.

  • Install Docker Engine
    wget -qO- https://get.docker.com/ | sudo sh
    
  • Clone this repository
    git clone https://github.com/Microsoft/mssql-monitoring.git
    
  • Browse to ./mssql-monitoring/influxdb
  • Edit run.sh and change the variables to match your environment
  • Run run.sh. This will pull down the InfluxDB image and create and run the container
    sudo bash ./run.sh
    

Setting up collectd on the Linux SQL Server you want to monitor

  • Install Docker Engine
    wget -qO- https://get.docker.com/ | sudo sh
    
  • Clone this repository
    git clone https://github.com/Microsoft/mssql-monitoring.git
    
  • Browse to ./mssql-monitoring/collectd
  • Edit run.sh and change the variables to match your environment
  • Run run.sh. This will pull down the collectd image, set it to start on reboot and create and run the container.
    sudo bash ./run.sh
    

Setting up Grafana

If you are doing a small (monitoring a few machines) scale setup, you should be fine running this on the same host as your InfluxDB container. We use the image created by Grafana Labs and we just added a run.sh file that you can use to run the container.

  • Install Docker Engine
    wget -qO- https://get.docker.com/ | sudo sh
    
  • If you are running this separately, you will need to clone this repository
    git clone https://github.com/Microsoft/mssql-monitoring.git
    
  • Browse to ./mssql-monitoring/grafana
  • Run run.sh. This will pull down the Grafana image and create and run the container
    sudo bash ./run.sh
    
  • Browse to the URL where Grafana is running http://[MACHINE_IP]:3000
  • Add your first data source
    • Name: influxdb
    • Type: InfluxDB
    • Url: http://[INFLUXDB_IP]:8086 (If you are running Grafana on the same host as InfluxDB, you will need to specify the public end point for your InfluxDB instance. Alternatively, you can edit the run.sh file to include /net=host and the default URL will work.)
    • Database: collectd_db
  • Click Add
  • Browse back to the main page and import the dashboards from the dashboard dropdown menu. Dashboards are located under ./sqlserver_perf_monitoring/grafana_dashboard_templates in the GitHub repo.

mssql-monitoring's People

Contributors

microsoft-github-policy-service[bot] avatar microsoftopensource avatar msftgits avatar scschneider avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mssql-monitoring's Issues

Error importing Core server metrics dashboard

When importing the template Core Server Metrics, the dashboard will load, but most disk related options will not populate. The error message on importing or switching to the dashboard is "Templating init failed - Error parsing query: found ., expected identifier, string, number, boot at line 1, char 81"

json_error

dbi_result_next_row failed

My config looks like this:

TypesDB "/usr/share/collectd/types.db.mssql"

<LoadPlugin dbi>
    Interval 15
</LoadPlugin>

<Plugin dbi>
    <Query "perfstats">
	Statement "select replace(rtrim(counter_name),' ','_') as counter_name, replace(rtrim(instance_name),' ','_') as instance_name,cntr_value from sys.dm_os_performance_counters where (counter_name in ('SQL Compilations/sec','SQL Re-Compilations/sec','User Connections','Batch Requests/sec','Logouts/sec','Logins/sec','Processes blocked','Latch Waits/sec','Full Scans/sec','Index Searches/sec','Page Splits/sec','Page Lookups/sec','Page Reads/sec','Page Writes/sec','Readahead Pages/sec','Lazy Writes/sec','Checkpoint Pages/sec','Database Cache Memory (KB)','Log Pool Memory (KB)','Optimizer Memory (KB)','SQL Cache Memory (KB)','Connection Memory (KB)','Lock Memory (KB)', 'Memory broker clerk size','Page life expectancy')) or (instance_name in ('_Total','Column store object pool') and counter_name in ('Transactions/sec','Write Transactions/sec','Log Flushes/sec','Log Flush Wait Time','Lock Timeouts/sec','Number of Deadlocks/sec','Lock Waits/sec','Latch Waits/sec','Memory broker clerk size','Log Bytes Flushed/sec','Bytes Sent to Replica/sec','Log Send Queue','Bytes Sent to Transport/sec','Sends to Replica/sec','Bytes Sent to Transport/sec','Sends to Transport/sec','Bytes Received from Replica/sec','Receives from Replica/sec','Flow Control Time (ms/sec)','Flow Control/sec','Resent Messages/sec','Redone Bytes/sec') or (object_name = 'SQLServer:Database Replica' and counter_name in ('Log Bytes Received/sec','Log Apply Pending Queue','Redone Bytes/sec','Recovery Queue','Log Apply Ready Queue') and instance_name = '_Total')) or (object_name = 'SQLServer:Database Replica' and counter_name in ('Transaction Delay'))"
	<Result>
            Type "sql_server_perf_stats"
            InstancesFrom "counter_name" "instance_name"
            ValuesFrom "cntr_value"
        </Result>
    </Query>

    <Query "waitstats">
        Statement "WITH WaitCategoryStats ( wait_category, wait_type, wait_time_ms, waiting_tasks_count, max_wait_time_ms) AS ( SELECT CASE WHEN wait_type LIKE 'LCK%' THEN 'LOCKS' WHEN wait_type LIKE 'PAGEIO%' THEN 'PAGE I/O LATCH' WHEN wait_type LIKE 'PAGELATCH%' THEN 'PAGE LATCH (non-I/O)' WHEN wait_type LIKE 'LATCH%' THEN 'LATCH (non-buffer)' WHEN wait_type LIKE 'LATCH%' THEN 'LATCH (non-buffer)' ELSE wait_type END AS wait_category, wait_type, wait_time_ms, waiting_tasks_count, max_wait_time_ms FROM sys.dm_os_wait_stats WHERE wait_type NOT IN ('LAZYWRITER_SLEEP', 'CLR_AUTO_EVENT' , 'CLR_MANUAL_EVENT', 'REQUEST_FOR_DEADLOCK_SEARCH', 'BACKUPTHREAD', 'CHECKPOINT_QUEUE' , 'EXECSYNC', 'FFT_RECOVERY', 'SNI_CRITICAL_SECTION', 'SOS_PHYS_PAGE_CACHE' , 'CXROWSET_SYNC', 'DAC_INIT', 'DIRTY_PAGE_POLL', 'PWAIT_ALL_COMPONENTS_INITIALIZED' , 'MSQL_XP', 'WAIT_FOR','DBMIRRORING_CMD','DBMIRROR_DBM_EVENT','DBMIRROR_EVENTS_QUEUE','DBMIRROR_WORKER_QUEUE', 'XE_TIMER_EVENT','XE_DISPATCHER_WAIT','WAITFOR_TASKSHUTDOWN','WAIT_FOR_RESULTS' ,'SQLTRACE_INCREMENTAL_FLUSH_SLEEP','WAITFOR' ,'QDS_CLEANUP_STALE_QUERIES_TASK_MAIN_LOOP_SLEEP','QDS_PERSIST_TASK_MAIN_LOOP_SLEEP','HADR_FILESTREAM_IOMGR_IOCOMPLETION','LOGMGR_QUEUE','FSAGENT' ) AND wait_type NOT LIKE 'PREEMPTIVE%' AND wait_type NOT LIKE 'SQLTRACE%' AND wait_type NOT LIKE 'SLEEP%' AND wait_type NOT LIKE 'FT_%' AND wait_type NOT LIKE 'XE%' AND wait_type NOT LIKE 'BROKER%' AND wait_type NOT LIKE 'DISPATCHER%' AND wait_type NOT LIKE 'PWAIT%' AND wait_type NOT LIKE 'SP_SERVER%') select  wait_category, sum(wait_time_ms) as wait_time_ms, sum(waiting_tasks_count) as waiting_tasks_count, max(max_wait_time_ms) as max_wait_time_ms from WaitCategoryStats where wait_time_ms >100 group by wait_category"
        <Result>
            Type "sql_server_wait_stats"
            InstancesFrom "wait_category"
            ValuesFrom "wait_time_ms" "waiting_tasks_count" "max_wait_time_ms"
        </Result>
    </Query>

    <Database "master">
        Host "MSSQL"
        Driver "freetds"
        DriverOption "host" "fake_host.eu-central-1.rds.amazonaws.com"
        DriverOption "username" "fake_user"
        DriverOption "password" "fake_pass"
        Query "perfstats"
	#Query "waitstats"
    </Database>
</Plugin>

But unfortunatelly I don't really get any usable stats in collectd. It outputs this error message:

dbi plugin: cdbi_read_database_query (master, perfstats): dbi_result_next_row failed: dbi_conn_error failed with status -6

I am using Debian 9 with collectd 5.7.1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.