Giter VIP home page Giter VIP logo

sqlcover's Introduction

SQLCover - Code coverage for SQL Server T-SQL

licence badge stars badge forks badge issues badge

Redgate

Thanks to Redgate for supporting this open source project

Supported by Redgate

Overview

This is a code coverage tool for SQL Server 2008+, it was designed to be generic to work with any build server and tests but includes specific filters to mean that it is well suited to running tSQLt tests using the Redgate DLM Automation Suite.

Navigation:

Installation

You will either need to build the project and grab the output SQLCover.dll or you can download the pre-built binary from: http://the.agilesql.club/SQLCover/download.php

Usage

There are three basic ways to use it:

1. Redgate DLM Automation Suite

If you have the DLM automation suite then create a nuget package of your database, deploy the project to a test database and then use the example powershell script (https://github.com/GoEddie/SQLCover/blob/master/example/SQLCover.ps1 and included in the download above):

Get-CoverRedgateCITest "SQLCover-path.dll" "server=servername;integrated security=sspi;" "nuget-package-path.nupkg" "servername" "database-name"

To create the nupkg of your database you can use sqlci.exe or create a zip of your .sql files see: https://www.simple-talk.com/blogs/2014/12/18/using\_sql\_release\_with\_powershell/

The Get-CoverRedgateCITest will return an array with two objects in, the first object is a:

RedGate.SQLRelease.Compare.SchemaTesting.TestResults

The second object is a:

SQLCover.CoverageResult

This has two public properties:

public long StatementCount;
public long CoveredStatementCount;

It also has two public methods:

public string Html()

This creates a basic html report to view the code coverage, highlighting the lines of code in the database which have been covered and:

public string OpenCoverXml()

which creates an xml file in the OpenCoverageXml format which can be converted into a very pretty looking report using reportgenerator: https://github.com/danielpalme/ReportGenerator

For a complete example see:

$results = Get-CoverRedgateCITest "path\to\SQLCover.dll" "server=.;integrated security=sspi;initial catalog=tSQLt_Example" "tSQLt_Example"
    Export-DlmDatabaseTestResults $results[0] -OutputFile c:\temp\junit.xml -force
    Export-OpenXml $results[1] "c:\output\path\for\xml\results"
    Start-ReportGenerator "c:\output\path\for\xml\results" "c:\path\to\reportgenerator.exe"

2. Cover T-SQL Script

If you have a script you want to cover then you can call:

Get-CoverTSql  "SQLCover-path.dll" "server=servername;integrated security=sspi;"  "database-name" "exec tSQLt.RunAll

This will give you a CoverageResults where you can either examine the amount of statement covered or output the full html or xml report.  

3. Cover anything else

If you want to have more control over what is covered, you can start a coverage session, run whatever queries you like from whatever application and then stop the coverage trace and get the CoverageResults which you can then use to generate a report.

$coverage = new-object SQLCover.CodeCoverage($connectionString, $database)
$coverage.Start()
#DO SOMETHING HERE
$coverageResults = $coverage.Stop()

 

4. Tidying up

When we target local sql instances we delete the trace files but when targetting remote instances we are unable to delete the files as we do not (or potentially) do not have access. If this is the case keep an eye on the log directory and remove old "SQLCover-Trace-.xel" and "SQLCover-Trace-.xem" files.

License

Apache 2.0

Development

To run the integration tests, create a sql instance or using the ./src/SQLCover/test/CreateDockerDbInstance.ps1 script to create a docker version of sql. Then run ./src/SQLCover/test/deployLocal.ps1 (if you use your own instance deploy the DatabaseProject ssdt project to the instance). The connection string that the integration tests use is Server=tcp:docker-instance-ip;uid=sa;pwd=Psgsgsfsfs!!!!!;initial catalog=DatabaseProject where docker-instance-ip is the ip address of the container created by ./src/SQLCover/test/CreateDockerDbInstance.ps1. If you want the tests to use a different connection string, in the \src\SQLCover\DatabaseProject\bin\Debug folder put a ConnectionString.user.config which is a text file with one line that is the connection string that you want to use. The unit tests need no connection string.

sqlcover's People

Contributors

aaatechguy avatar aboatswain avatar epsilony avatar fsvreddit avatar goeddie avatar ktaranov avatar lastlink avatar mauvo avatar riserrad avatar tgruetzm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sqlcover's Issues

Cannot use or export using Cobertura on CoverageResult

Describe the bug
I am able to run and get coverage result using Get-CoverTSql PowerShell wrapper function.
Source has a the public string Cobertura(string packageName = "sql", Action customCoverageUpdater = null) line 315 but the method is not visible when consuming in powershell

To Reproduce
Steps to reproduce the behavior:

  1. Run $result = Get-CoverTSql "$openCoverSourceFolder\SQLCover.dll" $connectionString $databaseName $runtSQLtTestsCmd
  2. Try to run $result.Cobertura()
  3. See error
    Method invocation failed because [SQLCover.CoverageResult] does not contain a method named 'Cobertura'.
    At line:1 char:1
  • $result.Cobertura()
  •   + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
      + FullyQualifiedErrorId : MethodNotFound
    
    

Expected behavior
$result = Get-CoverTSql "$openCoverSourceFolder\SQLCover.dll" $connectionString $databaseName $runtSQLtTestsCmd
$result.Cobertura() | Out-File $path
$result.SaveSourceFiles($reportPath)

@GoEddie

Clustered Index coverage

Hi Team,

I have a stored procedure which creates a table variable with a clustered index.

SQLCover doesn't give the code coverage for it. Please suggest how to cover that statement.

Please refer the screenshot below
ClusteredIndexIssue

Thanks.
Surbhi Jain

SQLCover.CodeCoverage.Stop() gives Exception calling "Stop" with "0" argument(s): "Value cannot be null. Parameter name: source"

Describe the bug
SQLCover.CodeCoverage.Stop() results in error message: Exception calling "Stop" with "0" argument(s): "Value cannot be null. Parameter name: source".

To Reproduce
Steps to reproduce the behavior:

  1. Install SQL Server 2019
  2. Install Redgate SQL Change Automation 4.3
  3. Install SQLCover release 0.5.0 to the "SQLCover" directory
  4. Run this PowerShell script:
    $packageVersion = "1.0.0"
    $packageID = "MyPackage"
    $serverInstance = "."
    $temporaryConnectionString = "Data Source=.;Integrated Security=True"
    $sqlCoverDllPath = "SQLCover\SQLCover.dll"

Import-Module SqlChangeAutomation -ErrorAction SilentlyContinue -ErrorVariable +ImportErrors

$statePath = Split-Path -Parent $MyInvocation.MyCommand.Path
$scriptsFolder = Join-Path -Path $statePath -ChildPath "..\State"

$validatedScriptsFolder = Invoke-DatabaseBuild $scriptsFolder -TemporaryDatabaseServer $temporaryConnectionString
$packageFilename = $packageID + "." + $packageVersion + ".nupkg";
If (Test-Path $packageFilename)
{
Remove-Item $packageFilename
}

$databasePackage = New-DatabaseBuildArtifact $validatedScriptsFolder -PackageId $packageID -PackageVersion $packageVersion
Export-DatabaseBuildArtifact $databasePackage -Path "."

$database = New-Guid

Unblock-File -Path $sqlCoverDllPath

Add-Type -Path $sqlCoverDllPath

Invoke-SqlCmd -ServerInstance $serverInstance -Query "CREATE DATABASE [$database]; ALTER DATABASE [$database] SET TRUSTWORTHY ON"
$coverage = New-Object SQLCover.CodeCoverage($temporaryConnectionString, $database)
$coverage.Start()
$temporaryDatabase = New-DatabaseConnection -ServerInstance $serverInstance -Database $database
$testResults = Invoke-DatabaseTests $databasePackage -TemporaryDatabase $temporaryDatabase
$coverageResults = $coverage.Stop()

Write-Host "CoverageResults: $coverageResults"

Expected behavior
$coverageResults referencing a CoverageResult object instance.

Desktop (please complete the following information):

  • OS: Windows 10 Enterprise

Additional context
The exception happens even if there are no tests to be run.

If I change
"$testResults = Invoke-DatabaseTests $databasePackage -TemporaryDatabase $temporaryDatabase"
to:
"#$testResults = Invoke-DatabaseTests $databasePackage -TemporaryDatabase $temporaryDatabase"

then the new script runs without an error.

Output:

PS> .\Reproduce.ps1
Invoke-DatabaseBuild, SQL Change Automation 4.3.20205.21469, Copyright © Red Gate Software Ltd 2014-2019
Validating SQL Source Control project 'REMOVED'.
Schema validation successful.
New-DatabaseBuildArtifact, SQL Change Automation 4.3.20205.21469, Copyright © Red Gate Software Ltd 2014-2019
Export-DatabaseBuildArtifact, SQL Change Automation 4.3.20205.21469, Copyright © Red Gate Software Ltd 2014-2019
Database build artifact exported to file 'REMOVED\MyPackage.1.0.0.nupkg'.
New-DatabaseConnection, SQL Change Automation 4.3.20205.21469, Copyright © Red Gate Software Ltd 2014-2019
Invoke-DatabaseTests, SQL Change Automation 4.3.20205.21469, Copyright © Red Gate Software Ltd 2014-2019
Cleaning database 'aab10b86-1eba-48a1-8fe7-ed7de7db3479' on server 'REMOVED'.
Running tests contained in SQL Source Control build artifact
Collecting results
All tests passed
Exception calling "Stop" with "0" argument(s): "Value cannot be null.
Parameter name: source"
At Reproduce.ps1:34 char:1

  • $coverageResults = $coverage.Stop()
  •   + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
      + FullyQualifiedErrorId : ArgumentNullException
    
    

CoverageResults:

===
Please at me @GoEddie so I receive a notification

News about release 0.6 or what's next?

Any news about release 0.6 or SQLCover version that works cross-platform?

It would be awesome to push it even further to the .NET 6.

@GoEddie - Please give us your thoughts about future maintaining SQLCover project.

SQL Azure Databases return empty string when querying compatibility_level

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
It raises an exception when trying to cover tSQLt tests on an Azure SQL Database.

**If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via http://sqlfiddle.com/
Try to run a unit test/code coverage session on an Azure SQL Database. For example, I'm using the following PowerShell script:

$coverage = new-object SQLCover.CodeCoverage($connectionString, $database, $null, $true, $true)

$startResult = $coverage.Start()

try{
    Invoke-SqlCmd -ServerInstance "$server" -Database "$database" -Username "$username" -Password "$password" -QueryTimeout 60 -Query "EXEC tSQLt.RunAll"
} catch [SqlPowerShellSqlExecutionException] {
    $testsFailed = $true
}

$coverageResults = $coverage.Stop()

The exception stack trace is:

Error starting trace: System.NullReferenceException: Object reference not set to an instance of an object.
   at SQLCover.Gateway.DatabaseGateway.GetString(String query) in C:\dev\SQLCover\src\SQLCover\SQLCover\Gateway\DatabaseGateway.cs:line 32
   at SQLCover.Source.DatabaseSourceGateway.GetVersion() in C:\dev\SQLCover\src\SQLCover\SQLCover\Source\DatabaseSourceGateway.cs:line 24
   at SQLCover.Trace.TraceControllerBuilder.GetTraceController(DatabaseGateway gateway, String databaseName, TraceControllerType type) in C:\dev\SQLCover\src\SQLCover\SQLCover\Trace\TraceControllerBuilder.cs:line 27
   at SQLCover.CodeCoverage.Start() in C:\dev\SQLCover\src\SQLCover\SQLCover\CodeCoverage.cs:line 66

It happens because of the following code:

public virtual string GetString(string query)
{
    using (var conn = new SqlConnection(_connectionString))
        {
            conn.Open();
            conn.ChangeDatabase(_databaseName);
            using (var cmd = conn.CreateCommand())
            {
                cmd.CommandText = query;
                return cmd.ExecuteScalar().ToString();
            }
        }
}

It is not checking if cmd.ExecuteScalar() returns null or not. Then, calling .ToString() raises the error.

What is the expected behavior?

Run the script and start the coverage session. Actually, SQL Azure seems to block it periodically. This same implementation was working properly yesterday. Running it today again produced the error.

Which versions of SQL Server and which OS are affected by this issue? Did this work in previous versions of our procedures?
Azure SQL Database

Allign to the OpenCover format https://github.com/csMACnz/coveralls.net is using

Hi @GoEddie and thanks for your efforts.

There is a fileid attribute on each SequencePoint.
https://github.com/OpenCover/opencover/blob/e715910bb52e327ae6df059aeea71dff4b54ee50/main/OpenCover.Framework/Model/SequencePoint.cs#L35

I'm directly feeding it into the coveralls uploader https://github.com/csMACnz/coveralls.net, without using the reportgenerator.

For now, I use this Powershell snippet as a workaround, to make it working in my TeamCity Build.

[xml]$xmldocument = Get-Content -Path "./SQLCoverResults.xml"
Foreach ($class in ($xmldocument.CoverageSession.Modules.Module.Classes.GetElementsByTagName("Class"))) {
    $fileref = $class.Methods.Method.FileRef.uid
    foreach ($SequencePoint in $class.Methods.Method.SequencePoints.GetElementsByTagName("SequencePoint")) {
        $SequencePoint.Setattribute("fileid",$fileRef)
    }
}
$xmldocument.Save("./SQLCoverResults_2.xml");`

Not sure if you see this issue more here or in coveralls.net .

Thanks Oli

Default Tracing error with SQLLocalDB datasource based on Windows named pipe line

Describe the bug
When datasource is some thing like: np:\\.\pipe\LOCALDB#8216D766\tsql\query, and if Default

To Reproduce
Steps to reproduce the behavior:

  1. Create a tSql test based on LocalDB
  2. Pass the named pipeline url to the Coverage construction with default trace settings:
    var cc = new CodeCoverage("np:\\\\.\\pipe\\LOCALDB#8216D766\\tsql\\query", connectionString, "master", null, true);
  3. do any coverage testing:
    _codeCoverage.Cover($"exec tSQLt.Run '{_testFullName}'");
  4. See error
    SQLCover.SqlCoverException : Unable to start the trace - errors are recorded in the debug output
    Error starting trace: System.Data.SqlClient.SqlException (0x80131904): Failed to open loopback connection. ...
    Expected behavior

No exceptions

Desktop (please complete the following information):

  • OS: Windows 10
  • DB: SQLocalDB 2017

===
Please at me @GoEddie so I receive a notification

Publish Coverage Results in Azure DevOps

Describe the bug
When using the Cobertura format and using the Publish Coverage Results task in ADO I get the following results in the logs

"C:\Program Files\dotnet\dotnet.exe" C:\Agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\1.186.0\netcoreapp2.0\ReportGenerator.dll -reports:C:\Agent\_work\101\a/**/code-coverage-report*.xml -targetdir:C:\Agent\_work\_temp\cchtml -reporttypes:HtmlInline_AzurePipelines
2021-05-20T14:44:42: Arguments
2021-05-20T14:44:42:  -reports:C:\Agent\_work\101\a/**/code-coverage-report*.xml
2021-05-20T14:44:42:  -targetdir:C:\Agent\_work\_temp\cchtml
2021-05-20T14:44:42:  -reporttypes:HtmlInline_AzurePipelines
2021-05-20T14:44:43: File '' does not exist (any more).
... (1120 more lines)
2021-05-20T14:44:44: File '' does not exist (any more).
2021-05-20T14:44:46: Writing report file 'C:\Agent\_work\_temp\cchtml\index.html'
2021-05-20T14:44:46: Report generation took 3.5 seconds
Generated code coverage html report: C:\Agent\_work\_temp\cchtml
Reading code coverage summary from 'C:\Agent\_work\101\a\_core-edw-combined-ci\core-edw-dwh-tests\DiFramework.UnitTests\code-coverage-report-datawarehouse-2021-05-20T14.44.30.xml'
Async Command Start: Publish code coverage
Publishing coverage summary data to TFS server.

When looking at the published result in ADO it has the metrics but not the file or code, see screenshot below
image

I've noticed there is a field on the taks called Path to Source files with the following tooltip

Path to source files is required when coverage XML reports do not contain absolute path to source files. For e.g., JaCoCo reports do not use absolute paths and when publishing JaCoCo coverage for Java apps, the pattern would be similar to $(System.DefaultWorkingDirectory)/MyApp/src/main/java/.
This input is also needed if tests are run in a docker container. This input should point to absolute path to source files on the host. For e.g., $(System.DefaultWorkingDirectory)/MyApp/

We are not using JaCoCo so I'm not sure if the Cobertura format generated by SQLCover is incorrect or if I need to supply something in the Path to Source files field to make this work correctly

Any ideas are welcome, thanks.

@GoEddie hope you are well mate :)

Inline Table Valued Functions

When I have a tSQLt test that is written to test an inline table function there is no coverage results provided for any of the tests for that function. Please let me know if there is anything I need to provide to assist with resolving this issue. I am using Red Gate's SQL Test product that now uses SQL Cover so I am not very familiar with this specific project but was told that this was the best place to report this issue.

Thanks,

Adam

Question: Is it possible to run SQLCover on Linux?

We have moved our CICD process to run in a container using a SQL Server 2017 Linux image (mcr.microsoft.com/mssql/server: 2017-CU13-ubuntu) and run flyway.

We now have a requirement to get code coverage for the unit test we have, the solution looks to be SQLCover, but looking at the repo code it looks to be NetFramework 4.6,1.

Are there any suggestions to get SQLCover running on Linux?

MIT or apache 2.0 licence?

The project page on GitHub is referencing both (MIT and Apache 2.0) licence. Can we use both licences?

sql-cover-licence

Extended Event logs are not cleaned up when running against a remote server

When running against a remote server, the extended event log file is not cleaned up properly. This is because the Drop method in SqlTraceController assumes that the tool is running on the local SQL Server and just deletes from the local filesystem, leaving the .xel files in place on the remote server.

Short of using xp_cmdshell or similar there's not much that could be done to resolve this other in SQLCover itself, but it is probably worth noting this in documentation to ensure that people are prepared.

Memory issues using datasets

Great tool! I have been testing it with our suite of DBFit tests. Everything works fine when I only run a small set of tests, but when the trace data gets large I am seeing memory issues. I created a console app to test SQLCover in the debugger and see where the issue was. I am getting out of memory exceptions when the DataTable is loaded from the the SqlDataReader (DatabaseGateway.GetRecords() called by ReadTrace()). I am not sure what can be easily changed to address this, since datasets are always loaded in memory.

github actions

Is your feature request related to a problem? Please describe.
Very useful to have an automated pipeline whenever there is a new pull request and to show sqlcover in action. You can use github actions and service containers to acomplish this.

Describe the solution you'd like

  • automated github actions running unit tests against a sql server db

Describe alternatives you've considered

Support SQL Server inline table valued functions?

Is your feature request related to a problem? Please describe.
We've been changing several of our table valued functions from the standard format like below to inline table valued functions

CREATE FUNCTION dbo.FN_SomeFunction(
@inputvar VARCHAR(50)
)
RETURNS @functionResult TABLE (
output VARCHAR(50)

)
AS
BEGIN
INSERT INTO @functionResult(output)
Select resultOutput as output
from dbo.TableA
where something = @inputvar;

 RETURN

END;

We've found numerous performance improvements by using inline TVF functions where it's written like:
CREATE FUNCTION dbo.FN_SomeFunction(
@inputvar VARCHAR(50)
)
RETURNS TABLE
AS
RETURN (
Select resultOutput as output
from dbo.TableA
where something = @inputvar;
);

Describe the solution you'd like
@GoEddie
Is there a way to get SQLCover to cover these functions when a tSQLt test is created for this function? It does not show up in the list of evaluated code objects or in the coverage.coverxml outputs.

Describe alternatives you've considered
The only alternative I have is to put the functions back but then we degrade our performance.

===
Please at me @GoEddie so I receive a notification

Report statement counting

Do you want to request a feature or report a bug?
Up to Ed and team

What is the current behavior?
Report states 20 statements executed out of 22 accounted for

**If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via http://sqlfiddle.com/
Create procedure with goto statement and define label. The label is counted towards as a valid statement, but it never gets executed. Hence, reporting is off.

What is the expected behavior?
Reporting needs to count all executed statements.
So, if procedure has label(s) that do not get drawn into the set of executed statements, the label statement would be very nice to be placed on the "IsIgnoredType" list, so that statements, available and executed tallied up for report correctly and completely

Which versions of SQL Server and which OS are affected by this issue? Did this work in previous versions of our procedures?
SQL Server 2016 please. I didn't check earlier, but I'd assume those will behave identically.

The change is very simple, add lines

        if (statement is LabelStatement)
            return true;

to the IsIgnoredType statement.
Please consider change.

Code coverage not reported correctly for CTEs at the end of a stored procedure if the 'with' is immediately preceded with a semicolon

If a CTE is preceded by a semicolon e.g.

;with MyCTE as
(
...

and it is the last statement in a batch, the executed statement is matched against the immediately preceding statement instead. This is because the Overlaps function matches the preceding statement first due to the way that the coveredEnd variable is assigned the same value as the coveredOffset.

(This technically applies to other statements preceded with a semicolon, but it's unusual to see those).

I believe that the logic should change to treat those where coveredStatement.OffsetEnd is -1 (i.e. last statement) separate from others, and return true where the coveredOffset is equal or less than the tested statement's offset.

For other covered statements, the logic can now be simplified as we're no longer having to cope with both cases.

I've developed and tested a fix locally on my fork and will submit a pull request shortly.

Trace Controller choice causes "This operation is not supported in this version of SQL Server"

When referencing an VM hosted instance of SQL Server running in Azure (SQL Server 2016), it defaulted to the AzureTraceController instead of SqlTraceController.

This causes the "This operation is not supported in this version of SQL Server" error since it tries to create an event session at the database scope rather than server scope. Also the dmv references are incorrect.

Missing max_rollover_files option in definition of event session target file leads to lost of coverage info

When using SQLCover.dll to gather coverage information on large project with great amount of TSQLT tests we found that MSSQLServer creates only 5 trace files (SQLCover-Trace-*.xel), and then overwrites older files. Each file size is close to 1GB.

@@Version = Microsoft SQL Server 2016 (SP1) (KB3182545) - 13.0.4001.0 (X64) Oct 28 2016 18:17:30 Copyright (c) Microsoft Corporation Developer Edition (64-bit) on Windows Server 2016 Datacenter 6.3 (Build 14393: ) (Hypervisor)

The reason of the problem: when creating event session with asynchronous_file_target, parameter max_rollover_files is not defined. Default value for max_rollover_files in MSSQL is 5, therefore after writing 5 files it overwrites first one, so coverage information for first tests gets lost before reading.

Fix: to solve that problem we had to modify \src\SQLCover\SQLCoverLib\Trace\SqlTraceController.cs by adding max_rollover_files=(999) into definition of target in constant CreateTrace. Fixed line with max_rollover_files added looks like this:

 protected const string CreateTrace = @"CREATE EVENT SESSION [{0}] ON SERVER 
ADD EVENT sqlserver.sp_statement_starting(action (sqlserver.plan_handle, sqlserver.tsql_stack) where ([sqlserver].[database_id]=({1})))
ADD TARGET package0.asynchronous_file_target(
     SET filename='{2}', max_rollover_files=(999))
WITH (MAX_MEMORY=100 MB,EVENT_RETENTION_MODE=NO_EVENT_LOSS,MAX_DISPATCH_LATENCY=1 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=NONE,TRACK_CAUSALITY=OFF,STARTUP_STATE=OFF) 
";

Azure SQL DB shows 0 lines covered

Azure SQL DB is showing no lines covered, seems to be that not all events are being received (need to be more aggresive in draining events after finishing a run) and also statements don't seem to be matching up.

Cobertura updates/adherence

Currently SQLCover doesn't populate the filename field in its Cobertura report. Per the DTD, this is a required field.

Since many downstream tools that consume Cobertura will assume this field has a value, I think it is better to put something rather than nothing, to make a stronger attempt to adhere to the format and increase output ingestion compatibility.

There is also a 0.4 DTD available vs the 0.3 in use currently.

System diagram procedures are currently included in the code coverage results

The system diagram procedures are currently included in the code coverage results.
These shoudn't be included as they are system procedures so we shouldn't need to write tests for them, and it negatively impacts the total coverage %.

I believe they can be recognised by either their name (sp_%diagram) or by the extended property microsoft_database_tools_support.

2019-10-31 08_44_18-SQL Test - Microsoft SQL Server Management Studio

I am currently using Code Coverage as part of the RedGate SQLTest plugin.
I had approached RedGate about this and they pointed me to here :)
https://forum.red-gate.com/discussion/85304/how-can-i-exclude-the-system-stored-procedures-for-diagrams-from-code-coverage-results

@GoEddie

Encrypted SQL objects causing issue..

If any of my SP's/Functions are created as Encrypted, while stopping it gives me error as.
(as a work around deleting encrypted objects.)

Exception calling "Stop" with "0" argument(s): "Unable to cast object of type 'System.DBNull' to type 'System.String'."
At C:\Tools\SQLCover.ps1:106 char:17

  •             $coverageResults = $coverage.Stop()
    
  •             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

Thanks
Vidhyashankar

Test coverage is reported incorrectly if extended events files roll over

Many events are missing from the Extended Events trace files when the output is large enough to cause a rollover to the next file (by default, this is 1Gb).

We have a very large test suite - over 2500 tests across 187 test classes. This generates around 4.5Gb worth of extended events logs in the default settings.

I noticed that coverage wasn't being reported as good as I was expected, and some stored procedures that I know have good coverage were reporting not being visited. Their object IDs were missing from the .xel files when analysed directly, but the tests themselves were run when I checked tSQLt.TestResult.

I noticed when I changed the SQL statement run from tSQLt.RunAll to just run the test class in question the test coverage was correct for the stored procedures I was examining.

I have worked around this by editing the SQL statement that sets up the extended event session to specify max_file_size=(65536) on the target - the output is now 10Gb, over twice the size of the split files.

Obviously this is a bug in SQL Server itself, not SQL Cover, but it may be worth working around the issue. One option might be to make the max_file_size configurable (similar to the timeout, which as you can imagine that I'm having to increase!).

A better option might be to split the test runs - rather than using tSQLt.RunAll, it might be better to batch up by test class and retrieve the extended events data after each test class is run. This will reduce disk space consumption on the SQL Server as well.

Generating partial test coverage report for the Azure DB.

Describe the bug
I have run the Power shell for SQL coverage report for Azure DB, it shows me partial sqlunit test coverage as well and some time it is covering more or less test cases. I wrote SQL Unit Tests with SQL Server Data Tools.

To Reproduce

I have made script to run all azure test cases and to get sql cover report.


$coverDllPath =  "C:\Downloads\SQLCover-0.5.0\SQLCover-0.5.0\src\SQLCover\releases\template\SQLCover.dll"

$connectionString = "Server=server_name;Database=db_name;Persist Security Info=False;User ID=master;Password=pa$$word;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"
$databaseName = "db_name"


$outputPath ="C:\Downloads\SQLCover-0.5.0\SQLCover-0.5.0\src\SQLCover\releases\template\result"


if(!(Test-Path $coverDllPath)){
Write-Error "SQLCover.dll path was not found ($coverDllPath)"
return
}

Unblock-File -Path $coverDllPath
   
Add-Type -Path $coverDllPath
    
$coverage = new-object SQLCover.CodeCoverage ($connectionString, $databaseName)

$coverage.Start()

# Run all test cases .
mstest.exe /testcontainer:C:\Project\test_project\bin\Debug\RamSoft.AzureSQL.Test.dll

$result = $coverage.Stop() 

$xmlPath = Join-Path -Path $outputPath -ChildPath "Coverage.html"
    $result.Html() | Out-File $xmlPath
    $result.SaveSourceFiles($outputPath) 

Snapshot
image
image

As you can see only I have test cases which executes more than 1000 statements but it covers partially.

Expected behavior
Should cover all the statements of the test cases

Additional context

I am using latest version of SQLCover i.e. 0.5.0.
I unlocked the SQLcover.dll as well.
It run all the test cases means it is trigger the statements. But not covered all of them while generating cover report.
I have also tried with CoverExe function but it gave same result for me.
I have also tried to increase timeout, but it had same results

Please @GoEddie help me to solve this issue.

Two integration tests failing

Don't know if it's a bug or if the test are wrong :-)

Tests have been executed on AppVeyor with SQL Server 2014 Express and on my computer with SQL Server 2016 Developer edition and exactly gave the same result:

  • Code_Coverage_Includes_Last_Statement_Of_Large_Procedure : (line 45) : Expected 2 but was 0
  • Code_Coverage_Returns_All_Covered_Statements : (line 69) : Expected string containing (HitCount="1") but wasn't.

Issue Warning on Encrypted Stored Procs/Functions

I was running Get-CoverTSql on a database that contains some business logic. There were several encrypted procedures (most were unencrypted) and the Get-CoverTSql command would not finish its execution. It took a while to trace it back to the fact that the batches.Add (line 63 in DatabaseSourceGateway.cs) was throwing an error when it tried to parse the encrypted object.

How to exclude database objects from coverage reports?

Hey @GoEddie :)

Thanks for this awesome library. I am running some tests against our DB in a Docker container in a CI pipeline and while there's nothing necessarily wrong, I am trying to figure out how to exclude objects from coverage. There appears to be little (to possibly no) documentation no how to do the exclusions correctly?

The SQLCover.CodeCoverage (PowerShell) has a couple of options, the one I am interested is:
New-Object SQLCover.CodeCoverage($connectionString, $DbName, $ObjectFilters, $DebugLogging)

It seems to support the notion of filtering out objects but I actually don't know how to do this 😢

Any help would be nice.

In particular, I have two schemas on my DB:

  • dbo
  • Versioning

I would like to exclude the Versioning schema from coverage.

SQLCover filters out the SP that has index created on table variable

Hi @GoEddie ,

I have SP that has the below declaration with index .

CREATE  PROCEDURE [dbo].[my_procedure_v1]
AS
BEGIN
    SET NOCOUNT ON
    SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTE
DECLARE @user TABLE (
      [name] [varchar](20) NULL,
      [age] [int ] NULL,
      [ssn] [varchar](20) NOT NULL
      PRIMARY KEY (ssn),
      INDEX  IX01 (name,age,ssn)
)
//DO SOMETHING
END

I am executing tsqlt tests via sqlcover ,
Get-CoverTSql ".\SQLCover.dll" "server=localhost,XXX;User ID=****;Password=******" "db_name" "exec tsqlt.RunAll" -o TestCover

 function Get-CoverTSql{
    param(
        [string]$coverDllPath
    ,[string]$connectionString
    ,[string]$databaseName
    ,[string]$query
    )
    if(!(Test-Path $coverDllPath)){
        Write-Error "SQLCover.dll path was not found ($coverDllPath)"
        return
    }

    Add-Type -Path $coverDllPath

    $coverage = new-object SQLCover.CodeCoverage ($connectionString, $databaseName)
    $coverage.Cover($query)
}

However when I check the xml report generated , the SP (my_procedure_v1 in this case), doesn’t appear . But when I remove the INDEX that I have on the TABLE variable, it comes up in the xml report. Could you please help guide what is the issue here ? Why is SQLCover filtering out the SP. This is a blocker for us as we have bunch of SP that has a similar pattern.

Thanks,
Priya

Error starting trace: System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.

This is an awesome tool. We have been using it for a while now and it is really helping us in our CI/CD efforts. We have several databases that are local and SQL Cover works very well. Other databases are remote and we get timeout errors when it is trying to start the SQL trace. Is there a way to change the SQL timeout? It appears to default to 30 seconds.

Here is the snippet of code we are using to call SQL Cover:

$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = "server=our_server;database=our_database;integrated security=SSPI;"
Add-Type -Path D:\SQLCover\SQLCover\SQLCover.dll
$coverage = new-object SQLCover.CodeCoverage($connection.ConnectionString, $connection.Database, $true, $true) 

    ######## This is where it times out. #######
$status = $coverage.Start(); 
    #########################

    #Execute the unit tests here  (using VSTest) 

    $coverageResults = $coverage.Stop()

Thank you for a great product!

[BUG] Timeout error "Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding"

Do you want to request a feature or report a bug?
BUG

What is the current behavior?
The build is executed on build machine is failing with error message
2018-02-27T16:36:32.6985021Z Starting code coverage...
2018-02-27T16:37:07.3044875Z ##[error]Exception calling "Cover" with "1" argument(s): "Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Output from the tests was shown in the output of the build, so the execution was not stuck.
I ran with 'exec tSQLt.RunAll'

**If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via http://sqlfiddle.com/
N/A

What is the expected behavior?
Expecting to have this step finished within a longer period than it is hardcoded in the code.

Which versions of SQL Server and which OS are affected by this issue? Did this work in previous versions of our procedures?
SQL2008R2
Win2012
TFS2015
This error was noticed on the first run.

Exception running code coverage: Unable to cast object of type 'System.DBNull' to type 'System.String'

Hi @GoEddie

I am running SQLCover against a local database. However I am getting the below exception.
at SQLCover.Source.DatabaseSourceGateway.GetBatches(List1 objectFilter) at SQLCover.CodeCoverage.GenerateResults(List1 filter, List`1 xml)
at SQLCover.CodeCoverage.Cover(String command)

On running the query,
SELECT sm.object_id, ISNULL('[' + OBJECT_SCHEMA_NAME(sm.object_id) + '].[' + OBJECT_NAME(sm.object_id) + ']', '[' + st.name + ']') object_name, sm.definition, sm.uses_quoted_identifier FROM sys.sql_modules sm LEFT JOIN sys.triggers st ON st.object_id = sm.object_id WHERE sm.object_id NOT IN(SELECT object_id FROM sys.objects WHERE type = 'IF');

I am getting results with values in all the columns.

Could you please help ?

Thanks,
Priya

SQLCover for replicated database

Hi Team,

When I create a SQL server publication on my database, and then use SQLCover for it, I get the following error:
image

I have attached the Test.ps1 below.
test.txt

Now when I remove the SQL replication via executing "EXEC sp_removedbreplication 'TestDb'", then the coverage script works fine.

Is there a way for me to find the coverage without actually removing the replication?

@GoEddie

Postgres, mysql support

Is your feature request related to a problem? Please describe.
Be able to support postgres and mysql unit test for getting coverage.
Postgres has several sql unit testing frameworks such as pgtap and mysql has some perl based unit tests https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_QUICK_START.html or mytap

Describe the solution you'd like
An additional parameter to specify sql provider used. May have to add additional sub projects to deploy the different databases looks liked sqitch May be the best tool for that. github actions will work fine w/ a postgres and mysql integrated docker test.

Describe alternatives you've considered
Could add more support if tracing in different providers supported and or a new project completely forked off of this maybe if you think this deviates from the current project.

I may await your response on whether this should be part of this project or a new project.

===
Please at me @GoEddie so I receive a notification

Please add support for SQL Server 2017

Please add support for SQL Server 2017

SQLCover / src / SQLCover / SQLCover / Objects / SqlServerVersion.cs

limit list by 130, while 140 already here.

Please update for support sakes.

Feature Requests

If you have any feature requests, add a comment here with as much detail - when I work on the next version then I will consider them.

If you want to see a feature implemented it, add a comment with the feature number and a +1

Getting error: Write-Error "Redgate SQLRelease Module missing."

Do you want to request a feature or report a bug?

What is the current behavior?

**If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via http://sqlfiddle.com/

What is the expected behavior?

Which versions of SQL Server and which OS are affected by this issue? Did this work in previous versions of our procedures?

dotnet core tool published

Is your feature request related to a problem? Please describe.
Could we get the new SQLCoverCore packed and published as a nuget dotnet core tool?

===
Please at me @GoEddie so I receive a notification

xp_readerrorlog in SqlTraceController.Start() fails in SqlLocalDb

For SqlLocalDb, Start() fails with following exception, while running xp_readerrorlog.

EXEC xp_readerrorlog 0, 1, N'Logging SQL Server messages in file'
Msg 22004, Level 16, State 1, Line 3
Failed to open loopback connection. Please see event log for more information.
Msg 22004, Level 16, State 1, Line 3
Error log location not found.

This could easily be avoided by providing a default location that SqlServer can write trace/logs to.

[bug] Database Triggers Cause Exception

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Exception calling "Stop" with "0" argument(s): "Unable to cast object of type 'System.DBNull' to type 'System.String'."

How to reproduce:
run sqlcover against a database without database level triggers, it will succeed. Add 1 database trigger to the database and sqlcover will fail.

I took your script from the encryption error issue and found that Database Level Triggers do not have object_names pulled from the query you provided.

SELECT object_id, '[' + object_schema_name(object_id) + '].[' + object_name(object_id) + ']' as object_name, definition, uses_quoted_identifier from sys.sql_modules where object_id not in (select object_id from sys.objects where type = 'IF')

This is when using the covering other stuff scripts.

$coverage = new-object SQLCover.CodeCoverage($connectionString, $database, $true, $false)
$coverage.Start() 
#do stuff here
$coverageResults = $coverage.Stop()

Trigger coverage

Hi,

I just started using tSQLt and will be very helpful if I can build coverrage to the test. Thanks for your helpful tool and articles!

But I seems found a bug, SQLCover doesn't seems to cover logic in a trigger

How to reproduce

  • Have a table A with insert trigger T
  • Fake table A
  • Apply trigger T to faked table A
  • Create test to perform insert to table A, trigger T is executed
  • Run SQLCover

Expected behavior:
SQLCover reports code coverage in trigger T

Actual behavior:
SQLCover shows no code coverage in trigger T

This reproduction steps are executed in SQL Server 12.0.4100.1

Thanks
Hendra

Download Problems and Instructions on Generating Cobertura Report

@GoEddie I'm new to this project, and have had several issues getting it working.

First, this link just downloads a file "download.php", and not anything usable.
http://the.agilesql.club/SQLCover/download.php

Because of that, I've been using the files in src/SQLCover/releases/template as the executables. Are these even up to date? There are no instructions on how to build the project ...

Using the files in src/SQLCover/releases/template, I was able to get a coverage report generated, but I need a coverage report in Cobertura format. I can't find any examples or instructions on how to generate a Cobertura report.

Thanks for your help! I'm excited by what my company may be able to do with this project.

Exception calling "Cover" with "1" argument(s) - also "2" arguments.

Do you want to request a feature or report a bug?
bug or help

What is the current behavior?

I am getting the following when calling Get-CoverTSql. Where is the debug output?

Exception calling "Cover" with "1" argument(s): "Unable to start the trace - errors are recorded in the debug output"
At C:\Users\hoffman\Documents\Apps\SQLCOver.0.4.1\SQLCover.ps1:19 char:2
+     $coverage.Cover($query)
+     ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : SqlCoverException

Also tried using Get-CoverExe and is got an error message from OSQL.EXE indicating that I didn't set up the arg list correctly. The relevant line is commented out below.

Error: No user selected. Try with -U or -E switches
You cannot call a method on a null-valued expression.
At C:\Users\hoffman\Documents\Apps\SQLCOver.0.4.1\SQLCover.ps1:89 char:5
+     $result.OpenCoverXml() | Out-File $xmlPath
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : InvokeMethodOnNull

Server version:
Microsoft SQL Server 2017 (RTM-CU15) (KB4498951) - 14.0.3162.1 (X64) May 15 2019 19:14:30 Copyright (C) 2017 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2012 R2 Datacenter 6.3 (Build 9600: ) (Hypervisor)

**If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via http://sqlfiddle.com/

# file: test.ps1 
$ErrorActionPreference = "Stop"  

cd c:\Users\hoffman\Documents\Apps\SQLcover

# https://the.agilesql.club/2016/04/sqlcover-code-coverage-for-sql-server-t-sql/
$coverPath='C:\Users\hoffman\Documents\Apps\SQLCOver.0.4.1'
$coverDLL="$coverPath\SQLCover.dll"
$coverPS1="$coverPath\SQLCover.ps1"
# osql and osqlArgs used for Get-CoverExe
$osql='C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Binn\OSQL.EXE'
$server='GTR_DEV16'
$database='GTR'
$SQLuser='gtr_write'
$SQLpass='removed'  # 
$query="EXEC tSQLt.Run 'zut_clinvar_sub_curation.Test_update_sub_date'"
$connectionString = "Persist Security Info=False;User ID=$SQLuser;Password=$SQLpass;Initial Catalog=$database;Server=$server"
$osqlArgs="-S $server -d $database -U $SQLuser -P $SQLpass -i test.sql"
. $coverPS1
$result = Get-CoverTSql "${coverDLL}" "${connectionString}" "${database}" "${query}"
#$result = Get-CoverExe "${coverDLL}" "${connectionString}" "${database}" "${osql}" "${osqlArgs}"
Export-OpenXml $result "testout.xml"

What is the expected behavior?
Not receiving an error message

Which versions of SQL Server and which OS are affected by this issue? Did this work in previous versions of our procedures?

Microsoft SQL Server 2017 (RTM-CU15) (KB4498951) - 14.0.3162.1 (X64) May 15 2019 19:14:30 Copyright (C) 2017 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2012 R2 Datacenter 6.3 (Build 9600: ) (Hypervisor)

Client desktop computer: Windows 10 Enterprise
Version 1806 (OS Build 17134.1006)

First time attempting to use SQLCover.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.