azure / azure-kusto-go Goto Github PK

Title says it all. Looking through the library source code it looks like private link should work, but I'm wondering if it has ever been tested.

init method was missed when using MsiAuthentication way to generate TokenCredential in ./kusto/kcsb.go file, it should be added like this:

case kcsb.MsiAuthentication:
    init = func(ci *CloudInfo, cliOpts *azcore.ClientOptions, appClientId string) (azcore.TokenCredential, error) {
	    opts := &azidentity.ManagedIdentityCredentialOptions{}
	    if kcsb.ClientOptions != nil {
		    opts.ClientOptions = *kcsb.ClientOptions
	    }
	    if !isEmpty(kcsb.ManagedServiceIdentity) {
		    opts.ID = azidentity.ClientID(kcsb.ManagedServiceIdentity)
	    }
    
	    cred, err := azidentity.NewManagedIdentityCredential(opts)
	    if err != nil {
		    return nil, fmt.Errorf("error: Couldn't retrieve client credentials using Managed Identity: %s", err)
	    }
	    tkp.tokenCred = cred
    }

if init==nil, it won't call setInit() and tokenWrapper(),

and the scope is assigned a value in the tokenWrapper method: scopes := []string{fmt.Sprintf("%s/.default", resourceURI)},

without assigning a value to the scopes, you will end up with an error when calling the GetToken method in managed_identity_credential.go

func (c *ManagedIdentityCredential) GetToken(ctx context.Context, opts policy.TokenRequestOptions) (azcore.AccessToken, error) {
	if len(opts.Scopes) != 1 {
		err := errors.New(credNameManagedIdentity + ": GetToken() requires exactly one scope")
		return azcore.AccessToken{}, err
	}
	...
}

Original issue: #10

When the Kusto client logs messages, it uses the log.Print/Fatal(..) logger.

Investigate the option of providing a custom log.Logger that the package would use instead of the built in logger.

I want to convert management query results to struct.

simplified example:

type Result struct {
	Command string `kusto:"Command"`
}

code:

iter, err := client.Mgmt(ctx, databaseName, kusto.NewStmtUnsafe(".show tables | project Command=\"some command\""))
if err != nil {
panic(err)
}

defer iter.Stop()

var recs []Result
err = iter.Do(
func(row *table.Row) error {
	rec := Result{}
	if err := row.ToStruct(&rec); err != nil {
		return err
	}
	recs = append(recs, rec)
	fmt.Println(recs)
	fmt.Println("recs length is", len(recs))
	return nil
},
)
if err != nil {
  fmt.Println("ERROR", err.Error())
  panic(err)
}

By running the KQL:

.show tables | project Command="some command"

Let's say I have 3 tables, so I get 3 lines of "some command".

Code output:

[{some command}]
recs length is 1
[{some command} {some command}]
recs length is 2
[{some command} {some command} {some command}]
recs length is 3
[{some command} {some command} {some command} {{"Visualization":null,"Title":null,"XColumn":null,"Series":null,"YColumns":null,"AnomalyColumns":null,"XTitle":null,"YTitle":null,"XAxis":null,"YAxis":null,"Legend":null,"YSplit":null,"Accumulate":false,"IsQuerySorted":false,"Kind":null,"Ymin":"NaN","Ymax":"NaN"}}]
recs length is 4
ERROR Op(OpMgmt): Kind(KClientArgs): row does not have the correct number of values(10) for the number of columns(1)
panic: Op(OpMgmt): Kind(KClientArgs): row does not have the correct number of values(10) for the number of columns(1)

It seems like it tries to convert all received rows including the metadata, statistics etc and tries to convert them to the desired struct, which fails.

When ingesting data using the ingest.IngestionMappingRef option, the mapping's data type should be used as default data type instead of CSV.

As an example, relying on the ingest.IngestionMappingRef("ExampleMapping", ingest.JSON) option should change the default data type from CSV to JSON, resulting in the ingest.IngestionMappingRef not relying on another ingest.FileFormat option.

It felt counter-intuitive to specify the format twice.

It looks like the progressive query option, which is on by default, can't be used in cross-cluster queries. These queries will fail with a bad request error, saying:

Errors occurred while resolving remote entities. Failed to resolve entityNameOrPattern='xxx' in one or more scopes: ($Cluster='https://xxx.kusto.windows.net/', Database='xxx')

Putting this in docs may save hours of debugging...

Following error occurs when I execute a query with make-series clause.

Kind(KInternal): in row 0: column sum_value, conversion error: Column with type 'dynamic' was not stored as a string, was []interface {}

Example query:

 metrics 
| where uuid == 'obs111bd266-cc3d-434f-99f1-122bdb53e16d'
| where callerConnectorType == 'ballerina/http/Caller'
| where timeSeriesId == 'metric_requests_total'
| make-series sum(value) on timestamp in range(datetime(2020-05-05), datetime(2020-05-07), 2h) by httpStatusCodeGroup

As I understood its because the generated column sum_value is of dynamic type. But it should be supported by the SDK and I believe this is a bug.
Further, I am able to execute the same query using both REST API and Azure query UI without any issue.

What's checked in isn't buildable.
query_builder.go includes it as:
ilog "github.com/Azure/azure-kusto-go/kusto/internal/log"
and calls it in an (unused) function called UnsafeStmt.

We are getting a weird error from the SDK:

Op(OpIngestStream): Kind(KHTTPError): streaming ingest issue(404 NotFound):\n\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0004\u0000UO�j�0\f�\u000f�\u000f�纤K�,�e0h\u0019�\u001d�/�-�\u0018Z{���Q��s��\u0016���\u001e��t��S�4�}����L����T ��d���z��<Rΰ\u001f�O�\t�!�Ui�\u0012RT\u0014�+�(*d�!�;\u0010.g\u0002/>E���M'�C�(;�\u001e��\u001b�2�o_�t&?\b=��\u001dvT\u0019v\u0015�Ƣ5�e6\u000e-u�n+��e\u001d5�K8\u0005���:X9t���l�i\u001e��\u0000se��\u0019\u001ar������Nk\u0019�_߈\u0016\u000b1\u0001\u0000\u0000

We believe this is a "Streaming is disabled" error. Could this error be changed to return something that we can actually read?

Decimal values can be just big integers (rather than floating points, i.e without the .) when they are returned by the REST api, and the dceRE expects something with dot
var decRE = regexp.MustCompile(`^\d*\.\d+$`)
which fails parsing valid kusto responses

In addition it might be a good idea to support *big.Int as well as *big.Float in that case

Hi

We are writing a client that ingest data into ADX using this golang library, we created in Azure portal an ADX cluster & database and in the client we run the following queries, first to create table and ingestion setting and then ingest data and query the ingest row size.

1. kusto.Client.Mgmt(..., ".create table [Table Name] (col1: type1, ....)")

2. kusto.Client.Mgmt(..., ".add table [Table Name] ingestors ('...;...') '...'")

3. kusto.Client.Mgmt(..., ".create-or-alter table [Table Name] ingestion json mapping [mapping name] '[' ' { "column" : "col1", "datatype" : "type1", "Properties":{"Path":"$.col1"}} ', '...' ']'")

4. kusto.Client.Mgmt(..., ".alter table [Table Name] policy streamingingestion '{ "NumberOfRowStores": 1 }'")

5. ingest.New(kusto.Client, [Database Name], [Table Name]).Stream(ctx, data, ingest.JSON, "[mapping name]")

6. kusto.Client.Query(..., ", [Table Name] | count")

7. kusto.Client.Mgmt(..., ".drop table [Table Name] ifexists")

When I run all commands in the following sequence one by one, all return successfully (no errors) but the count query returns 0 and in ADX portal as well the new table is empty.

Only if I wait after the management queries (step 1 to 4) for 5 minutes and then do the ingest data, then new table is not empty and count query return number > 0

Why do I need this long delay between table creation and table ingestion and can it be decreased by client/server ?

Also why do the ingest method not return an error in case of no delay, where in reality no rows were inserted into the table.

Thanks
Raanan

One of the dependencies has a vulnerability CVE-2022-27191. Can this be updated to get the patched version?
The vulnerability is specifically in the module autorest/azure/auth

Hi,

I need to add queryOptions to my query. The problem is that nothing is exported and I see that I could use the functions in the file queryopts.go, only that for some reason most if it is commented. For example the function CustomQueryOption. Can you please make it useable?

I got the following panic while trying to query data. Let me know on Teams or email if you need more specific details (my alias is jasch). It doesn't seem to always happen as I'm unable to reproduce it consistently. The kusto.ResultsProgressiveDisable() is being used on the query that causes the panic.

panic: interface conversion: interface {} is []interface {}, not map[string]interface {}

goroutine 210 [running]:
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.rawToOneAPIErr(0xc01ee00000, 0x43bb7d4, 0x7fffe00, 0x7d0000, 0xc0152cb4d0, 0x8b5320)
		/home/jason/.go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/v2.go:217 +0x17f
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*DataTable).UnmarshalRaw(0xc0152cb4d0, 0xc01ee00000, 0x43bb7d4, 0x7fffe00, 0x0, 0x0)
		/home/jason/.go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/v2.go:61 +0x14f
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).decode(0xc0000fa040, 0x8bb868, 0xc009e57980, 0xc0129fe000, 0x0, 0x0)
		/home/jason/.go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:117 +0xb1e
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).decodeFrames(0xc0000fa040, 0x8bb868, 0xc009e57980, 0xc0129fe000)
		/home/jason/.go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:83 +0x85
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).Decode.func1(0x8b7ca0, 0xc0000fe840, 0xc0129fe000, 0xc0000fa040, 0x8bb868, 0xc009e57980)
		/home/jason/.go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:60 +0x3cc
created by github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).Decode
		/home/jason/.go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:33 +0x147

Provide ability to access the response that contains other table such as @ExtendedProperties, and QueryResourceConsumption
this is very important for using the kusto database cursor feature which emits the Cursor property on the @ExtendedProperties table
Another interesting aspect is getting the performance metadata on the QueryResourceConsumption for every executed query.
I saw that there was an unused, unexported function named getNonPrimary which hints that you were going to export this feature, currently I just forked and exported it.

there's a spot in conn.go where we initialize a request, and this is where the ADAL libraries actually do the work to get us our token. If we've got invalid credentials, the request returned will be nil, and when we pass that in to autorest, we can trigger a segfault.

my code like this

	ctx := context.Background()
	iter, err := client.Query(ctx, datababe, kusto.NewStmt(tableName))
	if err != nil {
		panic("add error handling")
	}
	defer iter.Stop()
	info, err := iter.GetQueryCompletionInformation()
	fmt.Println(info)
	fmt.Println(info.Rows...)
	_, _, err2 := iter.NextRowOrError()
	fmt.Println(err2)

I got info by GetQueryCompletionInformation :

{{DataTable} 2 QueryCompletionInformation QueryCompletionInformation [{Timestamp datetime} {ClientRequestId string} {ActivityId guid} {SubActivityId guid} {ParentActivityId guid} {Level int} {LevelName string} {StatusCode int} {StatusCodeName string} {EventType int} {EventTypeName string} {Payload string}] [] [[2022-10-20T07:03:21.5530279Z KGC.execute;6fc43e22-5046-469f-865f-cf5b05754740 653274e5-da25-4701-a169-d55eb5d27e90 52ddc8c8-acde-4c39-8092-bb680d9f500c 2ac265a2-e9cb-46a7-8d83-2a54e78ddf52 4 Info 0 S_OK (0) 4 QueryInfo {"Count":1,"Text":"Query completed successfully"}] [2022-10-20T07:03:21.7413073Z KGC.execute;6fc43e22-5046-469f-865f-cf5b05754740 653274e5-da25-4701-a169-d55eb5d27e90 52ddc8c8-acde-4c39-8092-bb680d9f500c 2ac265a2-e9cb-46a7-8d83-2a54e78ddf52 4 Info 0 S_OK (0) 5 WorkloadGroup {"Count":1,"Text":"default"}] [2022-10-20T07:03:21.7413073Z KGC.execute;6fc43e22-5046-469f-865f-cf5b05754740 653274e5-da25-4701-a169-d55eb5d27e90 52ddc8c8-acde-4c39-8092-bb680d9f500c 2ac265a2-e9cb-46a7-8d83-2a54e78ddf52 6 Stats 0 S_OK (0) 0 QueryResourceConsumption {"ExecutionTime":0.4374625,"resource_usage":{"cache":{"memory":{"hits":441,"misses":0,"total":441},"disk":{"hits":0,"misses":0,"total":0},"shards":{"hot":{"hitbytes":0,"missbytes":0,"retrievebytes":0},"cold":{"hitbytes":0,"missbytes":0,"retrievebytes":0},"bypassbytes":0}},"cpu":{"user":"00:00:00","kernel":"00:00:00","total cpu":"00:00:00"},"memory":{"peak_per_node":6292560},"network":{"inter_cluster_total_bytes":1299903,"cross_cluster_total_bytes":0}},"input_dataset_statistics":{"extents":{"total":23,"scanned":23,"scanned_min_datetime":"2022-08-31T06:47:15.7381112Z","scanned_max_datetime":"2022-10-20T04:03:04.9728076Z"},"rows":{"total":380,"scanned":380},"rowstores":{"scanned_rows":0,"scanned_values_size":0},"shards":{"queries_generic":0,"queries_specialized":0}},"dataset_statistics":[{"table_row_count":380,"table_size":387214}],"cross_cluster_resource_usage":{}}]] [] OpQuery}

It's obviously that this request accesses the database , and 380 rows of data were scanned.

But info.Rows... is empty and i got an err in _, _, err2 := iter.NextRowOrError(), err2 is EOF

Azure core & azure identity (and maybe more) recently went to 1.x, while we are still at 0.x.
https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azcore/go.mod
https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azidentity/go.mod

A user had an issue with using these new packages in conjunction with our code.

Moving to the new packages is actually a breaking change - since they require a minimum version of go 1.18

So I've got something of a tricky problem. I'm getting an error that can only be reproduced with a combination of azure-kusto-go and a kusto plugin command that's in private preview. If I run the query through kusto explorer, it works fine. And if I run most non-plugin queries through my azure-kusto-go client, it works fine. Even some queries with the plugin via the azure-kusto-go client work, just not all of them.

The queries don't seem particularly complicated, they're basic heartbeat checks of the following form:

let mt = evaluate redacted_plugin_request('redacted_account_parameter', @'redacted_kqlm_parameter', ago(35m));
mt
| where Region in ("westus2")
| summarize percentHealthy=todouble(count())

When one of these queries breaks, it looks like this:

Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint(520 ):
{
  "error": {
    "code": "Internal service error",
    "message": "Request aborted due to an internal service error.",
    "@type": "System.IndexOutOfRangeException",
    "@message": "Index was outside the bounds of the array.",
    "@context": {
      "timestamp": "2020-10-06T23:09:50.4300781Z",
      "serviceAlias": "redacted",
      "machineName": "KEngine000035",
      "processName": "Kusto.WinSvc.Svc",
      "processId": 4836,
      "threadId": 9492,
      "appDomainName": "Kusto.WinSvc.Svc.exe",
      "activityStack": "(Empty activity stack)"
    },
    "@permanent": false
  }
}

I raised the issue with the team who built the plugin, they asked me to talk to the kusto team. I raised the issue to the kusto team, and I have a suspicion that they're going to ask me to raise it with the team behind the client I'm using, so I figured I'd parallelize.

This kind of error look familiar at all? Any chance I'm, like, somehow using the client incorrectly in a way that can be easily fixed?

The function DoOnRowOrError(f) returns an error "json: cannot unmarshal object into Go struct field DataSetCompletion.OneApiErrors of type string"
this happens intermittently, most likely when there is a kusto server side error
My callback (the parameter f) never returns an error there, I don't have the raw response but if I had to guess I would assume that the server does not return a []string (the type of OneApiErrors) but a single string instead.

Documentation says:

The value of the kth column will be decoded into the kth argument to Columns

but actually the column names are decoded into the arguments.

Queries may return table fragments of type DataReplace as per docs, which are not correctly handled. The result set may contain duplicated entries.

I have some management statements that I want to execute through the client.
These statements are built with fmt.Sprintf.

 command := fmt.Sprintf(...)
if _, err := client.Mgmt(ctx, db, command); err != nil {
	return err
}

So I saw the query is supposed to be of type Stmt, but I didn't find how to convert this string to Stmt.

When using NewStmt, I can only use a raw string as input, but not a predefined one:

working := kustoSDK.NewStmt("some command")
query := "some command"
notWorking := kustoSDK.NewStmt(query)

Am I missing something? How can I fix this issue?

I'm using azure-kust-go version 0.1.3 and go version go1.13.12 darwin/amd64

I want to set SourceMessageCreationTime option in my ingestion process that using (in this case) FromFile.

Seems like this option exists, but not exported (correct me if I'm wrong).

Why?
I want to override creationTime, the data shard's creation to make sure retention and caching policies are applied (to what I specify in the creationTime).

Using go1.16 and github.com/Azure/azure-kusto-go v0.3.2

Is there a plan for this package to be migrated from ADAL to MSAL? The autorest auth packages used for authentication here are ADAL-based and there is no plan for those to upgrade to MSAL (ref: Azure/go-autorest#636). The official guidance is to switch over to using github.com/Azure/azure-sdk-for-go/tree/main/sdk/azidentity which is MSAL-based.

Are you not able to query using a service principal? Or do you need to create it with specific permissions?

In current version of SDK a way to set client Version (similar to java sdk's ConnectionStringBuilder.setClientVersionForTracing) is missing
API can be implemented to set the client version so that it can be used for tracing and for connector/application specific stats which could be helpful for tracking clients' usage.

One of the dependencies in this package (github.com/satori/go.uuid) has a vulnerability that dates back quite a while with no published version that has a fix. The repository for that package appears to be abandoned. Here's the CVE: https://snyk.io/vuln/SNYK-GOLANG-GITHUBCOMSATORIGOUUID-72488

Is it possible to upgrade to a different package? I see that there is also a dependency on github.com/google/uuid here. As another reference point, the Azure SDK for Golang chose to move to github.com/gofrs/uuid when faced with the same issue: Azure/azure-sdk-for-go#14283

The Timespan type is a Go wrapper around time.Duration, which must convert to the string format of .Net's Timespan's type. Like any other .Net thing, its overly complicated and supports multiple string layouts. Also, like other .Net types, it defines it's own base type called a "tick" that no one else uses. A tick is 10 millionth of a second. Because of these special types and layouts, time.Time.Format() won't work.

Because there are multiple Timespan output formats, I'm not sure if I based mine on a Timespan format that supported nanosecond, or the parsers were more forgiving or what. The backend was honoring it, probably by chopping off the excess integers. But some change to the backend started making it not accept the values if the precision was more than a tick.

We need to change the marshal to remove precision of more than a tick. We can keep Unmarshal's nanosecond accuracy, as it will have no effect on us.

Hello,

I'm not sure whether this is a bug, or enhancement (missed use case?)- but, it seems that the library doesn't support HTTP connections. Why would I want HTTP? I'm wanting to use Kustainer in a test framework... Kustainer runs local (docker) and exposes HTTP.

Now, as it stands, I can't directly connect to it using HTTP, in the connection string... If I change the connection string prefix to 'https', I can connect. The problem is that, once connected, calling a Mgmt query fails with Op(OpServConn): Kind(KClientArgs): endpoint is not valid(http://127.0.0.1:8080), should be https://<cluster name>.*.

conn.go:46 checks that the endpoint matches regexp.MustCompile("https://([a-zA-Z0-9_-]+\.){1,2}.*").

Could we allow HTTP connections? Kustainer is too good of a testing tool to give up.... and I'd rather not work around you.

When ingesting logs to kusto in usgovvirginia region, I got the following error.
ingest.New client failed with error: problem reading ingestion resources from Kusto: the StorageRoot URI received(https://***.blob.core.usgovcloudapi.net/2022*******0Z&sp=rw) has an error: URI hostname does not end with '.core.windows.net'

Could we add interface for adding trusted endpoints? Just like in this doc: https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/connection-strings/kusto#trusted-endpoints

By default Kusto queries are limited to some number of return values (I don't remember what the value is).

So if you use the SDK to make a query and you reach that threshold, you will receive:

(panic: json: cannot unmarshal object into Go struct field DataSetCompletion.OneApiErrors of type string)

This means the error message being returned by Kusto cannot be converted from its JSON representation to whatever error format we have.

This is probably related to:
#12

I remember that being a mess I wanted to correct, but don't remember all the details and haven't gone back over it.

But this should be returning that limit error instead of this internal JSON decode error about decoding an error.

Summary

Recently (2022-10-24) queries to our Kusto cluster have stopped parsings result into the kusto.RowIterator object. Sending the same query using the QueryToJson(..) function does assert that the appropriate rows are being returned from the API. Which hints that this is a parsing problem within the azure-kusto-go library. Tests using these same DB+query using the azure-kusto-python library, were successful as well. So the Python lib could handle the oddity that our server is returning.

Note I have full request/response payload available and can share over our common MS Teams channel (I am a GitHub employee). Feel free to connect over Teams if you'd like to chat more.

kusto.Query(context.Background(), "<REMOVED>", kusto.NewStmt("ActivityLogs | project ResourceId | take 1"))

raw REQUEST payload for the above example. By default the following is set "results_progressive_enabled":true, since the query is of type errors.OpQuery. This is expected.

POST /v2/rest/query HTTP/1.1
Host: <REMOVED>.kusto.windows.net
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Accept: application/json
Authorization: Bearer <REMOVED>
Content-Type: application/json; charset=utf-8
X-Ms-Client-Request-Id: KGC.execute;<REMOVED>
X-Ms-Client-Version: Kusto.Go.Client: 0.8.1
Accept-Encoding: gzip

de
{"db":"<REMOVED>","csl":"ActivityLogs | project ResourceId | take 1","properties":{"Options":{"results_progressive_enabled":true,"servertimeout":"00:03:59.9999963"},"Parameters":null,"Application":"","User":""}}

0

The RAW response payload is provided below, with some pieces removed. But it does show that we have ROW array populated. What's also interesting is that our Kusto cluster is reporting "IsProgressive":false even though the client set results_progressive_enabled=true. The server is also returning TableProgress frames in this non progressive response. I am not entirely sure if this is intentional, or a problem in Kusto server itself. The end result is that the kusto.RowIterator object is empty and raises (EOF)

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/json
Date: Wed, 26 Oct 2022 16:51:52 GMT
Server: Kestrel
Strict-Transport-Security: max-age=2592000; includeSubDomains
Vary: Accept-Encoding
X-Ms-Activity-Id: <REMOVED>
X-Ms-Client-Request-Id: KGC.execute;<REMOVED>

eff
[{"FrameType":"DataSetHeader","IsProgressive":false,"Version":"v2.0"},{"FrameType":"DataTable","TableId":0,"TableKind":"QueryProperties","TableName":"@ExtendedProperties","Columns":[{"ColumnName":"TableId","ColumnType":"int"},{"ColumnName":"Key","ColumnType":"string"},{"ColumnName":"Value","ColumnType":"dynamic"}],"Rows":[[1,"Visualization","{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"AnomalyColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null,\"Ymin\":\"NaN\",\"Ymax\":\"NaN\",\"Xmin\":null,\"Xmax\":null}"]]},{"FrameType":"TableHeader","TableId":1,"TableKind":"PrimaryResult","TableName":"PrimaryResult","Columns":[{"ColumnName":"ResourceId","ColumnType":"string"}]},{"FrameType":"TableFragment","TableFragmentType":"DataAppend","TableId":1,"Rows":[["<VALUE REMOVED>"]]},{"FrameType":"TableProgress","TableId":1,"TableProgress":0.0},{"FrameType":"TableCompletion","TableId":1,"RowCount":1},{"FrameType":"DataTable","TableId":2,"TableKind":"QueryCompletionInformation","TableName":"QueryCompletionInformation",
... REMOVED ...
{"FrameType":"DataSetCompletion","HasErrors":false,"Cancelled":false}]
0

Work-around

I was able to fix this behaviour in two ways so far.

• As a test, I modified https://github.com/Azure/azure-kusto-go/blob/master/kusto/kusto.go#L236-L253 to always force the progressiveSM{} statemachine object. My reasoning was that the Kusto V2 Spec hints that TableProgress is the result of a "progressive" frame. Even through our Kusto server is indicating the response payload is Frame V2 , but "IsProgressive":false though weirdly providing "progressive" frames in our response anyway. I wonder if our Kusto server is misconfigured, or some other state has drifted.
• Instead I built my query with kusto.ResultsProgressiveDisable() as such -> kusto.Query(context.Background(), "<REMOVED>", kusto.NewStmt("ActivityLogs | project ResourceId | take 1"), kusto.ResultsProgressiveDisable()). The hope here was to stop the server from responding with "progressive" FrameTypes (when it says it shouldn't). This also worked, and didn't require further hacking of the lib to support.

Is the response payload that includes {"FrameType":"DataSetHeader","IsProgressive":false,"Version":"v2.0"} coupled with {"FrameType":"TableProgress","TableId":1,"TableProgress":0.0} (progressive tables) an allowed response from a Kusto server?

• If so, there may be a bug in the nonProgressiveSM parsers.
• If not, why is my Kusto server doing this ;-)

Management commands such as .create-or-alter can inaccurately fail with the error:

"Mgmt() attempted to do a write operation. This requires the AllowWrite() QueryOption to be passed. Please see documentation on that option before use"

If a command such as .create-or-alter contains any of the following strings it will fail unless AllowWrite() QueryOption is passed

• .set
• .append
• .set-or-append
• .set-or-replace

Example

.create-or-alter table MyTable ingestion json mapping "Mapping1"
'['
'    { "column" : "setid_dept", "datatype" : "string", "Properties":{"Path":"$.setid_dept"}},'
'    { "column" : "setid_supv_lvl", "datatype" : "string", "Properties":{"Path":"$.setid_supv_lvl"}}'
'    { "column" : "appended_results", "datatype" : "string", "Properties":{"Path":"$.appended_results"}}'
']'

.create-or-alter is not a "data modifying" command and should not require the AllowWrite() QueryOption to be passed.

However since the query contains $.setid_supv_lvl the RegEx matches on ".set" and throws the error.

Issue Location

/kusto/conn.go

The following regex matches the "data modifying commands" anywhere in the query string instead of only at the beginning

var writeRE = regexp.MustCompile((\.set|\.append|\.set-or-append|\.set-or-replace))

I've been taking a look at this library to see if it'll work for a project I'm working on at the moment, and I've been having some trouble getting the samples to build successfully. I'll copy over some of the changes I've been making once I can get it to build, would you mind if I made a PR to integrate some of this into the repo? I was thinking it might be helpful to move the sample code into actual go source files, maybe put together a low-key e2e test for verifying their continued correctness?

Running the Mgmt(..) query .show database Samples schema returns the error DataTable had column of type System.String, which was unknown.

I am seeing this error on a number of databases I have been developing against and tests against the Samples demonstration database also return this error.

The following code can reproduce.

package main

import (
	"context"
	"fmt"
	"io"
	"os"

	"github.com/Azure/azure-kusto-go/kusto"
	"github.com/Azure/go-autorest/autorest/azure/auth"
)

const (
	showTablesKql    = ".show tables | project TableName" // good query..
	showDbKql        = ".show database Samples schema"    // bad query
	dbName           = "Samples"                          // via https://help.kusto.windows.net
	connectionString = "https://help.kusto.windows.net"
)

func check(err error, where string) {
	if err != nil {
		fmt.Fprintf(os.Stderr, "ERROR: %s -> %v\n", where, err)
		os.Exit(1)
	}
}

func main() {
	azCreds, err := auth.NewAuthorizerFromCLIWithResource(connectionString)
	check(err, "authorizer")

	authorizer := kusto.Authorization{Authorizer: azCreds}
	client, err := kusto.New(connectionString, authorizer)
	check(err, "client")

	rows, err := client.Mgmt(context.Background(), dbName, kusto.NewStmt(showDbKql))
	check(err, "mgmt")

	for {
		row, err := rows.Next()
		if err != nil {
			if err == io.EOF {
				check(err, "eof")
			}
			check(err, "row")
		}
		fmt.Printf("%#v\n", row)
	}
}

Running the above returns

go run main.go 
ERROR: row -> DataTable had column of type System.String, which was unknown
exit status 1

Note that if you swap to another working query (eg showTablesKql), it confirms parsing of other mgmt commands are successful.

Description
cannot use time.Now() as parameters' default value;
such as

 timeFilter, err := kusto.NewDefinitions().With(
	kusto.ParamTypes{
		"beginTime": kusto.ParamType{Type: types.DateTime, Default: time.Now()},
		"endTime":   kusto.ParamType{Type: types.DateTime, Default: time.Now()},
		"name":      kusto.ParamType{Type: types.String, Default: ""},
		"mess":      kusto.ParamType{Type: types.String, Default: ""},
	},
)

it can be send to kusto, but will get a 400 bad request.

"@message": "Syntax error: SYN0002: A recognition error occurred. [line:position=1:50]",

Affected Modules, Packages, Versions and Symbols

Module: github.com/example/module
Package: github.com/example/module/package
Versions:

• Introduced: 1.2.0
• Fixed: 1.2.4
  Symbols:
• aFunction
• SomeType.AMethod

Module: github.com/example/module/v2
Package: github.com/example/module/v2/package
Versions:

• Fixed: 2.4.5
  Symbols:
• anotherFunction

Ingestion in the Go SDK use ".show ingestion mappings" to validate that calls that rely on Ingestion references stored on the servers actually exist.

When doing thousands of individual ingestions, this was causing thousands of these calls, which aren't designed to scale for this purpose.

The API is supposed to cache the response for 5 minutes. However, due to a bug, we were not updating the last write times and we didn't have a test or the cache part.

This section of code was schedule to be removed for other non-related reasons and to be replaced with Ingestion status calls when a user elects to check the status instead of all the time as is done here.

As we are at the verge of adding that code in, I am going to strip this code for a non-breaking code change.

In result.go

// Wait returns a channel that can be checked for ingestion results.
// In order to check actual status please use the IngestionStatus option when ingesting data.
func (r *Result) Wait(ctx context.Context) chan error {

"IngestionStatus" is incorrect. Should be "ReportResultToTable"

In kusto_examples_test.go

// Create an authorizer with an Azure MSI (managed identities).
msi, err := auth.NewMSIConfig().Authorizer()
if err != nil {
	panic("add error handling")
}

authorizer := Authorization{
	Authorizer: msi,
}

// Normally here you take a client.
_, err = New("endpoint", authorizer)
if err != nil {
	panic("add error handling")
}

This won't work, because we are setting an authorizer without a config. If you do this, you have to set the resource.

This should be:

// Create an authorizer with an Azure MSI (managed identities).
msi := auth.NewMSIConfig()

authorizer := Authorization{
	Config: msi,
}

// Normally here you take a client.
_, err = New("endpoint", authorizer)
if err != nil {
	panic("add error handling")
}

We recommend using the Config instead of Authorizer because of the whole resource not getting set.

I think there was a reason for us to not set it in the Authorizer (maybe it was so we had an escape hatch for some weird corner case??).

But we should try to check to see if it has a resource set. Baring that, we should add output to the error so it says:
"DID YOU SET THE RESOURCE IN YOUR AUTHORIZER???".

The deny message from Kusto will not tell you that your resource was not set.

Looks like v2.rawToOneAPIErr expects that errors are of type map[string]interface {} but actually an array of interfaces is returned...

panic: interface conversion: interface {} is []interface {}, not map[string]interface {}

goroutine 46 [running]:
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.rawToOneAPIErr({0xc000166000, 0x3bdd, 0x3e00}, 0xe800)
        /home/diweng/go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/v2.go:217 +0x113
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*DataTable).UnmarshalRaw(0xc0001142d0, {0xc000166000, 0x3bdd, 0x3e00})
        /home/diweng/go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/v2.go:61 +0x127
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).decode(0xc000218380, {0x92a968, 0xc0002e2e40}, 0xc00005ae88)
        /home/diweng/go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:117 +0x81f
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).decodeFrames(0xc000218380, {0x92a968, 0xc0002e2e40}, 0x0)
        /home/diweng/go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:83 +0x66
github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).Decode.func1()
        /home/diweng/go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:60 +0x37e
created by github.com/Azure/azure-kusto-go/kusto/internal/frames/v2.(*Decoder).Decode
        /home/diweng/go/pkg/mod/github.com/!azure/[email protected]/kusto/internal/frames/v2/decoder.go:33 +0x1af
exit status 2

The query response is as follows

{
    "FrameType": "DataTable",
    "TableId": 1,
    "TableKind": "PrimaryResult",
    "TableName": "PrimaryResult",
    "Columns": ["..."],
    "Rows": ["...", {
            "OneApiErrors": [
                {
                    "error": {
                        "code": "Internal service error",
                        "message": "Request aborted due to an internal service error.",
                        "@type": "System.AggregateException",
                        "@message": "...",
                        "@context": {
                            "timestamp": "2022-03-16T11:29:08.7676558Z",
                            "serviceAlias": "...",
                            "machineName": "KENGINE000004",
                            "processName": "Kusto.WinSvc.Svc",
                            "processId": 5180,
                            "threadId": 3784,
                            "appDomainName": "Kusto.WinSvc.Svc.exe",
                            "clientRequestId": "KGC.execute;c6525dc6-b281-42b8-9250-0da3d7c2b3ef",
                            "activityId": "f924dcd6-3176-4b37-986c-80413bff1e9b",
                            "subActivityId": "f924dcd6-3176-4b37-986c-80413bff1e9b",
                            "activityType": "PO.OWIN.CallContext",
                            "parentActivityId": "f924dcd6-3176-4b37-986c-80413bff1e9b",
                            "activityStack": "(Activity stack: CRID=KGC.execute;c6525dc6-b281-42b8-9250-0da3d7c2b3ef ARID=f924dcd6-3176-4b37-986c-80413bff1e9b > PO.OWIN.CallContext/f924dcd6-3176-4b37-986c-80413bff1e9b)"
                        },
                        "@permanent": false
                    }
                }
            ]
        }
    ]
}

conn.go on line 252 logs to the console the results of the http response:

log.Printf("resp code %s, response body:\n%s", resp.Status, respBody)
return execResp{}, errors.E(op, errors.KHTTPError, fmt.Errorf("received unexpected HTTP code %s, response body:\n%s", resp.Status, respBody))

Logging is fine from a library, but never directly to the console like this. The Kusto api should define its own logging interface, output using that, and then developers can a pick canned kusto implementation of that (just using the stdlib 'log' for eg) or provide their own that logs to a file, ignores it, whatever.

Additionally, the main issue is that the actual Kusto error response is basically lost. All that's returned from a Kusto call is an HTTP Error with a text string containing the http response body embedded inside it.

It appears as though Kusto returns a json struct for the error with code, message, @type, @context, etc. Shouldn't there be a formal Kusto error struct that's optionally set and returned with errors returned by calls to Kusto Go APIs ?

This is an ask for help, Thanks.

The Go SDK autorest.Authorizer is commonly initialized with ClientID, TenantID, which means I need to create an app and authorize it to access kusto cluster. Besides, I guess Kusto itself is an application(Client) ?
However, I hope to access the kusto cluster directly with an authorized user account. I have check https://docs.microsoft.com/zh-cn/azure/data-explorer/kusto/management/access-control/how-to-authenticate-with-aad but fail to find a solution. Could you give me a more detaied infomation?

Thanks a lot.

It outputs

Error: Op(OpMgmt): Kind(KInternal): DataTable.Columns had entry with .ColumnType set to a <nil> type

on the screen but doesn't actually return an error. There are apparently still cases where Kusto returns aren't being translated properly.

This is with the 0.1 pre-release.

Ran into a strange issue earlier today. I'm not sure if the problem can be traced back to this library, or to kusto's API, so I figured I'd try getting in touch with both teams.

Two (slightly) different queries, upon examination through the kusto explorer, appear to return identical results. Specifically, they both return a single column named percentHealthy, and a single row storing a kusto Real. However, if we try to run those queries through azure-kusto-go, we get different results. The results of one of these queries will be converted successfully to a floating point value in Go, i.e. 94.343, whereas the result of the other query will return a blank floating point: 0.0000. When we traced the code, we found that both query operations acquired the same value for this row, but that one query appeared to return an object with an incorrect column name, just a bunch of alphanumerics.

The only difference in the queries is at the very end, where we project our final value.

The query that works with azure-kusto-go ends with this:

| extend percentHealthy=((1.0 * (total-(unhealthy)))/total)*100
| project percentHealthy

The query that doesn't work with azure-kusto-go ends like this:

| project percentHealthy=((1.0 * (total-(unhealthy)))/total)*100

Any idea where to look to dig into this?

Originally a part of issue: #10

Errors returned as part of Kusto can come in a variety of shapes. Some are errors generated by authorization systems, some are generated by Kusto, etc....

These errors may come in two forms

• Standard string
• JSON encoded

The JSON encoded may also come in different forms. Some are OneAPI errors, but off an earlier spec that is no longer valid or some other JSON encoded structure.

Currently, we simply try to look at the OneAPI errors we have seen and simply put them as wrapped errors inside Go's error type. This probably has limited use.

Go has a wide variety of error detection methods. However, these are all predicated on that the Go code knows what the errors are going to be. At this time, I don't think it is realistic to believe we can do that. We simply understand the severity of the issue and where the error comes from.

In those issues, we have error codes that can be introspected via the errors library.

But we do not have anything to allow dynamic introspection of the data when it is JSON encoded. This is being asked for.

Here is the original part of the request:

Additionally, the main issue is that the actual Kusto error response is basically lost. All that's returned from a Kusto call is an HTTP Error with a text string containing the http response body embedded inside it.

It appears as though Kusto returns a json struct for the error with code, message, @type, @context, etc. Shouldn't there be a formal Kusto error struct that's optionally set and returned with errors returned by calls to Kusto Go APIs ?

Last week, there was a new release (v1.0.0) of github.com/Azure/azure-sdk-for-go/sdk/storage/azblob. This release introduces some breaking changes on the public API with respect to v0.6.1, the version used by azure-kusto-go.

To allow using v1.0.0 of the azblob module on a Go module together with azure-kusto-go, we would need a new release that uses the new API.

Problem

I want to connect to kusto using system assigned identity. To do that I use the azidentity.NewDefaultAzureCredentials function from the azure-sdk-for-go package to get a valid access token (cred). To build the needed authorizer (s. kusto.New(...)) I utilze azidext.NewTokenCredentialAdapter function from azidext

cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
 // ...
}

authorizer := &kusto.Authorization{
        Authorizer: azidext.NewTokenCredentialAdapter(cred, []string{endpoint}),
    }

client, err := kusto.New(endpoint, authorizer)

Because the code of the Azure Identity Extension (azidext) is not officially supported or intended for production use I want to get rid of it. How can I do that? How can I create an authorizer from a valid token or create a new kusto client with that token?

I've checked newer version descriptions without finding a hint that this will work or how it works with any of the newer versions.

Versions used

github.com/Azure/azure-kusto-go v0.9.2
github.com/jongio/azidext/go/azidext v0.4.0

Looks like azure-kusto-go doesn't respect environment HTTP_PROXY (or HTTPS_PROXY) when ingest data to Kusto. We have some customers need uploading data to Kusto but client doesn't have direct access to internet and all outgoing traffic needs go through outbound proxy.