Currently the Get Methods require one to parse the entire json string before calling a method on the resultant object.

There may be ways to facilitate a more performant means of achieving the use case where one cares about operating on only the subset of data that Get* is concerned with.

One option is to explore whether it's faster to find an index in the tsv string and extract only the value from the tsv string directly. If so, an additional API method to do so might be a nice addition to the project.

See if we can add some of the other badges we see on eg. SQL-runner and GO tracker repos

Currently we can only get the entire object for unstruct_event, contexts or derived_contexts values in the Getter functions.

We could potentially leverage jsoniter's Get function to allow one to extract fields from the self-describing JSON without needing to process the entire object .

We might consider specific methods for this kind of Get if it becomes too complicated/confusing an interface. Especially considering that contexts are arrays of JSON.

There are a range of JSON libraries that may be faster for our project (eg. easyjson).

Additionally, using jsonparser to extract from self-describing data, and another package for marshaling/encoding full json objects is an approach worth considering.

There's a new patch version for json-iterator. Since we're almost ready to move from beta to prod, might as well bump it before release.

We chose json-iterator to parse JSON in this project because it's much faster than encoding/json for our purposes, and it offers a means to grab data directly from json without needing to unmarshall the whole thing - which is key to performance for unstruct and context fields.

The API for the latter of these, however, is awkward - it expects a list of interface arguments. So when we're using it in customer-facing applications, we must either have an unintuitive configuration, or parse the configuration in a way that's not easily and reliably done.

GJson seems to offer a solution to this latter problem, by allowing jsonpath dot notation syntax (or something very similar at least). This could make the Get functions much more straightforward to work with: https://github.com/tidwall/gjson#path-syntax

It also claims having benchmarked better performance than json-iterator - however I'm hesitant to immediately accept that this means that it would be more performant for our purposes at face value. The characteristics of the job we need it to do may well differ from what those benchmarks are based on.

I suggest that we:

1. explore the utility of gjson for the Get methods, and benchmark the performance vs. current implementation. If it offers a better API without a big performance penalty, we should use it for that purpose.

2. Benchmark gjson vs. current implementation across the other tasks in this project, and make a decision as to what we should use moving forward based on those benchmarks.

We should be scanning this repository with Snyk just like we do with all our other repositories.

e.g..
https://github.com/snowplow/sql-runner/blob/master/.github/workflows/snyk.yml

In #4 we made a change which we believed would result in unicode characters like < being escaped. It turns out this was incorrect, because of a testing error.

We should return to this issue, but I now have new questions about it:

• Should we keep escaping html characters, or revert that config?
• Should we care that unicode characters are represented as unicode, if that's how go natively treats them?

We enabled this configuration, thinking that it would make unicode characters more consistent with the other analytics SDKs.

This turned out not to be the case, a fact which we missed because we're using the default JSON library in our tests, which also leaves chevrons as unicode.

We should address this, and use comparisons with hardcoded values to ensure that we don't make similar mistakes in future.

GetContextValue returns an interface, it should return a slice of interfaces.

There is currently a difference in how the go sdk transforms the schema uri to its major name in comparison to the Scala one.
Example schema uri to reproduce: iglu:com.acme.some-dash/include-dash/jsonschema/1-0-0 :
-from Go sdk: com_acme_some-dash_include-dash_1
-from Scala sdk: com_acme_some_dash_include_dash_1

We use Go 1.15 in the test action:

Latest version is now 1.18. Let's update setup-go step to v3 and use '>=1.18.0'
https://github.com/actions/setup-go

In GetContextValue(), we begin with two JSON strings, we parse them into a slice of maps, we then marshal the maps back to json and get the values we're looking for using jsoniter.

I suspect we might be able to find an efficiency improvement on this along the lines of this:

we start with two strings which look like this:

{
  "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0",
  "data": [
    {
      "schema": "iglu:org.schema/WebPage/jsonschema/1-0-0",
      "data": {
        "genre": "blog",
        "inLanguage": "en-US",
        "datePublished": "2014-11-06T00:00:00Z",
        "author": "Fred Blundun",
        "breadcrumb": [
          "blog",
          "releases"
        ],
        "keywords": [
          "snowplow",
          "javascript",
          "tracker",
          "event"
        ]
      }
    },
    {
      "schema": "iglu:org.w3/PerformanceTiming/jsonschema/1-0-0",
      "data": {
        "navigationStart": 1415358089861,
        "unloadEventStart": 1415358090270,
        "unloadEventEnd": 1415358090287,
        "redirectStart": 0,
        "redirectEnd": 0,
        "fetchStart": 1415358089870,
        "domainLookupStart": 1415358090102,
        "domainLookupEnd": 1415358090102,
        "connectStart": 1415358090103,
        "connectEnd": 1415358090183,
        "requestStart": 1415358090183,
        "responseStart": 1415358090265,
        "responseEnd": 1415358090265,
        "domLoading": 1415358090270,
        "domInteractive": 1415358090886,
        "domContentLoadedEventStart": 1415358090968,
        "domContentLoadedEventEnd": 1415358091309,
        "domComplete": 0,
        "loadEventStart": 0,
        "loadEventEnd": 0
      }
    }
  ]
}

The parse functions currently parse these into these objects:

type SelfDescribingData struct {
	Schema string
	Data   map[string]interface{} // TODO: See if leaving data as a string or byte array would work, and would be faster.
}

type Contexts struct {
	Schema string
	Data   []SelfDescribingData
}

Instead of this, if we leave SelfDescribingData.Data as a string, we can avoid parsing things into a map and back to a JSON.

If it makes sense, we could modify the existing SelfDescribingData object (and related code) - and possibly also find efficiency improvements in the toJSON functions too. This may be complicated however, so we could also just create a separate object and a parse function specific to GetContexts for this purpose.

A 0.2.0 tag was created in error on a working branch, and the go cache now contains incorrect code for 0.2.0.

To correct this, we should commit a note on the README to specify that 0.2.1 should be used instead of 0.2.0, update the 0.2.0 release notes to mention this, and push a new 0.2.1 release.

Could have saved ourselves some work by just reading the docs. We can make our tests better by using the JSONEq method to test equivalence of JSON strings.

I have no idea why I chose to do this, but I didn't provide a full runnable example in the quickstart snippet in the readme, I just listed some methods. We should just provide a runnable main function instead

GetUnstructValue() returns an error when the path provided doesn't exist, whereasGetContextValue() returns nil with no error.

I believe these were simply two API design choices that were made independently, and there's good reason for these in isolation to behave this way - for contexts, there can be several of the same context - some of which may contain an optional field whereas some don't. In that case, we don't want to error if we find one but not the other.

We should reconsider whether or not this is the best design, since it's unintuitive - one would expect these to behave in something of a consistent manner.

Currently, the type assumed for Data in SelfDescribingData struct is a map.
This results in error (Error unmarshaling context JSON) when trying to parse unstruct events or contexts having a different type (e.g. json array) for their data.
Error can be reproduced in tests when replacing the contextString with something like:

var contextsString = `{"schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0","data":[{"schema":"iglu:com.acme.test/testing/jsonschema/1-0-0","data":["aa","bb"]}]}`

At least when printing output to console, unicode characters don't seem to be rendered appropriately after transformation.

For example, < appears as \u003c.

Since the output object for JSON is bytes in the first place, this mightn't be an issue. But we should investigate whether or not it is for both JSON and map outputs.

A fix in Go 1.18 breaks jsoniter because of its dependencies. They have patched this: json-iterator/go@024077e

If there are no complications and it's easy, we should bump our dependencies in 0.3.0. If that does present challenges we should release 0.3.0 and bump deps in 0.3.1.

Then, we should update tests as per #30

In most of our projects, CI/CD checks that versions and tags are correct before deploying releases. We haven't done so on this repo up to now, because Go automatically picks up the release based on the tag itself, rather than having us deploy ourselves.

This leaves the release process open to errors which can be messy. I think we can address this and introduce a version check with a change to the release process:

• Add a VERSION file
• Once PR is merged, developer creates a tag of the format X.X.X (eg. 0.2.0) - since Go requires format vX.X.X this won't get published to the go cache
• CI/CD runs on master only, checks that tag version == VERSION file
• If version is correct, CD tags latest commit as vVERSION

This would only run on the master branch, if we wish to publish a pre-release from a branch we can do so by manually tagging the branch with format vX.X.X-beta.1.