At the moment, Indicative event name is retrieved from enriched event's event_name (see this line). This field corresponds to the name of the event's schema (see this line).

In case of structured events, the value of this field is always "event" (see this line and this line).

We should make it possible to specify in the configuration of the lambda which field to use as event name when the value is just "event" (e.g. se_action or se_category).

with Snowplow-Relay as value

• AWS_DEPLOY_ACCESS_KEY_ID
• AWS_DEPLOY_SECRET_ACCESS_KEY

This should clearly explain how the Lambda function works.

Pay particular attention to:

• Event mapping
• Batching
• Handling of failure states

See: https://snplow.atlassian.net/browse/TECOPS-1938 for more context

This should reduce the size of the payload by quite a bit and hopefully increase the throughput of the relay.

One option to improve the performance of the relay would be to reduce the properties sent to what makes sense in an Indicative world:

app_id
platform
event
event_id
page_url
page_urlhost
page_urlpath
refr_medium
refr_term
page_referrer
refr_url
refr_urlhost
refr_urlpath
refr_source
page_title
mkt_medium
mkt_source
mkt_campaign
mkt_term
mkt_content
mkt_network
tr_orderid
tr_affiliation
tr_total
tr_tax
tr_shipping
tr_total_base
tr_tax_base
tr_shipping_base
tr_city
tr_state
tr_country
ti_orderid
ti_sku
ti_name
ti_category
ti_price
ti_price_base
ti_quantity
ti_currency
base_currency   
user_id
domain_userid
user_ipaddress
domain_sessionidx
network_userid
os_timezone
geo_country
geo_region
geo_city
geo_zipcode
geo_region_name
geo_timezone
br_family
br_name
br_type
br_lang
br_version
br_renderengine
os_name
os_family
os_manufacturer
useragent
dvce_type
event_vendor
se_action
se_property
se_label
se_category
se_value

As for contexts, it seems org.w3.PerformanceTiming can take a lot of space and is not useful in indicative.

It will make your build faster during deployment. See:

https://github.com/snowplow-incubator/snowplow-piinguin-relay/blob/a3787622c98888d9352893a8dd91e95099c91636/.travis.yml#L15-L20

https://aws.amazon.com/serverless/serverlessrepo/

Would allow for easier user deployment process. The blocker is... our jar size. Currently it's ~55 MB, the limit is 50 (even when hosting code on a S3 bucket).

The indicative batch endpoint is limited to 100 events per request but we could be pulling more than 100 events from kinesis at a time.

Errors are not easily searchable in logs. Cloudwatch logs offers both filtering by type and regex search, but currently the relay makes it v hard to find messsages for diagnosis.

It would be best to both add log levels, and introduce a searchable format to the error messages eg. "ERROR: some error description"/ "INFO: x events sent".

The unit test suit currently uses org.specs2.Specification. However, this specification does not allow multiple assertions to be tested subsequently in the same spec. This means any test that does that, actually only passes the last assertion. For example:

def e9 = {
    val base = "a" -> Json.fromString(List.fill(20)("a").mkString)
    val js   = List(Json.obj(base))
    val (toSend, tooBig) =
      Transformer.constructBatches(Transformer.getSize _, Transformer.constructJson("a") _, js, 10, 10)
    toSend shouldEqual Nil
    tooBig shouldEqual js
  }

will pass if tooBig == js, regardless of whether toSend == Nil is true or false.

Refactor the suit, so it uses org.specs2.mutable.Specification, which allows for multiple assertions to be tested in the way we want.

Every event in Indicative must have a user identifier (eventUniqueId).

The eventUniqueId is one of:

• the user_id field in atomic.events, or if that is missing
• the user_id field from the client_session context, or if that is missing
• the domain_userid from atomic.events.

For documentation on user_id and domain_userid, see the canonical event model.

For documentation on client_session_userId, see the schema.

There is a typo that misspells client_session_userId as client_session_user_id, with the upshot that all mobile events sent through the relay are considered to not have a eventUniqueId unless a user_id field is also present.

val userId = extractField(flattenedEvent, "user_id")
      .leftFlatMap(_ => extractField(flattenedEvent, "client_session_user_id"))
      .leftFlatMap(_ => extractField(flattenedEvent, "domain_userid"))
      .toOption

should be

val userId = extractField(flattenedEvent, "user_id")
      .leftFlatMap(_ => extractField(flattenedEvent, "client_session_userId"))
      .leftFlatMap(_ => extractField(flattenedEvent, "domain_userid"))
      .toOption

Events that have neither:

• a user_id
• a client_session_user_id
• a domain_userid

should be discarded since it won't be possible to identify them on indicative's side of things.

Currently the new commits are on master, which isn't our process.

this is done in #18

hammock represents too much overhead for a lambda.

This is already done in #18

In addition to a cap in the number of events, there is a cap in the total payload size.

This is done in #18.

@carabaestlein not sure whom this one should be assigned to...

Also requires:

• bump Scala to 2.12.13
• bump scalacheck-schema to 0.2.1

Currently it is embedded in the README, which is not a great approach (as it makes it hard to evolve the documentation).

A new configuration option named INDICATIVE_URI. The default value of this option is the existing "https://api.indicative.com/service/event". In the case of high-throughput deployments, it may be necessary on Indicative's end to configure dedicated endpoints to handle the load.

Source: #48

Following Snowplow's out of the box configuration documentation, Indicative was not receiving link_click events, but was receiving page_view and impression events. I triggered link_click a few times to be certain it was firing. After editing the Lambda function to explicitly exclude one of the unused atomic fields, Indicative started receiving link click events.
Link_click events aren't listed in unused events, but seems to be excluded by default.