Giter VIP home page Giter VIP logo

cognite-sdk-scala's Introduction

Cognite Scala SDK

Under development, not recommended for production use cases

This is the Cognite Scala SDK for developers working with Cognite Data Fusion.

Visit Maven Repository to see the available versions, and how to include it as a dependency for sbt, Maven, Gradle, and other build tools.

Authentication is specified using an implicit Auth parameter.

The GenericClient requires a backend for sttp, which can use any effects wrapper (for example, cats-effect, but users who do not want to use an effect wrapper can use Client to easily create a client using an identity effect.

Examples

Create a simple client, specifying an application name:

import com.cognite.sdk.scala.v1._

val c = Client("scala-sdk-examples", "https://api.cognitedata.com")

Create a client using the cats-effect IO:

import com.cognite.sdk.scala.v1._
import com.cognite.sdk.scala.common._
import java.util.concurrent.Executors
import scala.concurrent._
import cats.effect.IO
import com.softwaremill.sttp.asynchttpclient.cats.AsyncHttpClientCatsBackend

implicit val cs = IO.contextShift(ExecutionContext.fromExecutor(Executors.newCachedThreadPool()))
implicit val sttpBackend = AsyncHttpClientCatsBackend[cats.effect.IO]()
val auth = ApiKeyAuth(your API key comes here) // you can get the key at https://openindustrialdata.com/
val c = new GenericClient[IO, Nothing]("scala-sdk-examples", projectName="publicdata", auth)

The following examples will use Client with the identity effect. Add .unsafeRunSync or similar to the end of the examples if you use IO or another effect wrapper.

Getting started with Ammonite

You can use Ammonite for a better interactive Scala environment, and load the SDK directly like this:

import $ivy.`com.cognite::cognite-sdk-scala:0.1.2`
import com.cognite.sdk.scala.v1._

Discover time series

List and filter endpoints use Stream from fs2, which loads more data as required. You can use .compile.toList to convert it to a list, but note that this could end up fetching a lot of data unless you limit it, for example by using .take(25) as we do here.

For the next examples, you will need to supply ids for the time series that you want to retrieve. You can find some ids by listing the available time series:

val timeSeriesList = c.timeSeries.list().take(25).compile.toList
val timeSeriesId = timeSeriesList.head.id
print(c.timeSeries.retrieveById(timeSeriesId))

Retrieve data points

If you have a time series ID you can retrieve its data points:

val dataPoints = c.dataPoints.queryById(
      timeSeriesId,
      inclusiveStart=Instant.ofEpochMilli(0),
      exclusiveEnd=Instant.now())

It is also possible to query aggregate values using a time series ID. Possible aggregate values can be found in the API documentation You must also specify a granularity.

val aggregates = Seq("count", "average", "max")
c.dataPoints.queryAggregatesById(timeSeriesId, Instant.ofEpochMilli(0L), Instant.now(), granularity="1d", aggregates)

If you need only the last data point for a time series or group of timeseries, you can retrieve these using:

val latestPoints: Map[Long, Option[DataPoint]] = c.dataPoints.getLatestDataPointsByIds(Seq(timeSeriesId))

This returns a map from each of the time series IDs specified in the function call to the latest data point for that time series, if it exists, or None if it does not.

There are analogous functions available to execute these queries using external IDs instead of IDs and returning string valued data points rather than numeric valued data points.

Insert and Delete Data

It is possible to insert and delete numeric data points for specified time series. To insert data, you must specify the time series for which to insert data and the data points to insert:

val testDataPoints = (startTime to endTime by 1000).map(t => DataPoint(Instant.ofEpochMilli(t), math.random()))
c.dataPoints.insertById(timeSeriesId, testDataPoints)

To delete data, you must specify the time series and the range of times for which to delete data:

c.dataPoints.deleteRangeById(timeSeriesId, Instant.ofEpochMilli(0L), Instant.now())

Create an asset hierarchy

val root = AssetCreate(name = "root", externalId = Some("1"))
val child = AssetCreate(name = "child", externalId = Some("2"), parentExternalId = Some("1"))
val descendant = AssetCreate(name = "descendant", externalId = Some("3"), parentExternalId = Some("2"))
val createdAssets = c.assets.create(Seq(root, child, descendant))

// clean up the assets we created
c.assets.deleteByExternalIds(Seq("1", "2", "3"))

Print an asset subtree using filter

def children(parents: Seq[Asset]): List[Asset] =
  c.assets.filter(AssetsFilter(parentIds = Some(parents.map(_.id)))).compile.toList

def printSubTree(parents: Seq[Asset], prefix: String = ""): Unit = {
  if (parents.nonEmpty) {
    val assetsUnderThisLevel = children(parents)
    parents.foreach { p =>
      println(prefix + p.name)
      val assetsUnderThisAsset = assetsUnderThisLevel.filter(_.parentId.exists(pid => pid == p.id))
      printSubTree(assetsUnderThisAsset, prefix + "  ")
    }
  }
}

val rootId = 2780934754068396L
printSubTree(c.assets.retrieveByIds(Seq(rootId)))

Find all events in an asset subtree

def eventsForAssets(parents: Seq[Asset]): List[Event] =
  c.events.filter(EventsFilter(assetIds = Some(parents.map(_.id)))).compile.toList

def eventsInSubTree(parents: Seq[Asset]): Seq[Event] = {
  if (parents.nonEmpty) {
    val assetsUnderThisLevel = children(parents)
    eventsForAssets(parents) ++ parents.flatMap { p =>
      val assetsUnderThisAsset = assetsUnderThisLevel.filter(_.parentId.exists(pid => pid == p.id))
      eventsInSubTree(assetsUnderThisAsset)
    }
  } else {
    Seq.empty[Event]
  }
}

eventsInSubTree(c.assets.retrieveByIds(Seq(rootId))).foreach(event => println(event.`type`))

3D

To list 3D models:

val models = c.threeDModels.list().take(10).compile.toList

You can create, update, delete, and retrieve models based on their ID. To access model revisions for a specific model:

val modelId = modelIds.head.id
val revisions = c.threeDRevisions(modelId).list().take(10).compile.toList

You can also create, update, delete, and retrieve revisions based on their IDs. You can also list all the nodes in a particular revision, as well as the ancestors of a particular node (including the node itself):

val revisionId = revisionIds.head.id
val nodes = c.threeDNodes(modelId, revisionId).list().listWithLimit(10).compile.toList
val nodeId = nodeIds.head.id
val ancestorNodes = c.threeDNodes(modelId, revisionId).ancestors(nodeId).compile.toList

Running tests

To be able to run most tests locally, these environment variables need to be set:

#.env

TEST_AAD_TENANT="a valid azure ad tenant id"
TEST_CLIENT_ID="the id of a valid client credential, belonging to the given tenant, and with access to the playground project"
TEST_CLIENT_SECRET="a valid client secret for the given client id"

cognite-sdk-scala's People

Contributors

andreavs avatar andrews47288 avatar audunska avatar cfzlp avatar cognite-cicd avatar dmivankov avatar ebrattli avatar evertoncolling avatar exyi avatar hakontro avatar kornelione avatar mathialo avatar psalaberria002 avatar renovate[bot] avatar sebbert avatar sighol avatar silvavelosa avatar sush03 avatar sush1208 avatar tapped avatar tharindudg avatar thorkildcognite avatar voron86 avatar vuhainguyen avatar vvemel avatar wjoel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

amevck vm-udayya

cognite-sdk-scala's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

  • WARN: Package lookup failures

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • chore(deps): update circeversion to v0.14.7 (io.circe:circe-core, io.circe:circe-generic, io.circe:circe-literal, io.circe:circe-parser)
  • chore(deps): update dependency com.thesamet.scalapb:compilerplugin to v0.11.15
  • chore(deps): update dependency org.scalatest:scalatest to v3.2.18
  • chore(deps): update dependency sbt/sbt to v1.9.9
  • chore(deps): update jettytestversion to v9.4.54.v20240208 (org.eclipse.jetty:jetty-server, org.eclipse.jetty:jetty-servlet)
  • chore(deps): update dependency ch.epfl.scala:sbt-scalafix to v0.12.1
  • chore(deps): update dependency commons-io:commons-io to v2.16.1
  • chore(deps): update dependency org.typelevel:typelevel-scalafix to v0.3.0
  • chore(deps): update dependency scalafmt to v3.8.1
  • chore(deps): update actions/checkout action to v4
  • chore(deps): update codecov/codecov-action action to v4
  • chore(deps): update dependency com.google.protobuf:protobuf-java to v4
  • ๐Ÿ” Create all rate-limited PRs at once ๐Ÿ”

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.


Warning

Renovate failed to look up the following dependencies: Could not determine new digest for update (github-tags package uesteibar/reviewer-lottery).

Files affected: .github/workflows/lottery.yaml


Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/all-hands.yaml
  • mr-celo/stale-invite v1.3@4d29c05856f0ce4e45daa096333eecffd633c93b
.github/workflows/lottery.yaml
  • actions/checkout v3
  • uesteibar/reviewer-lottery v3.1@c291d74388da1cb583aff994b8be945e8eefbcd5
.github/workflows/scala.yml
  • actions/checkout v4
  • coursier/setup-action v1
  • actions/cache v4
  • actions/cache v4
  • actions/checkout v4
  • coursier/setup-action v1
  • actions/cache v4
  • actions/cache v4
  • actions/checkout v4
  • coursier/setup-action v1
  • actions/cache v4
  • actions/cache v4
  • codecov/codecov-action v3
  • actions/checkout v4
  • coursier/setup-action v1
sbt
build.sbt
  • org.typelevel:typelevel-scalafix 0.2.0
  • scala 2.13.8
  • commons-io:commons-io 2.15.0
  • org.eclipse.jetty:jetty-server 9.4.53.v20231009
  • org.eclipse.jetty:jetty-servlet 9.4.53.v20231009
  • org.typelevel:cats-effect 3.3.14
  • org.typelevel:cats-effect-laws 3.3.14
  • org.typelevel:cats-effect-testkit 3.3.14
  • co.fs2:fs2-core 3.3.0
  • co.fs2:fs2-io 3.3.0
  • com.google.protobuf:protobuf-java 3.25.1
  • org.tpolecat:natchez-core 0.3.1
  • org.scalatest:scalatest 3.2.17
  • com.softwaremill.sttp.client3:core 3.5.2
  • com.softwaremill.sttp.client3:circe 3.5.2
  • com.softwaremill.sttp.client3:async-http-client-backend-cats 3.5.2
  • io.circe:circe-core 0.14.6
  • io.circe:circe-generic 0.14.6
  • io.circe:circe-parser 0.14.6
  • io.circe:circe-literal 0.14.6
project/build.properties
  • sbt/sbt 1.9.8
project/plugins.sbt
  • ch.epfl.scala:sbt-scalafix 0.11.1
  • com.eed3si9n:sbt-buildinfo 0.11.0
  • org.xerial.sbt:sbt-sonatype 3.9.21
  • com.jsuereth:sbt-pgp 1.1.2-1
  • io.github.davidgregory084:sbt-tpolecat 0.4.4
  • org.scalameta:sbt-scalafmt 2.5.2
  • org.scalastyle:scalastyle-sbt-plugin 1.0.0
  • org.scoverage:sbt-scoverage 2.0.0
  • au.com.onegeek:sbt-dotenv 2.1.233
  • org.wartremover:sbt-wartremover 3.0.6
  • software.purpledragon:sbt-dependency-lock 1.5.0
  • com.thesamet:sbt-protoc 1.0.6
  • com.thesamet.scalapb:compilerplugin 0.11.14
scalafmt
.scalafmt.conf
  • scalafmt 3.5.8

  • Check this box to trigger a request for Renovate to run again on this repository

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (local>cognitedata/renovate-config)

Decoding error reading 3D Asset mappings

val client = new GenericClient[IO, Nothing]("cdp-spark-datasource")
client.threeDAssetMappings(modelId.toLong, revisionId.toLong).read().unsafeRunSync()

Attempting to read 3D Asset mappings of an existing modelId + revisionId throws a decoding error:

Attempt to decode value on failed cursor: DownField(treeIndex),DownArray,DownField(items)
DecodingFailure(Attempt to decode value on failed cursor, List(DownField(treeIndex), DownArray, DownField(items)))

Fix bugs in retrieving certain aggregates.

client.dataPoints.queryAggregatesById(
      3385857257491234L,
      1509490000001L,
      1510358399999L,
      "1d",
      Seq("min")
    ).unsafeRunSync()

This will return the counts from publicdata. Using count seems to return nothing. I have not checked other aggregates than min, max, sum and count

Add separate trait for creating a single item

Some, or maybe only Files, of our resource only support creating a single item per request. Currently we handle this with a runtime check on the length of the items passed to create, but it would be better if we handled this in a separate createOne method: it would remove the need for the runtime check, and it would allow the method to return the single response without wrapping it in a sequence/array.

Add a CreateOne trait for this method, which can easily be implemented in terms of create for resource types that support it (wrapping and unwrapping the single item in a sequence and using create), and have Files implement that trait directly (but not the Create trait).

Do chunking of all requests where necessary

Some requests have limits, like max 1000 items per delete request on assets. The SDK user shouldn't have to care about those, and we should handle it for them in the SDK itself.

As a simpler alternative we could also throw an exception on the client side if a request that is too large for the API is made, but that's hardly better than just leaving the decision to the API.

PS.

A challenge if we take care of this on the API side is that the return type of delete would then be a sequence of responses, instead of a single response. We may have to add some cats-effect to summarize the response into a single one (either a single success response, or a single failed response).

However, that may be considered a separate issue, and returning a list of responses that is usually not inspected anyway is acceptable.

Use reflection to improve update tests

Testing updates is a bit tedious, so we don't always test all different updates available.

They could be automated by using reflection on the update and/or read class to generate updates attempting to set each properties to different valid values (and set to null when allowed), and automatically compare results with expectations.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.