port-labs / ocean Goto Github PK

🌊 Ocean 🌊 is an innovative solution developed by Port to seamlessly integrate various third-party systems with our developer portal product, empowering engineers to effortlessly prioritize key features and streamline the integration process.

Home Page: https://ocean.getport.io

License: Apache License 2.0

Makefile 6.17% Python 90.90% Dockerfile 0.10% HCL 1.95% Shell 0.87%

developerportal devportal integration-framework portio servicecatalog datasourceintegrations getportio portintegration

ocean's Introduction

Ocean

Ocean is an innovative solution developed by Port to seamlessly integrate various third-party systems with our developer portal product, empowering engineers to effortlessly prioritize key features and streamline the integration process.

Prerequisites

Python 3.11

Installation

In order to install the Ocean Framework, run the following command:

pip install port-ocean[cli]

poetry add port-ocean[cli]

Run Integration

source the integration venv
```
. .venv/bin/activate
```
Run
```
ocean sail ./path/to/
integration
```

Export Architecture

Real-Time updates Architecture

Integration Lifecycle

Folder Structure

The Ocean Integration Framework follows a specific folder structure within this mono repository. This structure ensures proper organization and easy identification of integration modules. The suggested folder structure is as follows:

port-ocean/
├── port_ocean (framework)/
│ ├── ocean.py
│ ├── core/
| └── ...
└── integrations/
│  └─── integration_name/
│     ├──── main.py
│     ├──── pyproject.toml
│     └──── Dockerfile
├── ...
└── ...

The framework folder contains the core logic for managing the integration lifecycle.
Each integration is represented by a separate folder inside the integrations directory.
Inside each integration folder, you'll find a main.py file that implements the core functionality of the integration for the specific third-party system.
The pyproject.toml file inside each integration folder lists the required dependencies for that integration.

Configuration

The Integration Framework utilizes a config.yaml file for its configuration. This file defines both the framework configuration and the integration configuration within it. Each integration is identified by its type and unique identifier, which are utilized during initialization to appropriately update Port.

Example config.yaml:

# This is an example configuration file for the integration service.
# Please copy this file to config.yaml file in the integration folder and edit it to your needs.

port:
  clientId: PORT_CLIENT_ID # Can be loaded via environment variable: PORT_CLIENT_ID
  clientSecret: PORT_CLIENT_SECRET # Can be loaded via environment variable: PORT_CLIENT_SECRET
  baseUrl: https://api.getport.io/v1
# The event listener to use for the integration service.
eventListener:
  type: KAFKA / WEBHOOK
integration:
  # The name of the integration.
  identifier: "my_integration"
  # The type of the integration.
  type: "PagerDuty"
  config:
    my_git_token: "random"
    some_other_integration_config: "Very important information"

Contributing

The reason Ocean is open source is that we aim for the Port integration library to offer numerous useful out-of-the-box integrations. We have confidence that developers and DevOps professionals who rely on Port in their everyday work will be inclined to contribute and assist in making it a comprehensive tool.

In order to learn how you can contribute to Ocean, read our contributing guide

Local Development (Framework)

Clone the repository
Install dependencies:
```
make install
```
Or (For installing integrations dependencies as well)
```
make install/all
```
source the integration venv
```
. .venv/bin/activate
```

Local Development (Integration)

Clone the repository
For new integration run
```
make new
```
and follow the instructions
Install dependencies

cd DESIRED_INTEGRATION_FOLDER && make install

source the integration venv
```
. .venv/bin/activate
```
Run the integration
```
make run
```
Or
```
ocean sail
```

License

The Ocean Framework is open-source software licensed under the Apache 2.0 License. See the LICENSE file for more information.

Contact

For any questions or inquiries, please reach out to our team at [email protected]

ocean's People

Contributors

Stargazers

Watchers

ocean's Issues

[Bug] [Integration] [LaunchDarkly] : failing to ingest flags -

Ocean Framework Version

🌊 Ocean version: 0.5.9

Integration Version

no version in the spec.yaml file !! and docker image is ghcr.io/port-labs/port-ocean-launchdarkly:latest so not really helpfull !

cat .port/spec.yaml
type: launchdarkly
description: launchdarkly integration for Port Ocean
docs: https://docs.getport.io/build-your-software-catalog/sync-data-to-catalog/launchdarkly
icon: Launchdarkly
features:
  - type: exporter
    section: Feature Management
    resources:
      - kind: project
      - kind: flag
      - kind: auditlog
      - kind: environment
configurations:
  - name: launchdarklyToken
    required: true
    sensitive: true
    type: string
  - name: launchdarklyHost
    required: false
    type: string
    default: https://app.launchdarkly.com
  - name: appHost
    type: url
    required: false

Steps to reproduce

launched the helm install command for the integration and looked at the pod logs filled with errors.

What did you expect to see?

projects/environments/flags accessible from the catalog

What did you see instead?

only projects and environments where created, no flags where available.
the error logs in the pod are the following:

Request failed with status code: 422, Error: {"ok":false,"error":"invalid_request","message":"\"relations/environments\" must be string"}

And indeed looking at the auditlog online the relations are json objects.

  "relations": {
    "environments": {
      "aws-apac": {
        "_environmentName": "aws prod APAC",
        "_site": {

Are you willing to submit a PR?

I'm willing to submit a PR!

[Bug] [Integration] [Jira] A configuration error occurs when the integration runs

Ocean Framework Version

Ocean Version

🌊 Ocean version: 0.4.7

Helm Version

version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

K8s Version

1.28
Platform version: eks.6
Kubelet version: v1.28.3-eks-e71965b
AMI: amazon-eks-node-1.28-v20231201

Integration Version

ghcr.io/port-labs/port-ocean-jira:latest

Steps to reproduce

Fill secret fields and run, then container logs error.

helm upgrade --namespace observability-portio --create-namespace --install ocean-jira port-labs/port-ocean \  
        --set port.clientId="{SECRET}"  \                                    
        --set port.clientSecret="{SECRET}"  \
        --set port.baseUrl="https://api.getport.io"  \
        --set initializePortResources=true  \
        --set integration.identifier="jira"  \
        --set integration.type="jira"  \
        --set integration.eventListener.type="POLLING"  \
        --set integration.config.jiraHost="https://{SECRET}.atlassian.net"  \
        --set integration.secrets.atlassianUserEmail="{SECRET}"  \
        --set integration.secrets.atlassianUserToken="{SECRET}"

What did you expect to see?

I'm waiting to see it work.

What did you see instead?

ocean-jira-jira-container.log

Are you willing to submit a PR?

I'm willing to submit a PR!

Add issue templates

Motiviation

For making sure the Ocaen project is managed properly, and issues are opened with enough details to fix them, we would love to create issue templates.

Suggested Solution

Create the following templates inside the .github/ISSUE_TEMPLATE:

ocean framework enhancement
ocean framework bug report
integration bug report
integration enhancement

[Enhancement/Feature] [Framework] Allow ingesting values in Port based on CI/CD variables values

Motivation

We are using the GitLab ingestion of data, and we stumbled upon a use case where we want to use the value of a CI/CD variable while ingesting data into Port.

For us, the use case is that we store a key for the project in another system in CI/CD variables that are standardized in all projects, and we want to extract this key so that we can automatically connect the GitLab project and the other system project.

Proposed Changes

No response

Are you willing to submit a PR?

I'm willing to submit a PR!

[Enhancement] [Integration] [Sonar Qube] - Update raw data and default blueprints

Motivation

As for today, the Sonar Qube integration initiated with data that lacks the main features of SonarQube and people struggle to gain real value from it from the first installation.

For improving that it would be best to enrich the Raw Data that the SonarQube integration brings by addressing more SonarQube APIS.

Moreover, to reduce the time, people gain value from the integration we would like to change the default resources of the integration to include more properties.

Proposed Changes

In order to achieve the above goals, we need to change completely the blueprints and data that sonarQube brings out-of-the-box.

Here is my suggestion to blueprints:

Blueprints

[
{
  "identifier": "sonarQubeProject",
  "title": "SonarQube Project",
  "icon": "sonarqube",
  "schema": {
    "properties": {
      "organization": {
        "type": "string",
        "title": "Organization",
        "icon": "TwoUsers"
      },
      "link": {
        "type": "string",
        "format": "url",
        "title": "Link",
        "icon": "Link"
      },
      "lastAnalysisStatus": {
        "type": "string",
        "title": "Last Analysis Status",
        "enum": [
          "PASSED",
          "FAILED"
        ],
        "enumColors": {
          "PASSED": "green",
          "FAILED": "red"
        }
      },
      "lastAnalysisDate": {
        "type": "string",
        "format": "date-time",
        "icon": "Clock"
      },
      "numberOfBugs": {
        "type": "number",
        "title": "Number Of Bugs"
      },
      "numberOfCodeSmells": {
        "type": "number",
        "title": "Number Of CodeSmells"
      },
      "numberOfVulnerabilities": {
        "type": "number",
        "title": "Number Of Vulnerabilities"
      },
      "numberOfHotSpots": {
        "type": "number",
        "title": "Number Of HotSpots"
      },
      "numberOfDuplications": {
        "type": "number",
        "title": "Number Of Duplications"
      },
      "coverage": {
        "type": "number",
        "title": "Coverage"
      },
      "mainBranch": {
        "type": "string",
        "icon": "Git",
        "title": "Main Branch"
      },
      "tags": {
        "type": "array",
        "title": "Tags"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {}
},
{
  "identifier": "sonarQubeAnalysis",
  "title": "SonarQube Analysis",
  "icon": "sonarqube",
  "schema": {
    "properties": {
      "branch": {
        "type": "string",
        "title": "Branch",
        "icon": "GitVersion"
      },
      "fixedIssues": {
        "type": "number",
        "title": "Fixed Issues"
      },
      "newIssues": {
        "type": "number",
        "title": "New Issues"
      },
      "coverage": {
        "title": "Coverage",
        "type": "number"
      },
      "duplications": {
        "type": "number",
        "title": "Duplications"
      },
      "createdAt": {
        "type": "string",
        "format": "date-time"
      }
    }
  },
  "relations": {
    "sonarQubeProject": {
      "target": "sonarQubeProject",
      "required": false,
      "title": "SonarQube Project",
      "many": false
    }
  }
},
{
  "identifier": "sonarQubeIssue",
  "title": "SonarQube Issue",
  "icon": "sonarqube",
  "schema": {
    "properties": {
      "type": {
        "type": "string",
        "title": "Type",
        "enum": [
          "CODE_SMELL",
          "BUG",
          "VULNERABILITY"
        ]
      },
      "severity": {
        "type": "string",
        "title": "Severity",
        "enum": [
          "MAJOR",
          "INFO",
          "MINOR",
          "CRITICAL",
          "BLOCKER"
        ],
        "enumColors": {
          "MAJOR": "orange",
          "INFO": "green",
          "CRITICAL": "red",
          "BLOCKER": "red",
          "MINOR": "yellow"
        }
      },
      "link": {
        "type": "string",
        "format": "url",
        "icon": "Link",
        "title": "Link"
      },
      "status": {
        "type": "string",
        "title": "Status",
        "enum": [
          "OPEN",
          "CLOSED",
          "RESOLVED",
          "REOPENED",
          "CONFIRMED"
        ]
      },
      "assignees": {
        "title": "Assignees",
        "type": "array",
        "icon": "TwoUsers",
        "items": {
          "type": "string",
          "format": "user"
        }
      },
      "tags": {
        "type": "array",
        "title": "Tags"
      }
    }
  },
  "relations": {
    "sonarQubeProject": {
      "target": "sonarQubeProject",
      "required": false,
      "title": "SonarQube Project",
      "many": false
    }
  }
}
]

Are you willing to submit a PR?

I'm willing to submit a PR!

[Feature][Framework] - Development mode for newly created integrations

Motivation

Currently only released integrations can be seen and managed in the data sources section in the Port UI.

This means that when developing new integrations the way to test the resync of integrations is through the port api.

This effects the development testing as well as viewing how a user of the integration will use and experience the integration.

following #368

Proposed Changes

Add option to run integration in kind of "development mode" which will create as similar as possible experience to how user will interact with the integration in production.

Are you willing to submit a PR?

I'm willing to submit a PR!

[Bug] [Integration] [argoCD] : failing to synchronize all apps

Ocean Framework Version

🌊 Ocean version: 0.5.12

Integration Version

🚢 Integration version: 0.1.38

Steps to reproduce

launched the "docker run command"
may entities where ingested properly but one is failing

What did you expect to see?

everything worked

What did you see instead?

the following error

2024-04-21 09:30:22.551 | INFO     | Sending request to ArgoCD API: GET https://readacted.domain.com/api/v1/applications/messaging/managed-resources
2024-04-21 09:30:22.594 | ERROR    | Failed to execute resync function, error: 'items'
Traceback (most recent call last):
  File "/usr/local/bin/ocean", line 8, in <module>
    sys.exit(cli_start())
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/port_ocean/cli/commands/sail.py", line 74, in sail
    run(path, log_level, port, initialize_port_resources, override)
  File "/usr/local/lib/python3.11/site-packages/port_ocean/run.py", line 57, in run
    uvicorn.run(app, host="0.0.0.0", port=application_settings.port)
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 578, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
    handle._run()
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/sync_raw.py", line 196, in _register_in_batches
    async for items in generator:
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/utils.py", line 52, in resync_generator_wrapper
    with resync_error_handling():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
> File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/utils.py", line 23, in resync_error_handling
    yield
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/utils.py", line 53, in resync_generator_wrapper
    result = await anext(generator)
  File "/app/./main.py", line 47, in on_managed_resources_resync
    managed_resources = await argocd_client.get_managed_resources(
  File "/app/client.py", line 107, in get_managed_resources
    managed_resources = (await self._send_api_request(url=url))["items"]
Exception: 'items'
2024-04-21 09:30:22.600 | INFO     | Finished registering change for 1 raw results for kind: managed-resource. 736 entities were affected
2024-04-21 09:30:22.600 | INFO     | Starting resync diff calculation
2024-04-21 09:30:22.601 | ERROR    | Resync failed with 1. Skipping delete phase due to incomplete state
2024-04-21 09:30:22.601 | INFO     | Event finished
2024-04-21 09:30:22.602 | INFO     | Once event listener finished
2024-04-21 09:30:22.602 | INFO     | Exiting application

Are you willing to submit a PR?

I'm willing to submit a PR!

Adding mapping config to helm charts

Currently, you can update the integration mapping config and default blueprints through the ocean integration after deploying.

Adding the option to provide those as parameters in the helm chart could make the process of deploying integrations faster and better.

https://port.canny.io/admin/board/integrations/p/adding-mapping-config-to-helm-charts

[Enhancement] [Integration] [PagerDuty] Rename "Create At" property to "Created At"

Motivation

Currently the default blueprint for PagerDuty creates a property called createdAt, but it's title is set as "Create At", it would make more sense if it was called "Created At" instead.

Proposed Changes

ocean/integrations/opsgenie/.port/resources/blueprints.json

Line 139 in 71670fe

"title": "Create At",

-          "title": "Create At",
+          "title": "Created At",

Are you willing to submit a PR?

I'm willing to submit a PR!

[Bug] [Integration] [Sonarqube] - not supporting self hosted sonarqube

Ocean Framework Version

Any

Integration Version

Latest

Steps to reproduce

There is no OrganizationKey for the self hosted version + Some of the routes are missing as well

What did you expect to see?

Be able not to pass the organizaton id
Use routes that are common to cloud and self hosted

What did you see instead?

Can't install

Are you willing to submit a PR?

I'm willing to submit a PR!

Issues with pyjq 2.6.0 when installing on Windows

Discussed in #837

^{Originally posted by nicholasamato July 23, 2024}
I've been getting issues attempting to install ocean framework with a regular pip install and also with poetry.

The error always results when the install reachespyjq. I get the following error with poetry:

error: [WinError 2] The system cannot find the file specified

at ~\pipx\venvs\poetry\Lib\site-packages\poetry\installation\chef.py:164 in _prepare
160│
161│ error = ChefBuildError("\n\n".join(message_parts))
162│
163│ if error is not None:
→ 164│ raise error from None
165│
166│ return path
167│
168│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pyjq (2.6.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pyjq (==2.6.0)"'.

Does anyone have a solution? (Im using python 3.11.9)

[Bug] [Integration] [OpsGenie]

Ocean Framework Version

Linux ocean-opsgenie-opsgenie-deployment-*** 5.15.117 #1 SMP Thu Jul 6 03:52:13 UTC 2023 x86_64 GNU/Linux
🌊 Ocean version: 0.4.2

Integration Version

latest
sha256:798c81817034a3b3a8b36df247cd572c58fc014049e5fc5c2bb946f0c214f3bc

Steps to reproduce

Run integration with 40+ OpsGenie Incidents, 429 responses will be abundant.

What did you expect to see?

Maybe one 429 then a sleep and retry from the integration.

What did you see instead?

20 or so 429 HTTP errors reported in the logs. The integration did not attempt to resync to remedy the issue.

Are you willing to submit a PR?

I'm willing to submit a PR!

[Enhancement/Feature] [Integration] [argoCD] : allow for Cluster name override

Motivation

We are using argoCD in an embedded mode, that means we have an instance of ArgoCD running in each of our clusters.
each cluster définition comes with name and a server values.

The server value is always the same as it is internal to the cluster but the cluster name changes in order to be able to identify it.
The current integration uses the api/v1/applications api to list all applications and uses the cluster name to link the app to a cluster.
Unfortunately the API does not returns the cluster name, see the issue I created on argoCD side.

Proposed Changes

Would it be possible to allow an override of cluster name in the integration to work around that issue ?
Adding a new config for the user to specify the cluster name.

Are you willing to submit a PR?

I'm willing to submit a PR!

[Bug] [Integration] [Azure] Timeout

Ocean Framework Version

0.9.1

Integration Version

0.1.66

Steps to reproduce

Launch a Sync

What did you expect to see?

Resources in Port

What did you see instead?

Timeout

2024-07-03 07:15:46.352 | ERROR    | Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/tls.py", line 140, in _call_sslobject_method
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 921, in read
    v = self._sslobj.read(len)
        ^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2580)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_tasks.py", line 115, in fail_after
    yield cancel_scope
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 34, in read
    return await self._stream.receive(max_bytes=max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/tls.py", line 205, in receive
    data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/tls.py", line 147, in _call_sslobject_method
    data = await self.transport_stream.receive()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1133, in receive
    await self._protocol.read_event.wait()
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f936f308e90

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with anyio.fail_after(timeout):
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/anyio/_core/_tasks.py", line 118, in fail_after
    raise TimeoutError
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 121, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 164, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http11.py", line 200, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 31, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/port_ocean/utils/repeat.py", line 66, in loop
    await func()  # type: ignore
    ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/event_listener/polling.py", line 85, in resync
    await running_task
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/sync_raw.py", line 446, in sync_raw_all
    creation_results.append(await task)
                            ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/sync_raw.py", line 197, in _register_in_batches
    entities, register_errors = await self._register_resource_raw(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/sync_raw.py", line 144, in _register_resource_raw
    await self.entities_state_applier.upsert(
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/handlers/entities_state_applier/port/applier.py", line 113, in upsert
    await self.context.port_client.upsert_entity(
  File "/usr/local/lib/python3.11/site-packages/port_ocean/clients/port/mixins/entities.py", line 39, in upsert_entity
    response = await self.client.post(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/port_ocean/helpers/retry.py", line 158, in handle_async_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

As a side note, some resources get synced, and it randomly timeouts. Sometime It finishes until the end without timing out, sometime it times out, it's very random.

Are you willing to submit a PR?

I'm willing to submit a PR!

[Bug] [Integration] [Integration Name (PagerDuty/Jira/Other..)]

Ocean Framework Version

mac OS 10.5 ocean 0.3.1

Integration Version

0.1.20

Steps to reproduce

Run the integration
Wait for the integration to pull the oncalls from PagerDuty API

What did you expect to see?

List of on calls being imported

What did you see instead?

only 1 on call was being fetched
The function that queries the on calls from the PagerDuty API was parsing the query incorrectly.
The escalation policies ID needs to have a query filter for each ID, instead of being sent together grouped.
The query was structured as:
&escalation_policy_ids[]=id1,id2,id3 which seems to be incorrect.
The correct query seems to be:
&escalation_policy_ids[]=id1&escalation_policy_ids[]=id2&escalation_policy_ids[]=id3
I have changed how the params are being parsed, which should solve the issue.

Are you willing to submit a PR?

I'm willing to submit a PR!

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

dockerfile

integrations/_infra/Dockerfile

python 3.11-slim-bookworm

port_ocean/cli/cookiecutter/{{cookiecutter.integration_slug}}/Dockerfile

python 3.11-slim-bookworm

github-actions

.github/workflows/apply-release.yml

actions/checkout v4

actions/setup-python v5

snok/install-poetry v1

peter-evans/create-pull-request v6

.github/workflows/ci.yml

actions/checkout v4

actions/checkout v4

docker/setup-qemu-action v3

docker/setup-buildx-action v3

docker/build-push-action v6

.github/workflows/combine-dependabot-prs.yml

actions/github-script v7

.github/workflows/create-new-sonarcloud-project.yml

actions/checkout v4

tj-actions/changed-files v44

gliech/create-github-secret-action v1

.github/workflows/lint.yml

actions/checkout v4

tj-actions/changed-files v44.5.7

actions/setup-python v5

actions/checkout v4

.github/workflows/pr-labeler.yml

actions/labeler v5

pascalgn/size-label-action v0.5.2

.github/workflows/release-framework.yml

actions/setup-python v5

actions/checkout v4

fjogeleit/http-request-action v1

.github/workflows/release-integrations.yml

actions/checkout v4

docker/login-action v3

actions/checkout v4

docker/setup-qemu-action v3

docker/setup-buildx-action v3

docker/login-action v3

docker/build-push-action v6

actions/checkout v4

aws-actions/configure-aws-credentials v4

.github/workflows/sonarcloud-framework.yml

actions/checkout v4

.github/workflows/sonarcloud-integrations.yml

actions/checkout v4

tj-actions/changed-files v44

actions/checkout v4

.github/workflows/test.yml

actions/checkout v4

tj-actions/changed-files v44.5.7

actions/setup-python v5

actions/checkout v4

mikepenz/action-junit-report v4

.github/workflows/upgrade-integrations.yml

actions/checkout v4

actions/setup-python v5

snok/install-poetry v1

peter-evans/create-pull-request v6

.github/workflows/validate-integration-files.yml

actions/setup-python v5

actions/checkout v4

.github/workflows/verify-docs-build.yml

actions/checkout v4

actions/setup-node v4

npm

docs/framework-guides/package.json

@docusaurus/core ^3.0.1

@docusaurus/plugin-client-redirects ^3.0.1

@docusaurus/plugin-content-blog ^3.0.1

@docusaurus/plugin-ideal-image ^3.0.1

@docusaurus/preset-classic ^3.0.1

@docusaurus/theme-common ^3.0.1

@docusaurus/theme-live-codeblock ^3.0.1

@easyops-cn/docusaurus-search-local ^0.23.3

@mdx-js/react ^3.0.0

@stackql/docusaurus-plugin-hubspot ^1.0.1

clsx ^1.1.1

docusaurus-plugin-module-alias ^0.0.2

docusaurus-plugin-sass ^0.2.5

prettier ^2.8.2

prism-react-renderer ^2.3.1

react ^18.2.0

react-dom ^18.2.0

redocusaurus ^2.0.0

remark-math ^5.1.1

stream ^0.0.2

timers ^0.1.1

xml2js ^0.6.2

@docusaurus/module-type-aliases ^3.0.1

@docusaurus/tsconfig ^3.0.1

@docusaurus/types ^3.0.1

pretty-quick ^3.1.3

typescript ^5.3.3

pep621

integrations/argocd/pyproject.toml

poetry-core >=1.0.0

integrations/aws/pyproject.toml

poetry-core >=1.0.0

integrations/azure-devops/pyproject.toml

poetry-core >=1.0.0

integrations/azure/pyproject.toml

poetry-core >=1.0.0

integrations/datadog/pyproject.toml

poetry-core >=1.0.0

integrations/dynatrace/pyproject.toml

poetry-core >=1.0.0

integrations/firehydrant/pyproject.toml

poetry-core >=1.0.0

integrations/gcp/pyproject.toml

poetry-core >=1.0.0

integrations/gitlab/pyproject.toml

poetry-core >=1.0.0

integrations/jenkins/pyproject.toml

poetry-core >=1.0.0

integrations/jira/pyproject.toml

poetry-core >=1.0.0

integrations/kafka/pyproject.toml

poetry-core >=1.0.0

integrations/kubecost/pyproject.toml

poetry-core >=1.0.0

integrations/launchdarkly/pyproject.toml

poetry-core >=1.0.0

integrations/linear/pyproject.toml

poetry-core >=1.0.0

integrations/newrelic/pyproject.toml

poetry-core >=1.0.0

integrations/octopus/pyproject.toml

poetry >=0.12

integrations/opencost/pyproject.toml

poetry-core >=1.0.0

integrations/opsgenie/pyproject.toml

poetry-core >=1.0.0

integrations/pagerduty/pyproject.toml

poetry-core >=1.0.0

integrations/sentry/pyproject.toml

poetry-core >=1.0.0

integrations/servicenow/pyproject.toml

poetry-core >=1.0.0

integrations/snyk/pyproject.toml

poetry-core >=1.0.0

integrations/sonarqube/pyproject.toml

poetry-core >=1.0.0

integrations/statuspage/pyproject.toml

poetry-core >=1.0.0

integrations/terraform-cloud/pyproject.toml

poetry-core >=1.0.0

integrations/wiz/pyproject.toml

poetry-core >=1.0.0

port_ocean/cli/cookiecutter/{{cookiecutter.integration_slug}}/pyproject.toml

poetry-core >=1.0.0

pyproject.toml

poetry-core >=1.0.0

poetry

integrations/argocd/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/aws/pyproject.toml

python ^3.11

port_ocean ^0.10.0

python-dotenv ^1.0.1

aioboto3 ^12.4.0

boto3-stubs 1.34.76

types-aioboto3 ^12.4.0

types-aiobotocore ^2.12.3

pytest ^7.2

black ^24.3.0

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/azure-devops/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.2.2

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/azure/pyproject.toml

python ^3.11

port_ocean ^0.10.0

azure-mgmt-resource 23.0.1

azure-identity ^1.13.0

aiohttp ^3.9.4

cloudevents ^1.9.0

azure-mgmt-subscription ^3.1.1

aiostream ^0.5.2

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/datadog/pyproject.toml

python ^3.11

port_ocean ^0.10.0

loguru ^0.7.2

pytest ^7.2

black ^24.3.0

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/dynatrace/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/firehydrant/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/gcp/pyproject.toml

python ^3.11

port_ocean ^0.10.0

google-cloud-asset ^3.25.1

google-cloud-pubsub ^2.21.1

google-cloud-resource-manager ^1.12.3

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.3.4

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/gitlab/pyproject.toml

python ^3.11

aiofiles ^0.6.0

python-gitlab ^3.14.0

pathlib ^1.0.1

jsonschema ^4.17.3

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/jenkins/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pip ^23.3.1

python-dotenv ^1.0.0

loguru ^0.7.2

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/jira/pyproject.toml

python ^3.11

port_ocean ^0.10.0

httpx ^0.24.1

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/kafka/pyproject.toml

python ^3.11

port_ocean ^0.10.0

confluent-kafka ^2.2.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/kubecost/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/launchdarkly/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/linear/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.3.0

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/newrelic/pyproject.toml

python ^3.11

port_ocean ^0.10.0

httpx ^0.24.1

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/octopus/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^23.3.0

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

integrations/opencost/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/opsgenie/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/pagerduty/pyproject.toml

python ^3.11

port_ocean ^0.10.0

httpx ^0.24.1

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/sentry/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/servicenow/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/snyk/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/sonarqube/pyproject.toml

python ^3.11

port_ocean ^0.10.0

rich ^13.5.2

cookiecutter ^2.3.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/statuspage/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.3.0

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/terraform-cloud/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

integrations/wiz/pyproject.toml

python ^3.11

port_ocean ^0.10.0

pytest ^7.2

black ^24.4.2

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pytest-xdist ^3.6.1

port_ocean/cli/cookiecutter/{{cookiecutter.integration_slug}}/pyproject.toml

python ^3.11

pytest ^7.2

pytest-xdist ^3.6.1

pre-commit ^3.7.1

requests ^2.32.3

black ^23.3.0

mypy ^1.3.0

ruff ^0.0.278

pylint ^2.17.4

towncrier ^23.6.0

pyproject.toml

python ^3.11

pydantic ^1.10.8

loguru ^0.7.0

pyyaml ^6.0

werkzeug >=2.3.4,<4.0.0

fastapi >=0.100,<0.112

uvicorn >=0.22,<0.31

confluent-kafka ^2.1.1

httpx >=0.24.1,<0.28.0

pyjq ^2.6.0

urllib3 >=1.26.16,<3.0.0

six ^1.16.0

pyhumps ^3.8.0

pydispatcher ^2.0.7

tomli ^2.0.1

aiostream >=0.5.2,<0.7.0

python-dateutil ^2.9.0.post0

click ^8.1.3

rich ^13.4.1

cookiecutter ^2.1.1

jinja2-time ^0.2.0

pytest >=7.2,<9.0

black >=23.3,<25.0

mypy ^1.3.0

pylint >=2.17.4,<4.0.0

ruff >=0.0.278,<0.4.11

towncrier ^23.6.0

yamllint ^1.32.0

pytest-xdist ^3.6.1

pre-commit ^3.7.1

pyenv

.python-version

python 3.11

terraform

deployment/terraform/aws/ecs/main.tf

deployment/terraform/aws/ecs/providers.tf

aws ~> 5.1

env 0.0.2

jsonschema 0.2.1

Check this box to trigger a request for Renovate to run again on this repository

[Enhancement/Feature] [Framework] - fix vulnerabilities found in the source code

Motivation

In order to improve security of your product, you should fix these vulnerabilities. Check the scan report file below, you have also recommendations for fixing them.

DetailedReport_Ocean_29_May_2024.pdf

Proposed Changes

DetailedReport_Ocean_29_May_2024.pdf

Are you willing to submit a PR?

I'm willing to submit a PR!

[Enhancement/Feature] [Integration] Linear

Motivation

We're using Linear for Issue / Project management and would like to use Port.

Proposed Changes

Create a linear app plugin

Are you willing to submit a PR?

I'm willing to submit a PR!

Ocean | Support Self-signed certificates

Add support for self-signed certificates in ocean integrations.

This will allow support to self-hosted integrations with self-signed certificates.

https://port.canny.io/admin/board/ideas/p/ocean-support-self-signed-certificates

[Bug] [Integration] [Azure DevOps] Unable to fetch Pull Requests from disabled git repos

Ocean Framework Version

0.5.4

Integration Version

0.1.3

Steps to reproduce

Have Azure DevOps organization with disabled repositories
Deploy Azure DevOps integration via Helm Chart and configure it sync Pull Requests
Observe the integration failing to get pull requests for disabled repo

What did you expect to see?

Azure DevOps integration should skip disable Git repos when getting Pull Requests from ADO API

What did you see instead?

Exception in the log:

Found 27 objects in url https:[REDACTED]/_apis/projects with params: {'$top': 50, 'continuationToken': '', 'includeCapabilities': 'true'}
2024-03-05 19:50:12.217 | ERROR    | Request with bad status code 404: GET to url https:[REDACTED]/5b27c16d-1730-4cd9-ae15-dd4cef681412/_apis/git/repositories/5e3293e9-0a05-410c-9f33-1b76b8356e5a/pullrequests
Traceback (most recent call last):
  File "/usr/local/bin/ocean", line 8, in <module>
    sys.exit(cli_start())
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/port_ocean/cli/commands/sail.py", line 74, in sail
    run(path, log_level, port, initialize_port_resources, override)
  File "/usr/local/lib/python3.11/site-packages/port_ocean/run.py", line 57, in run
    uvicorn.run(app, host="0.0.0.0", port=application_settings.port)
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 578, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once
    handle._run()
  File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/sync_raw.py", line 196, in _register_in_batches
    async for items in generator:
  File "/usr/local/lib/python3.11/site-packages/port_ocean/core/integrations/mixins/utils.py", line 53, in resync_generator_wrapper
    result = await anext(generator)
  File "/app/./main.py", line 61, in resync_pull_requests
    async for pull_requests in azure_devops_client.generate_pull_requests(
  File "/app/azure_devops/client/azure_devops_client.py", line 72, in generate_pull_requests
    async for filtered_pull_requests in self._get_paginated_by_top_and_skip(
  File "/app/azure_devops/client/base_client.py", line 76, in _get_paginated_by_top_and_skip
    objects_page = (await self.send_request("GET", url, params=params)).json()[
> File "/app/azure_devops/client/base_client.py", line 35, in send_request
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)

Are you willing to submit a PR?

I'm willing to submit a PR!

Help/Docs on how to test integration locally

Hi, I'm trying to develop a new integration with Ocean, but I have no idea how to get it to show up in Port to test it.

Do only merged integrations show up in the Port UI as a datasource?

How to I get my local ocean integration synching with Port during development?

`config.yaml`

initializePortResources: true
scheduledResyncInterval: 60
port:
  clientId: "{{ from env PORT_CLIENT_ID }}" # Can be loaded via environment variable: PORT_CLIENT_ID
  clientSecret: "{{ from env PORT_CLIENT_SECRET }}" # Can be loaded via environment variable: PORT_CLIENT_SECRET
# The event listener to use for the integration service.
eventListener:
  type: POLLING
integration:
  # The identifier of this integration instance.
  identifier: "flipt"
  # The type of the integration.
  type: "flipt"
  config:
    fliptToken: "{{ from env FLIPT_API_TOKEN }}"
    fliptURL: "{{ from env FLIPT_URL }}"

`.port/spec.yaml`

type: flipt
description: Flipt integration for Port Ocean
icon: Cookiecutter # Should be one of the available icons in Port
features:
  - type: exporter
    section: CICD # IDK if this is correct or what the valid sections
    resources:
      - kind: namespaces
      - kind: flags
configurations:
  - name: fliptToken
    required: true
    type: string
    sensitive: true
  - name: fliptURL
    type: url
    required: true

Logs

Setting sail... ⛵️⚓️⛵️⚓️ All hands on deck! ⚓️
🌊 Ocean version: 0.4.12
🚢 Integration version: 0.1.0
2024-01-27 14:30:18.717 | INFO     | Registering resync event listener for kind namespaces
2024-01-27 14:30:18.718 | INFO     | Creating integration with id: flipt
2024-01-27 14:30:18.718 | INFO     | No token found, fetching new token
2024-01-27 14:30:18.718 | INFO     | Fetching access token for clientId: REDACTED
2024-01-27 14:30:19.220 | ERROR    | Request failed with status code: 409, Error: {"ok":false,"error":"integration_installed","message":"The app for your organization already installed"}
2024-01-27 14:30:19.220 | ERROR    | Failed to create resources: {"ok":false,"error":"integration_installed","message":"The app for your organization already installed"}. Rolling back changes...
2024-01-27 14:30:19.220 | WARNING  | Failed to create resources. Rolling back blueprints : []
INFO:     Started server process [90575]
INFO:     Waiting for application startup.
2024-01-27 14:30:19.225 | INFO     | Starting integration
2024-01-27 14:30:19.225 | INFO     | Initializing integration components
2024-01-27 14:30:19.225 | INFO     | Initializing integration at port
2024-01-27 14:30:19.225 | INFO     | Initiating integration with id: flipt
2024-01-27 14:30:19.225 | INFO     | Fetching integration with id: flipt
2024-01-27 14:30:19.643 | INFO     | Checking for diff in integration configuration
Starting integration
2024-01-27 14:30:19.644 | INFO     | Integration with id: flipt successfully registered
2024-01-27 14:30:19.644 | INFO     | Event started
2024-01-27 14:30:19.645 | INFO     | Event finished
INFO:     Application startup complete.
2024-01-27 14:30:19.645 | INFO     | Initializing event listener
2024-01-27 14:30:19.645 | INFO     | Found event listener type: polling
2024-01-27 14:30:19.645 | INFO     | Setting up Polling event listener with interval: 60
2024-01-27 14:30:19.645 | INFO     | Setting up scheduled resync, the integration will automatically perform a full resync every 60 minutes)
2024-01-27 14:30:19.645 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:30:19.646 | INFO     | Fetching integration with id: flipt
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024-01-27 14:30:19.746 | INFO     | Detected change in integration, resyncing
2024-01-27 14:30:19.747 | INFO     | Resync was triggered
2024-01-27 14:30:19.747 | INFO     | Event started
2024-01-27 14:30:19.747 | INFO     | Fetching port app config
2024-01-27 14:30:19.747 | INFO     | Fetching integration with id: flipt
2024-01-27 14:30:19.855 | INFO     | Searching entities with query {'combinator': 'and', 'rules': [{'property': '$datasource', 'operator': 'contains', 'value': 'port-ocean/flipt/'}, {'property': '$datasource', 'operator': 'contains', 'value': '/flipt/exporter'}]}
2024-01-27 14:30:19.974 | INFO     | Event finished
2024-01-27 14:31:19.977 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:31:19.978 | INFO     | Fetching integration with id: flipt
2024-01-27 14:32:20.469 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:32:20.470 | INFO     | Fetching integration with id: flipt
2024-01-27 14:33:20.870 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:33:20.871 | INFO     | Fetching integration with id: flipt
2024-01-27 14:34:21.486 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:34:21.487 | INFO     | Fetching integration with id: flipt
2024-01-27 14:35:21.846 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:35:21.847 | INFO     | Fetching integration with id: flipt
2024-01-27 14:36:22.238 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:36:22.239 | INFO     | Fetching integration with id: flipt
2024-01-27 14:36:22.650 | INFO     | Detected change in integration, resyncing
2024-01-27 14:36:22.651 | INFO     | Resync was triggered
2024-01-27 14:36:22.652 | INFO     | Event started
2024-01-27 14:36:22.653 | INFO     | Fetching port app config
2024-01-27 14:36:22.653 | INFO     | Fetching integration with id: flipt
2024-01-27 14:36:22.756 | INFO     | Searching entities with query {'combinator': 'and', 'rules': [{'property': '$datasource', 'operator': 'contains', 'value': 'port-ocean/flipt/'}, {'property': '$datasource', 'operator': 'contains', 'value': '/flipt/exporter'}]}
2024-01-27 14:36:22.875 | INFO     | Event finished
2024-01-27 14:37:22.876 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:37:22.877 | INFO     | Fetching integration with id: flipt
2024-01-27 14:38:23.273 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:38:23.274 | INFO     | Fetching integration with id: flipt
2024-01-27 14:39:23.707 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:39:23.708 | INFO     | Fetching integration with id: flipt
2024-01-27 14:40:24.150 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:40:24.150 | INFO     | Fetching integration with id: flipt
2024-01-27 14:41:24.549 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:41:24.549 | INFO     | Fetching integration with id: flipt
2024-01-27 14:42:24.987 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:42:24.988 | INFO     | Fetching integration with id: flipt
2024-01-27 14:43:25.402 | INFO     | Polling event listener iteration after 60. Checking for changes
2024-01-27 14:43:25.402 | INFO     | Fetching integration with id: flipt
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [90575]

[Feature] [Integration] [Snyk] - New Snyk Integration

Motivation

Snyk is one of the leading tools to scan package vulnerabilities.

A developer portal can leverage having a unified view of the services and their package's vulnerabilities.

Currently, you can achieve that by using Port's custom webhook feature as described here, but it cannot bring history data about those vulnerabilities.

Therefore, it would be great to implement an Ocean integration that will help to bring data and to map into blueprints in the catalog.

Proposed Changes

Files

We will add a new Ocean integration under the integrations folder named snyk

Configuration

It's configuration would be as the following:

...
integration:
  # The identifier of this integration instance.
  identifier: my-snyk-integration
  # The type of the integration.
  type: "snyk"
  config:
    apiToken: REQUIRED
    webhookSecret: OPTIONAL
    apiUrl: OPTIONAL with default to the default Snyk API url
    organizationId: REQUIRED
    appHost: OPTIONAL

The webhookSecret + appHost are optional because we want to make event real-time optional so users won't have to configure a webhook URL public endpoint.

API

Use the latest Snyk API, 2023-08-21 or 2023-08-21~beta.

Blueprints

Preview

Blueprints JSON

[
  {
    "identifier": "snykProject",
    "description": "This blueprint represents a snyk project in our software catalog",
    "title": "Snyk Project",
    "icon": "Snyk",
    "schema": {
      "properties": {
        "url": {
          "type": "string",
          "title": "URL",
          "format": "url",
          "icon": "Snyk"
        },
        "owner": {
          "type": "string",
          "title": "Owner",
          "format": "user",
          "icon": "TwoUsers"
        },
        "businessCriticality": {
          "title": "Business Criticality",
          "type": "array",
          "items": {
            "type": "string",
            "enum": [
              "Critical",
              "High",
              "Medium",
              "Low"
            ]
          },
          "icon": "DefaultProperty"
        },
        "environment": {
          "items": {
            "type": "string"
          },
          "icon": "Environment",
          "title": "Environment",
          "type": "array"
        },
        "lifeCycle": {
          "title": "Life Cycle",
          "type": "array",
          "items": {
            "type": "string",
            "enum": [
              "Development",
              "Sandbox",
              "Production"
            ]
          },
          "icon": "DefaultProperty"
        },
        "highOpenVulnerabilities": {
          "icon": "Vulnerability",
          "type": "number",
          "title": "Open High Vulnerabilities"
        },
        "mediumOpenVulnerabilities": {
          "icon": "Vulnerability",
          "type": "number",
          "title": "Open Medium Vulnerabilities"
        },
        "lowOpenVulnerabilities": {
          "icon": "Vulnerability",
          "type": "number",
          "title": "Open Low Vulnerabilities"
        },
        "importedBy": {
          "icon": "TwoUsers",
          "type": "string",
          "title": "Imported By",
          "format": "user"
        },
        "tags": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "title": "Tags",
          "icon": "DefaultProperty"
        },
        "targetOrigin": {
          "title": "Target Origin",
          "type": "string",
          "enum": [
            "artifactory-cr",
            "aws-config",
            "aws-lambda",
            "azure-functions",
            "azure-repos",
            "bitbucket-cloud",
            "bitbucket-server",
            "cli",
            "cloud-foundry",
            "digitalocean-cr",
            "docker-hub",
            "ecr",
            "gcr",
            "github",
            "github-cr",
            "github-enterprise",
            "gitlab",
            "gitlab-cr",
            "google-artifact-cr",
            "harbor-cr",
            "heroku",
            "ibm-cloud",
            "kubernetes",
            "nexus-cr",
            "pivotal",
            "quay-cr",
            "terraform-cloud"
          ],
          "icon": "DefaultProperty"
        },
        "targetUrl": {
          "title": "Target URL",
          "type": "string",
          "format": "url",
          "icon": "Snyk"
        }
      },
      "required": []
    },
    "mirrorProperties": {},
    "calculationProperties": {},
    "relations": {}
  },
  {
    "identifier": "snykVulnerability",
    "description": "This blueprint represents a Snyk vulnerability in our software catalog",
    "title": "Snyk Vulnerability",
    "icon": "Snyk",
    "schema": {
      "properties": {
        "score": {
          "icon": "Star",
          "type": "number",
          "title": "Score"
        },
        "packageName": {
          "type": "string",
          "title": "Package Name",
          "icon": "DefaultProperty"
        },
        "packageVersions": {
          "icon": "Package",
          "title": "Package Versions",
          "type": "array"
        },
        "type": {
          "type": "string",
          "title": "Type",
          "enum": [
            "vuln",
            "license",
            "configuration"
          ],
          "icon": "DefaultProperty"
        },
        "severity": {
          "icon": "Alert",
          "title": "Issue Severity",
          "type": "string",
          "enum": [
            "low",
            "medium",
            "high",
            "critical"
          ],
          "enumColors": {
            "low": "green",
            "medium": "yellow",
            "high": "red",
            "critical": "red"
          }
        },
        "url": {
          "icon": "Link",
          "type": "string",
          "title": "Issue URL",
          "format": "url"
        },
        "language": {
          "type": "string",
          "title": "Language",
          "icon": "DefaultProperty"
        },
        "publicationTime": {
          "type": "string",
          "format": "date-time",
          "title": "Publication Time",
          "icon": "DefaultProperty"
        },
        "isPatched": {
          "type": "boolean",
          "title": "Is Patched",
          "icon": "DefaultProperty"
        }
      },
      "required": []
    },
    "mirrorProperties": {},
    "calculationProperties": {},
    "relations": {
      "snykProject": {
        "title": "Project",
        "target": "snykProject",
        "required": false,
        "many": false
      }
    }
  }
]

Are you willing to submit a PR?

I'm willing to submit a PR!

[Enhancement/Feature] [Integration] [SonarQube] Filter Applications out when syncing Projects

Motivation

Currently, SonarQube integration exports Applications when syncing Projects because it does not use qualifiers query string when talking to SonarQube API in its get_components method: https://github.com/port-labs/ocean/blob/main/integrations/sonarqube/client.py#L135

This results in Port displaying both Applications and Projects under Projects blueprint which is undesired.

Proposed Changes

Use qualifiers=TRK filter/query string when talking to SonarQube API to fetch Projects to exclude Applications.
SonarQube API docs: https://next.sonarqube.com/sonarqube/web_api/api/components?query=components

Are you willing to submit a PR?

I'm willing to submit a PR!

[Enhancement] [Integration] [Pager Duty] - Update raw data and default blueprints

Motivation

As for today, the Pager Duty integration initiated with data that lacks the main features of Pager Duty and people struggle to gain real value from it from the first installation.

For improving that it would be best to enrich the Raw Data that the Pager Duty integration brings by addressing more Pager Duty APIS such as Business Service and Service Dependencies.

Moreover, to reduce the time, people gain value from the integration we would like to change the default resources of the integration to include more properties such as Slack Channel, Maintenance window, Dependent services and more.

Proposed changes

Add a new kind of Business Services to bring data from the matching API.
Enrich the data on Services to include the dependencies of other services using the Service Dependencies API
Add a new kind of aggregatedIncidentData with the following data

{
  "service_id": "PWER234D",
  "total_open_incidents": 12
}

Change the default blueprints to the following blueprints and update the config.yaml.

Blueprints

[
{
  "identifier": "pagerdutyService",
  "description": "This blueprint represents a PagerDuty service in our software catalog",
  "title": "PagerDuty Service",
  "icon": "pagerduty",
  "schema": {
    "properties": {
      "description": {
        "icon": "Book",
        "type": "string",
        "title": "Description"
      },
      "status": {
        "icon": "pagerduty",
        "title": "Status",
        "type": "string",
        "enum": [
          "active",
          "warning",
          "critical",
          "maintenance",
          "disabled"
        ],
        "enumColors": {
          "active": "green",
          "warning": "yellow",
          "critical": "red",
          "maintenance": "turquoise",
          "disabled": "lightGray"
        }
      },
      "url": {
        "icon": "pagerduty",
        "title": "URL",
        "type": "string",
        "format": "url"
      },
      "oncall": {
        "icon": "TwoUsers",
        "title": "On Call",
        "type": "string",
        "format": "user"
      },
     "numberOfOpenIncidents": {
       "type": "number",
       "title": "Open Incidents"
      },
      "communication_channel": {
        "icon": "Slack",
        "title": "Communication Channel",
        "type": "string",
        "format": "url"
      },
      "maintenance_start_time": {
        "icon": "DevopsTool",
        "title": "Maintenance Start Time",
        "type": "string",
        "format": "date-time"
      },
      "maintenance_end_time": {
        "icon": "DevopsTool",
        "title": "Maintenance End Time",
        "type": "string",
        "format": "date-time"
      },
      "team": {
        "title": "Team",
        "type": "string",
        "icon": "DefaultProperty"
      },
      "last_incident": {
        "icon": "Clock",
        "title": "Last Incident",
        "type": "string",
        "format": "date-time"
      },
      "escalation_policy_unique_level": {
        "title": "Escalation Policy Unique Level",
        "type": "number",
        "icon": "DefaultProperty"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {
    "uses": {
      "title": "Uses",
      "target": "pagerdutyService",
      "required": false,
      "many": true
    }
  }
},
{
  "identifier": "pagerdutyBusinessService",
  "description": "This blueprint represents a PagerDuty business service in our software catalog",
  "title": "PagerDuty Business Service",
  "icon": "pagerduty",
  "schema": {
    "properties": {
      "point_of_contact": {
        "icon": "TwoUsers",
        "type": "string",
        "title": "Point Of Contact",
        "format": "user"
      },
      "team": {
        "type": "string",
        "title": "Team",
        "icon": "DefaultProperty"
      },
      "description": {
        "type": "string",
        "title": "Description",
        "icon": "DefaultProperty"
      },
      "status_page": {
        "type": "string",
        "title": "Status Page",
        "format": "url",
        "icon": "DefaultProperty"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {
    "services": {
      "title": "Services",
      "target": "pagerdutyService",
      "required": false,
      "many": true
    }
  }
},
{
  "identifier": "pagerdutyIncident",
  "description": "This blueprint represents a PagerDuty incident in our software catalog",
  "title": "PagerDuty Incident",
  "icon": "pagerduty",
  "schema": {
    "properties": {
      "status": {
        "icon": "DefaultProperty",
        "title": "Incident Status",
        "type": "string",
        "enum": [
          "triggered",
          "annotated",
          "acknowledged",
          "reassigned",
          "escalated",
          "reopened",
          "resolved"
        ],
        "enumColors": {
          "triggered": "red",
          "annotated": "red",
          "acknowledged": "green",
          "reassigned": "green",
          "escalated": "red",
          "reopened": "red",
          "resolved": "green"
        }
      },
      "url": {
        "type": "string",
        "format": "url",
        "title": "Incident URL",
        "icon": "DefaultProperty"
      },
      "urgency": {
        "icon": "Alert",
        "title": "Incident Urgency",
        "type": "string",
        "enum": [
          "high",
          "low"
        ],
        "enumColors": {
          "high": "lightGray",
          "low": "lightGray"
        }
      },
      "assignee": {
        "icon": "TwoUsers",
        "type": "string",
        "title": "Assignee",
        "format": "user"
      },
      "escalation_policy": {
        "icon": "BlankPage",
        "type": "string",
        "title": "Escalation Policy"
      },
      "created_at": {
        "icon": "DefaultProperty",
        "title": "Created At",
        "type": "string",
        "format": "date-time"
      },
      "updated_at": {
        "title": "Updated At",
        "type": "string",
        "format": "date-time",
        "icon": "DefaultProperty"
      }
    },
    "required": []
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "relations": {
    "pagerdutyService": {
      "title": "PagerDuty Service",
      "target": "pagerdutyService",
      "required": false,
      "many": false
    }
  }
}
]

Screenshots

[Feature] [Ocean] Set the default timezone to `UTC`

Motivation

While playing around with jq I experienced inconsistent behavior for when I ran the jq query in https://jqplay.org/ vs when I ran it on my machine.

This inconsistency happens when working with dates.

Example:

As you can see with the same input and jq query I get different output.

The jq manual states that:

Low-level jq interfaces to the C-library time functions are also provided: strptime, strftime, strflocaltime, mktime, gmtime, and localtime. Refer to your host operating system's documentation for the format strings used by strptime and strftime. Note: these are not necessarily stable interfaces in jq, particularly as to their localization functionality.

Proposed Changes

Setting the environment variable TZ to UTC solves that issue

I think that ocean framework should be set to UTC by default and support overrides.

Are you willing to submit a PR?

I'm willing to submit a PR!

[Bug] [deployment] [aws] Terrafrom module aws/ecs - aws_iam_role.task_execution_role is to long

Hi all,

Encounter the next issue:

Planning failed. Terraform encountered an error while generating this plan.

Error: expected length of name to be in the range (1 - 64), got ecs-task-execution-role-port-ocean-launchdarkly-launchdarkly_ocean

with module.ocean_integration.module.port_ocean_ecs.aws_iam_role.task_execution_role,
on .terraform/modules/ocean_integration/deployment/terraform/aws/ecs/modules/ecs_service/main.tf line 130, in resource "aws_iam_role" "task_execution_role":
130: name = "ecs-task-execution-role-${local.service_name}"

that because:

ocean/deployment/terraform/aws/ecs/modules/ecs_service/main.tf

Line 130 in cfffc9a

name = "ecs-task-execution-role-${local.service_name}"

ocean/deployment/terraform/aws/ecs/modules/ecs_service/main.tf

Line 2 in cfffc9a

 service_name = "port-ocean-${var.integration.type}-${var.integration.identifier}" 

I used the integration type launchdarkly and the identifier launchdarkly_ocean as our internal convention, which leads to a length of 66 while the maximum is 64.

port-labs / ocean Goto Github PK

ocean's Introduction

Ocean

Prerequisites

Installation

Run Integration

Export Architecture

Real-Time updates Architecture

Integration Lifecycle

Folder Structure

Configuration

Contributing

Local Development (Framework)

Local Development (Integration)

License

Contact

ocean's People

Contributors

Stargazers

Watchers

Forkers

ocean's Issues

Ocean Framework Version

Integration Version

Steps to reproduce

What did you expect to see?

What did you see instead?

Are you willing to submit a PR?

Ocean Framework Version

Ocean Version

Helm Version

K8s Version

Integration Version

Steps to reproduce

What did you expect to see?

What did you see instead?

Are you willing to submit a PR?

Motiviation

Suggested Solution

Motivation

Proposed Changes

Are you willing to submit a PR?

Motivation

Proposed Changes

Are you willing to submit a PR?

Motivation

Proposed Changes

Are you willing to submit a PR?

Ocean Framework Version

Integration Version

Steps to reproduce

What did you expect to see?

What did you see instead?

Are you willing to submit a PR?

Motivation

Proposed Changes

Are you willing to submit a PR?

Ocean Framework Version

Integration Version

Steps to reproduce

What did you expect to see?

What did you see instead?

Are you willing to submit a PR?

Discussed in #837

Ocean Framework Version

Integration Version

Steps to reproduce

What did you expect to see?

What did you see instead?

Are you willing to submit a PR?

Motivation

Proposed Changes

Are you willing to submit a PR?

Ocean Framework Version

Integration Version

Steps to reproduce

What did you expect to see?

What did you see instead?

Are you willing to submit a PR?

Ocean Framework Version

`config.yaml`

`.port/spec.yaml`