oss-know / airflow-jobs Goto Github PK

Currently the github profile init DAG only stores data in opensearch, and then run a ck_transfer DAG to copy all github profile data in opensearch to clickhouse. So when the dag run again, it will copy duplicated data to clickhouse. It's better to transfer clickhouse data at the same time(within one DAG).

The design:
Each developer's profile has an updated_at field, indicating when the profile is lastly modified. So select data in opensearch whose updated_at is greater than "latest updated_at in clickhouse", then insert them into clickhouse, is approximately enough.

Some data, like github commits, pr, issues are fetched and inserted by pages. Data in one page may not be as large as a batch. Setup a batch and collect these data into batch, when batch full, insert the batch data into opensearch

@fivestarsky #51

There are many different forms of code review/merging/maintaining mechanisms in opensource community. Like Apache foundation's incubation system; Linux kernel's mailing list discussion and patch review/merging; GitHub/GitLab's issue/PR/comments discussion collaboration. So we need to design a universal code review workflow, which obtains the whole process of code being written, discussed/reviewed, modified, and merged, then different code collaboration systems can be applied to the universal template. And any future forms can be adapted.

Optimize the code and structure of data analysis DAG.

It's super weird that the daily sync scheduling only works for gits/commits/pr, but not issues and related comments/timeline. Check if it's a bug of airflow, or misconfiguration.

Currently the table creation DAG create tables for basic data, like gits, github commits, issue, pr etc. More higher level table like gits_dir, metrics etc should also be created. To avoid creating tables manually.

The PyCharm IDE automatically formatted long SQL strings in the commit 7ea4e330bd3350bfac083569f2883a2d29de5ba8, while some becomes invalid. Check all the SQLs and re-claim them in a valid way.

GitHub repos might be removed after downloading. Then the request to that repos will be NotFound. Maybe we should come up with a solution to provide default/empty GitHub resources.

A current example, the get_github_pull_requests function in libs/util/github_api.py (@line 127) will fail if the response is NotFound.

When resolving issue #189 , we found that the github profile can be configure with new engine that will remove duplication, by keeping (github id, updated_at) fields tuple. Which defines a snap shot of a profile at a particular time.

So after changing the new engine, it is possible to sync github profile from remote clickhouse serivce by compare the search_key__updated_at and the new engine will keep removing duplication.

When the target server response 50x error, currently the base function just returns an empty list. For some modules that expect profile, it will cause error.

Make it a factory class, generate accommodator by different proxy service providers.

The opensearchpy library will print a line of log on each request like this:

[2023-04-25, 09:01:43 UTC] {base.py:270} INFO - POST https://dev-opensearch-node1:9200/gits/_delete_by_query [status:200 request:0.006s]

Which brings huge amount of useless text, consider turn off the logs by opensearchpy settings.

When a github repository is renamed, the current git_track_repo dag's issue timeline/comments task will never finish(and calling the api in a pretty high pace).

Daily sync DAGs needs to be scheduled by different intervals to balance resource consuming. The policy is to 2 level intervals in variables with different priority:

high: daily_sync_{DAG_NAME}_interval
middle: daily_sync_interval

If the interval variables above is not configured, do not trigger the DAG periodically.

Current github profiles init is done in a serial way, sending http request to timeline/commit API one after another, use executor pool to send multiple requests in parallel for better performance.

这是一个测试用issues

Both https://github.com/OWNER/REPO.git and https://github.com/OWNER/REPO are valid git urls. When specify 'includes' variable for daily sync, and its origin is different from the data in OpenSearch, like opensearch doc's origin contains '.git' suffix while 'includes' doesn't. Then there will be 2 different (owner, repo, origin) tuples.

If then we do a full repo daily sync, the 2 tuples will be considered as 2 code bases, and will sync data separately, introducing redundant data into OpenSearch, then to ClickHouse.

The solution is to make sure to eliminate the '.git' suffix before init or sync data.

Daily sync DAGs needs to be scheduled by different intervals to balance resource consuming. The policy is to 2 level intervals in variables with different priority:

high: daily_sync_{DAG_NAME}_interval
middle: daily_sync_interval

If the interval variables above is not configured, do not trigger the DAG periodically.

When resolving issue #189 , we found that the github profile can be stored with new engine that will remove duplication, by keeping (github id, updated_at) fields tuple distinct. Which indeed defines a snap shot of a profile at a particular time.

So after changing the new engine, it is possible to sync github profile from remote clickhouse serivce by compare the search_key__updated_at and the new engine will keep removing duplication.