labs42io / circleci-monorepo Goto Github PK

Hi, first of: thanks for the medium article + the repo!
Looks promising.

I was playing around with your setup and was wondering how you handle feature-branches. I hab a couple of commits on the branch which changed files in two packages. The script picked up the changes ok and ran the two jobs -> errors. Ok, lets fix them in the first one. Great fixed, but it didn't check the second package (build still failed) and on the PR page it said "all ok, no errors, you can merge into master ...".

My question: shouldn't it - when you are working on a branch - always check against parent branch to run tests for all packages that have changes compared to parent branch?

let say we have 2 branches development and master. we have checkout to branch development and make changes and push code. All is good here.
Now we want to merge our development branch to master branch, suppose that we added multiple features in development branch so we want to keep commit history.

When we merge development to master using

• squash, this work fine
• merge commit, here our monorepo.sh does not find any changes

The logs are

No config file found at /home/circleci/project/.circleci/monorepo.json. Using defaults.
Getting workflow status:
	GET: [200] https://circleci.com/api/v1.1/project/github/<user>/<repo>/tree/master?shallow=true&limit=100&offset=0
Created build-commit map /home/circleci/project/.circleci/temp/builds.json
First built commit in branch: 5fbce9392a203768006daa39617cf1a75bb37682
	Could not find parent commit relative to first build commit.
	Either branch was force pushed or build commit too old.
Parent commit: 8162e15

Thank you for your script. I adapted it to work with the rush monorepo manager.

I know this is not not an issue with the script per se, but right now, any secondary workflows that are triggered are using the CIRCLE_USER_TOKEN token which may not belong to the author of the original commit.

Any ideas on how to keep track of the original author?

Hi,
Thanks a lot for this. I am trying to run this but facing below issue:

There are no completed CI builds in branch master.
Searching for CI builds in branch 'master' ...
No CI builds for branch master. Using master.
Searching for changes since commit [master] ...

It seems the below line is not able to fetch the result from API
curl -Ss -u ${CIRCLE_TOKEN}: ${LAST_COMPLETED_BUILD_URL} > circle.json

I can verify that if I try to fetch using LAST_COMPLETED_BUILD_URL from browser, it fetches the expected result, but somehow not working during build.

I have double verified CIRCLE_TOKEN and it seems fine. Any idea as to where could be the issue?

Hi,

Many thanks for great project. I’m using the v2.

Is it possible to add a default workflow if a modified file does not match any of the packages defined in packages.json?

Some pre-amble:

I borrowed your script (Thank you so much) for a personal project of mine, and I wanted to give back with some feedback! I would also love to be involved with extending your work: I genuinely think the mono-repo is the way to go but it can be very difficult to get tools to play nicely with them! So yeah - thanks again!

Issue:

Github now encourages a default branch of 'main' instead of master, and I guess some people might prefer to use a different name (Who knows why??). So running your script for my new repo fails: it can't find master (my branch is called main)

Solutions:

Initially I thought maybe extracting the name of the main branch into a variable and allowing the user to set this in the script (or even as an Env var in circleci) would be useful, however on second thoughts I wonder if a slightly more robust way might be to query their remote for the "default branch from with .circleci/circle_trigger.sh.

Final comments:

I don't know the exact details of how to get this properly working (some googling led me to a solution that might work, but I'd anticipate hitting some caveats). I'd be happy to put a PR in if you're interested!

Edit: I don't understand bash well yet - so if this is already intended behaviour I may have missed it.

Hi,
we are using this repo for some time, and it's working great
a few weeks we move to use the v2 bash code (much better than the first one)
but we are starting to see that package.json change inside are packages not trigger a change in the workflow job
this is the summary we get

this is the full log
FullLog.txt

I just wanted to mention an issue that I ran into and a potential solution for others who might run into it as well.
I like to rebase my feature branches on to the target branch. This means I need to force push my changes. I ran into an issue with the last completed build SHA was no longer in the branch and caused git log to fail as the $LAST_COMPLETED_BUILD_SHA could not be found: fatal: bad object 2366eaabe7a5fe2f3ddf19fb5821e8a3073537d6
The solution I had was to just just the git log in and if to see if it would error. If it did, I would just move on as if there was no LAST_COMPLETED_BUILD_SHA

LAST_COMPLETED_BUILD_SHA=`curl -Ss -u "${CIRCLE_TOKEN}:" "${LAST_COMPLETED_BUILD_URL}" | jq -r 'map(select(.status == "success") | select(.workflows.workflow_name != "ci")) | .[0]["vcs_revision"]'`

-if  [[ ${LAST_COMPLETED_BUILD_SHA} == "null" ]]; then
+if  [[ ${LAST_COMPLETED_BUILD_SHA} == "null" ]] || ! $(git log -1 $CIRCLE_SHA1 ^$LAST_COMPLETED_BUILD_SHA --format=format:%H --full-diff ${PACKAGE_PATH#/}); then
  echo -e "\e[93mThere are no completed CI builds in branch ${CIRCLE_BRANCH}.\e[0m"

It works. Not sure if there is a better way to maybe try older completed build SHAs before jumping to the branches.

Hi

I got a weird issue where the script works perfectly fine when I am on a branch, all changes were detected correctly.

But then when I merge the branch back to master, the script doesn't seem to detect any changes hence master is not being built.

All I got is this, could you please help?

`There are no completed CI builds in branch master.

Searching for CI builds in branch 'master' ...

No CI builds for branch master. Using master.

Searching for changes since commit [cd3bf62] ...

Workflows currently in failed status: ().

[-] cart-web

[-] myaccount-web

[-] shared

[-] style

No changes detected in packages. Skip triggering workflows.`

The trigger script uses a <branch name>..<sha1> git revision range to obtain the LATEST_COMMIT_SINCE_LAST_BUILD. This revision range format does not work on CircleCI at this time.

https://stackoverflow.com/questions/61228078/git-rev-list-with-a-gitrevision-range-argument-on-circleci-build-not-working/61348399#61348399

Advanced logic in Config is a thing now in circleci, so, it should be documented, it wwould be a solution without change the script

the announcement is here

So, given 2 packages, api and service that depends on common, the script should be ok with

workflows:
  version: 2

  # The main workflow responsible for triggering all other workflows
  # in which changes are detected.
  ci:
    when: << pipeline.parameters.trigger >>
    jobs:
      - trigger-workflows


  # Workflows defined for each package.

  api:
    when: 
      or: 
        - << pipeline.parameters.api >>
        - << pipeline.parameters.common >>
    jobs:
      - build:
          name: api-build
          package_name: api
      - deploy:
          name: api-deploy
          package_name: api
          requires:
            - api-build

service:
    when: 
      or: 
        - << pipeline.parameters.service >>
        - << pipeline.parameters.common >>
    jobs:
      - build:
          name: service-build
          package_name: service
      - deploy:
          name: service-deploy
          package_name: service
          requires:
            - service-build

Right now I am running into the problem that the curl call to trigger the package workflow fails with a 404 response code:

Received status code: 404
Response:
{
  "message" : "Project not found"
}

The CIRCLE_TOKEN is set, so I am a bit clueless...

curl -s -v -u '****************************************:' -o response.txt -w '%{http_code}' -X POST --header 'Content-Type: application/json' -d '{ "branch": "build-docker-images", "parameters": { "trigger":false, "backoffice-helm":true } }' 'https://circleci.com/api/v2/project/github/********/pipeline'
*   Trying 18.208.34.245...
* TCP_NODELAY set
* Connected to circleci.com (18.208.34.245) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [89 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4838 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.circleci.com
*  start date: Nov 20 00:00:00 2019 GMT
*  expire date: Dec 20 12:00:00 2020 GMT
*  subjectAltName: host "circleci.com" matched cert's "circleci.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Server auth using Basic with user '****************************************'
} [5 bytes data]
> POST /api/v2/project/github/************/pipeline HTTP/1.1
> Host: circleci.com
> Authorization: Basic *************
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 94
> 
} [94 bytes data]
* upload completely sent off: 94 out of 94 bytes
{ [5 bytes data]
< HTTP/1.1 404 Not Found
< Content-Type: text/plain;charset=utf-8
< Date: Thu, 30 Jul 2020 16:34:21 GMT
< Server: nginx
< Set-Cookie: ring-session=**********;Path=/;HttpOnly;Expires=Fri, 30 Jul 2021 13:53:29 +0000;Max-Age=1209600;Secure
< Strict-Transport-Security: max-age=15724800
< X-CircleCI-Identity: circle-www-api-v1-75dfdb44f4-5m4dj
< X-Frame-Options: DENY
< X-RateLimit-Limit: 151
< X-RateLimit-Remaining: 150
< X-RateLimit-Reset: 0
< x-request-id: 9bad84c0-4a59-455c-a12c-35f9e19b94dd
< X-route: /api/v2/project/:project-slug/pipeline
< Content-Length: 37
< Connection: keep-alive
< 
{ [37 bytes data]
* Connection #0 to host circleci.com left intact
Received status code: 404
Response:
{
  "message" : "Project not found"
}

Thank you very much for providing such a great feature.

I am an apprentice front-end engineer living in Japan.
I am interested in mono repositories and I am glad to find this repository.

Here is the main topic.
I set up a Personal API Token in CircleCI, but when I run it in Bitbucket, it points out that the environment variable is not set.
When I run the same source code on Github, it works perfectly.

CIRCLE_TOKEN="Personal_API_TOKEN" // direct insertion

If you write the generated key in circle_trigger.sh and assign it directly as above, it works perfectly on Bitbucket.
Is there a difference in the way Github and Bitbucket are configured?

Attempting to use the example repo and get the following error;

+ ROOT=./packages
+ REPOSITORY_TYPE=github
+ CIRCLE_API=https://circleci.com/api
+ LAST_COMPLETED_BUILD_URL='https://circleci.com/api/v1.1/project/github/hashicorp/instruqt-packer/tree/fp/move_file?filter=completed&limit=100&shallow=true'
++ curl -Ss -u : 'https://circleci.com/api/v1.1/project/github/hashicorp/instruqt-packer/tree/fp/move_file?filter=completed&limit=100&shallow=true'
++ jq -r 'map(select(.status == "success") | select(.workflows.workflow_name != "ci")) | .[0]["vcs_revision"]'
jq: error (at <stdin>:2): Cannot index string with string "status"
+ LAST_COMPLETED_BUILD_SHA=```