git-xargs seems to be exactly what I need to use. I've been testing it to update files and automatically generate pull requests.

What I have seen is that the pull requests generated are always between the branch specified with "--branch-name" and "master".

Is it possible to specify that target branch to be different to "master"? Or a way to specify that the target should be the default branch (which is not always "master" on some of our repos).

Thanks.

Last time I used git-xargs was in May of this year, with the v0.0.14. And everything was fine.
I recently upgraded all my brew packages and nothing seems to works with the v0.0.16.

Describe the bug

When using --github-org to scan my company's org, git-xargs resolves only 25 of the few hundreds we have.

$ git-xargs --version
git-xargs version v0.0.16

$ git-xargs --loglevel DEBUG \
            --pull-request-title "Some title" \
            --commit-message "Doing things" \
            --branch-name "fix/my-things" \
            --max-concurrent-repos 1 \
            --github-org <ORG> --skip-archived-repos --dry-run touch helloworld.txt

[git-xargs] INFO[2022-07-20T16:56:14+02:00] git-xargs running...                         
[git-xargs] INFO[2022-07-20T16:56:14+02:00] Dry run setting enabled. No local branches will be pushed and no PRs will be opened in Github 
[git-xargs] DEBU[2022-07-20T16:56:15+02:00] Skipping archived repository                  Name=<ORG>/***
[git-xargs] DEBU[2022-07-20T16:56:15+02:00] Skipping archived repository                  Name=<ORG>/***
[git-xargs] DEBU[2022-07-20T16:56:15+02:00] Skipping archived repository                  Name=<ORG>/***
[git-xargs] DEBU[2022-07-20T16:56:15+02:00] Fetched repos from Github organization: <ORG>  Repo count=25
[git-xargs] DEBU[2022-07-20T16:56:15+02:00] Using Github org: <ORG> as source of repositories. Paging through Github API for repos. 
[...]
*****************************************************************
  GIT-XARGS RUN SUMMARY @ 2022-07-20 15:07:32.722381 +0000 UTC
  Runtime in seconds: 51
*****************************************************************


COMMAND SUPPLIED

[touch helloworld.txt]

REPO SELECTION METHOD USED FOR THIS RUN - (see README.md for more information)

github-org

 REPOS SUCCESSFULLY FETCHED VIA GITHUB API
| REPO NAME (22)          | REPO URL                                          |
|-------------------------|---------------------------------------------------|

To Reproduce

• Generate Personal Token
• • With repo:*, read:org, write:org, and read:project
• Execute git-xargs with the --github-org flag

Expected behavior

For git-xargs to actually follow up on the API pagination.

Nice to have

• Terminal output
• Screenshots

Additional context

Edit: ℹ️ This one was coming from an improper permission set on the token
I also tried supplying a list via --repos but every call ends up in a 404.

$ cat repos.txt | wc -l
      81
$ git-xargs --loglevel DEBUG \
            --pull-request-title "Some title" \
            --commit-message "Doing things" \
            --branch-name "fix/my-things" \
            --max-concurrent-repos 1 \
            --repos repos.txt --skip-archived-repos --dry-run touch helloworld.txt
[git-xargs] INFO[2022-07-20T17:17:02+02:00] git-xargs running...                         
[git-xargs] INFO[2022-07-20T17:17:02+02:00] Dry run setting enabled. No local branches will be pushed and no PRs will be opened in Github 
[...]
[git-xargs] DEBU[2022-07-20T17:17:02+02:00] Looking up filename provided repo             Name=documents Organization=<ORG>
[git-xargs] DEBU[2022-07-20T17:17:04+02:00] error getting single repo                     AllowedRepoName=documents AllowedRepoOwner=PayFit Error="GET https://api.github.com/repos/<ORG>/documents: 404 Not Found []" Response Status Code=404
[...]


$ gh repo view <ORG>/documents 
<ORG>/documents

[SHOWS THE README]

Currently, the tool runs in "headless" mode. You can either feed it:

* A Github organization via the `--github-org flag`
* A local file that contains repos, one on in each line, in gruntwork-io/fetch format via the `--repos` flag
* One or many repos explicitly on the command line via `--repo` 

However, whether you run it in --dry-run mode (to only see proposed changes without having them made) or you run it in normal mode such that changes are made, PRs are opened, etc, you have to sit there passively and watch the output stream by.

It would be nice to have the option to run the tool in "interactive mode" where each repo and the rich diff mentioned in #92 would be shown in your terminal and allow you to type [Y\n] to either proceed with or skip making the upgrade / change.

This would also give the tool broader appeal for folks less comfortable with the command line in general, perhaps leveraging something like https://pterm.sh/#/

It would be great if I could add reviewers (teams and single persons) to the pull request git-xargs opens.
This way my reviewers will automatically see their pull requests.

https://docs.github.com/en/rest/reference/pulls#request-reviewers-for-a-pull-request

https://github.com/asottile/all-repos

The CircleCI job for git-xargs currently sets the main go package's VERSION variable to the $CIRCLE_TAG which is either the branch name or the tag name.

This means that when we build and publish binaries through CircleCI and programmatically attach them to a new release, they correctly have their internal VERSION value set to the tag of the release and, therefore, when you run git-xargs --version against them you get back the associated tag, e.g., git-xargs version v0.0.5.

However, when you install git-xargs via go get github.com/gruntwork-io/git-xargs, the package is downloaded from Github and then built in your defined or default $GOBIN directory. In this case, the main package's VERSION value remains unset, so running git-xargs with the --version flag returns an error.

One initial idea would be to update a VERSION file in the repo each time we do a release - so that the HEAD of the default branch contains a VERSION file which always matches the latest release's tag.

In some places, git-xargs is forced to fall back to reading the error message value from GitHub's API responses, since the GitHub API can be inconsistent in what headers, status codes, etc it actually returns under various conditions.

Per this comment we should refactor all such instances, of error messages that we're keying off to determine some runtime status, into a new constants.go file so that they will be Easier to Change in the future.

Would you accept a PR to move cmd/* into cmd/git-xargs/* ? This way a simple go get .../cmd/git-xargs is all that is needed (once #14 is merged) to get the git-xargs binary.

Is there an interest for having this as a gitlab integration as well?

Formulated differently, is there a need and/or interest for a PR extending the functionality to gitlab (on premise or SaaS). I might be able to scramble something together regarding knowledge of the API but I'm not very well versed in Go atm.

git-xargs version
What does git-xargs --version show?
v0.0.10
Describe the bug
A clear and concise description of what the bug is.
I'm trying to push a blank file via git-xargs on a repository that I own.

To Reproduce
Steps to reproduce the behavior:

1. Run this command...
   export GITHUB_OAUTH_TOKEN=xxx

git-xargs --repo marshall7m/terraform-aws-github-ci --branch-name master --commit-message "add blank file" --skip-pull-requests touch foo.txt

2. See error:

[git-xargs] INFO[2021-06-02T09:41:25-07:00] git-xargs running...                         


*****************************************************************
  GIT-XARGS RUN SUMMARY @ 2021-06-02 16:41:26.996364 +0000 UTC
  Runtime in seconds: 1
*****************************************************************


COMMAND SUPPLIED

[touch foo.txt]


 ALL REPOS THAT WERE TARGETED FOR PROCESSING AFTER FILTERING MISSING / MALFORMED REPOS
│─────────────────────────│───────────────────────────────────────────────────────│
│ REPO NAME               │ REPO URL                                              │
│─────────────────────────│───────────────────────────────────────────────────────│
│ terraform-aws-github-ci │ https://github.com/marshall7m/terraform-aws-github-ci │
│─────────────────────────│───────────────────────────────────────────────────────│


 REPOS THAT WERE SUCCESSFULLY CLONED TO THE LOCAL FILESYSTEM
│─────────────────────────│───────────────────────────────────────────────────────│
│ REPO NAME               │ REPO URL                                              │
│─────────────────────────│───────────────────────────────────────────────────────│
│ terraform-aws-github-ci │ https://github.com/marshall7m/terraform-aws-github-ci │
│─────────────────────────│───────────────────────────────────────────────────────│


 REPOS FOR WHICH CHECKING OUT A NEW TOOL-SPECIFIC BRANCH FAILED
│─────────────────────────│───────────────────────────────────────────────────────│
│ REPO NAME               │ REPO URL                                              │
│─────────────────────────│───────────────────────────────────────────────────────│
│ terraform-aws-github-ci │ https://github.com/marshall7m/terraform-aws-github-ci │
│─────────────────────────│───────────────────────────────────────────────────────│

Expected behavior
A clear and concise description of what you expected to happen.

Expected foo.txt to be pushed to target repo's master branch
Additional context
Add any other context about the problem here.

I think, git-xargs should always open PRs even if the given command did not change anything locally or if the push returns already up-to-date.
The reason why I use your great tool, is that I want to merge some changes in to my main-branch. I don't care if the tool does these changes, commit them and push them to the branch I passed to the tool, or if these changes already exists in the remote branch, as long as a PR is opened so I can merge my changes into main.

Title says it all.

From Jim:

feature request that will most likely come up in the future is support for shell expansion. That is, given a command foo, the actual command you'd run here is something like bash -c "foo" (on *nix, at least). And then, of course, users will want to be able to configure the shell, and so on... Not worth tackling now. Perhaps just file a bug and come back to it.

Describe the solution you'd like
I would like to create a Backstage catalog-info.yaml file for each repository in an organization. This file needs to be unique for each repository. I would be helpful to be able to read name, description, owner.login, nameWithOwner for each repository while generating this file.

Describe alternatives you've considered
I'm not really sure what information is available while executing the script.

Additional context
Thank you

Describe the solution you'd like
To prevent large git-xargs jobs from generating excessive and potentially expensive continuous integration (CI) jobs, we should prepend [skip ci] to every commit message. We should also provide a new flag --no-skip-ci that allows users to opt-out of this behavior for the given run.

Describe the bug

Executing the touch command (as in the example) on any of my git repositories results in an error "208:22: unknown escape sequence (and 29 more errors)".

To Reproduce

My command:

git-xargs --loglevel debug  --repo hsf-training/organization --branch-name "test-branch"   --commit-message "Testing git-xargs"  touch "test123.test"   

The output is:

[git-xargs] INFO[2022-07-13T17:44:09-04:00] git-xargs running...
[git-xargs] DEBU[2022-07-13T17:44:09-04:00] Looking up filename provided repo             Name=organization Organization=hsf-training
[git-xargs] DEBU[2022-07-13T17:44:09-04:00] Successfully fetched repo                     Name=organization Organization=hsf-training
[git-xargs] DEBU[2022-07-13T17:44:09-04:00] Repo will have all targeted scripts run against it  Repository=organization
[git-xargs] DEBU[2022-07-13T17:44:09-04:00] Attempting to clone repository using GITHUB_OAUTH_TOKEN  Repo=organization
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Compressing objects: 100% (4/4), done.
Total 6 (delta 1), reused 0 (delta 0), pack-reused 0  Repo=organization
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Created branch                                Branch Name=refs/heads/test-branch Repo=organization
[git-xargs] DEBU[2022-07-13T17:44:10-04:00]                                               Repo=organization
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Executing command against local clone of repo...  Command="[touch test123.test]" Directory=/var/folders/g3/1y2_hl1j16scls01cxqm76s80000gn/T/git-xargs-organization653630454 Repo=organization
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Output of command [touch test123.test] for repo organization in directory /var/folders/g3/1y2_hl1j16scls01cxqm76s80000gn/T/git-xargs-organization653630454:
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Local repository worktree no longer clean, will stage and add new files and commit changes  Repo=organization
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Found untracked file. Adding to stage         Filepath=test123.test
[git-xargs] DEBU[2022-07-13T17:44:10-04:00] Error encountered while processing repo       Error="208:22: unknown escape sequence (and 29 more errors)" Repo name=organization


*****************************************************************
  GIT-XARGS RUN SUMMARY @ 2022-07-13 21:44:10.167923 +0000 UTC
  Runtime in seconds: 0
*****************************************************************


COMMAND SUPPLIED

[touch test123.test]

REPO SELECTION METHOD USED FOR THIS RUN - (see README.md for more information)

and this happens for me for all repositories that I have tried.

Versions:

• git-xargs 0.0.16

It would be great to have an option that make git-xargs open the PRs in draft mode. Sadly the GitHub Rest-API does not provide a resource to change the draft-state after creating a PR, so the only possibility is to do this at creation time.

Describe the solution you'd like
With #49 , we'll have introduced support for draft pull requests. Ideally, we'd have test coverage for this feature as well, using any mocks that make sense.

Describe alternatives you've considered
The alternative is to live without test coverage for this feature 😄

Additional context
N/A.

Log level is already configurable, but show an example in the README.

Per this question: #62 (comment)

It would be useful to be able to pass a flag to git-xargs that causes all Github API response headers to be dumped to STDOUT to ease debugging:

Something like:

• --dump-github-response-headers
• --show-response-headers

So that these dependencies don't go out of date too quickly:

Also worth bumping Go 1.17 now.

Describe the solution you'd like
It is often useful to submit signed commits to repos, it would be good if git-xargs could enable this (likely by a feature flag)

Describe alternatives you've considered
There might be a way to change my system defaults to always do this? It would depend on whether you're using the command line git tool or golang packages for git interaction, and I haven't dug into the code

Additional context
Great tool. Thanks for making and maintaining it!

Is your feature request related to a problem? Please describe.

I run git-xargs on a list of 500+ repositories and also from Github Actions. Running it on that scale logs a lot of information when cloning repositories.

For example:

Compressing objects: 100% (39/39)
Compressing objects: 100% (39/39), done.
Total 59 (delta 21), reused 35 (delta 12), pack-reused 0
Enumerating objects: 20, done.
Counting objects: 20% (1/5)
Counting objects: 40% (2/5)
Counting objects: 60% (3/5)
Counting objects: 80% (4/5)
Counting objects: 100% (5/5)
Counting objects: 100% (5/5), done.
Compressing objects: 33% (1/3)
Compressing objects: 66% (2/3)
Compressing objects: 100% (3/3)
Compressing objects: 100% (3/3), done.
Total 20 (delta 0), reused 0 (delta 0), pack-reused 15
Enumerating objects: 835, done.
Counting objects: 0% (1/111)
Counting objects: 1% (2/111)
Counting objects: 2% (3/111)
Counting objects: 3% (4/111)

Describe the solution you'd like

Would it be possible to move that information to a debug log for example or add a --quiet option? (I think the former solution is preferable because --quiet would have a much bigger scope for the tool in general.)

Describe alternatives you've considered

Send progress to the logger instead of stdout

Additional context
N/A

The PR was opened successfully: https://github.com/gruntwork-io/terraform-aws-ci/pull/448, but this error was still thrown. Also the report didn't get generated correctly.

git-xargs \
--loglevel debug \
--repo gruntwork-io/terraform-aws-ci \
--branch-name update-pr-template9 \
--seconds-between-prs 3 \
~/Development/repos/prototypes/git-xargs-assets/scripts/pull-request-template.sh
[git-xargs] INFO[2022-05-25T10:40:03-07:00] git-xargs running...
[git-xargs] DEBU[2022-05-25T10:40:03-07:00] Looking up filename provided repo             Name=terraform-aws-ci Organization=gruntwork-io
[git-xargs] DEBU[2022-05-25T10:40:03-07:00] Successfully fetched repo                     Name=terraform-aws-ci Organization=gruntwork-io
[git-xargs] DEBU[2022-05-25T10:40:03-07:00] Repo will have all targeted scripts run against it  Repository=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:03-07:00] Attempting to clone repository using GITHUB_OAUTH_TOKEN  Repo=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:05-07:00] Enumerating objects: 10181, done.
Counting objects: 100% (231/231), done.
Compressing objects: 100% (136/136), done.
Total 10181 (delta 105), reused 199 (delta 89), pack-reused 9950  Repo=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:05-07:00] Created branch                                Branch Name=refs/heads/update-pr-template9 Repo=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:05-07:00]                                               Repo=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:05-07:00] Executing command against local clone of repo...  Command="[/Users/rhozen/Development/repos/prototypes/git-xargs-assets/scripts/pull-request-template.sh]" Directory=/var/folders/l0/nr3fgm0j39zchyhxy4gpsyz80000gn/T/git-xargs-terraform-aws-ci385613384 Repo=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:05-07:00] Output of command [/Users/rhozen/Development/repos/prototypes/git-xargs-assets/scripts/pull-request-template.sh] for repo terraform-aws-ci in directory /var/folders/l0/nr3fgm0j39zchyhxy4gpsyz80000gn/T/git-xargs-terraform-aws-ci385613384:
[git-xargs] DEBU[2022-05-25T10:40:05-07:00] Local repository worktree no longer clean, will stage and add new files and commit changes  Repo=terraform-aws-ci
[git-xargs] DEBU[2022-05-25T10:40:06-07:00] Successfully pushed local branch to remote origin  Repo=terraform-aws-ci
[git-xargs] INFO[2022-05-25T10:40:06-07:00] Repository successfully processed             Repo name=terraform-aws-ci
ERROR: sync: WaitGroup is reused before previous Wait has returned

Command that was run:

#!/usr/bin/env bash 

set -e

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

cp -r "$SCRIPT_DIR/../data/github-templates/pr-template.md" "./.github/pull_request_template.md"

git-xargs version
git-xargs version v0.0.11

Describe the bug

Logs here. When running git-xargs, I get the error "author field is required" during the commit step. This seems to be true of any repo I try with git-xargs.

It looks related to my git config includes. I use a different email addresses for git at work and use includes to specify my user.name and user.email on different machines. My ~/.gitconfig contains:

# ~/.gitconfig

[include]
    path = ~/.gitconfig.local

# no [user] section exists

and

# ~/.gitconfig.local
[user]
    name = ...
    email = ...

I've never had any issue with this before, but notice this in the git-config man pages. It implies that git-config might not honor includes in some cases and that you might have to include an explicit option.

       --[no-]includes
           Respect include.*  directives in config files when looking up values.
           Defaults to off when a specific file is given (e.g., using --file,
           --global, etc) and on when searching all config files.

Just to make sure the includes are read properly by git:

$ git config --get user.name
Aaron D Borden

$ git config --global --get user.name
# exit 1

$ git config --global --includes --get user.name
Aaron D Borden

To Reproduce
Steps to reproduce the behavior:

1. Move your user.name and user.email from your ~/.gitconfig, to ~/.gitconfig.local
2. Add an [includes] section as above in ~/.gitconfig to include ~/.gitconfig.local
3. I'm using a script to generate templates for git-xargs, it looks similar to this:

git-xargs \
  --branch-name tts-bot/update-templates \
  --loglevel "${LOGLEVEL}" \
  --repos "${template_dir}/repo_list" \
  --skip-archived-repos \
  --pull-request-title "${pull_request_title}" \
  --pull-request-description "$(template pull_request_description)" \
  --commit-message "$(template commit_message)" \
  $(pwd)/bin/ci_update_templates.sh "${canonical_repository}" "${template_dir}/template_files"

repos file contains the single repo 18F/ghad.

Expected behavior
git-xargs respects the git config include and attributes the author correctly AND git-xargs warns when it cannot identify the author for the commit.

Screenshots
Logs https://gist.github.com/adborden/f0000d75107e1971940d48b58bc710be

Additional context
To work-around, I confirmed that you can set the user.name and user.email in the main config and git-xargs works as expected.

Describe the solution you'd like
Use pterm.sh to clean up / unify the run report git-xargs prints when all work is complete.

Describe the bug
The latest version, 0.0.15, no longer creates PRs. I tried same command with 0.0.14 binary and it works fine.

To Reproduce
Steps to reproduce the behavior including the relevant Terraform/Terragrunt/Packer version number and any code snippets and module inputs you used.

	./git-xargs   --branch-name mkow-test4  --repos ./java-apps.txt   --commit-message "My message"   --pull-request-title "My title"   --loglevel DEBUG   `pwd`/scripts/myScript.sh

Expected behavior
The PR should be created.

Nice to have

• Terminal output
• Screenshots

Additional context
Add any other context about the problem here.

Teminal output:

$ git-xargs   --branch-name mkow-test2  --repos ./java-apps.txt   --commit-message "My message"   --pull-request-title "My title"   --loglevel DEBUG   ***/updateServiceInfra.sh
[git-xargs] INFO[2022-06-23T14:48:59-04:00] git-xargs running...
[git-xargs] DEBU[2022-06-23T14:48:59-04:00] Looking up filename provided repo             Name=cplus-messaging-application Organization=****
[git-xargs] DEBU[2022-06-23T14:48:59-04:00] Successfully fetched repo                     Name=cplus-messaging-application Organization=***
[git-xargs] DEBU[2022-06-23T14:48:59-04:00] Repo will have all targeted scripts run against it  Repository=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:48:59-04:00] Attempting to clone repository using GITHUB_OAUTH_TOKEN  Repo=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:49:00-04:00] Enumerating objects: 951, done.
Counting objects: 100% (130/130), done.
Compressing objects: 100% (78/78), done.
Total 951 (delta 44), reused 106 (delta 28), pack-reused 821  Repo=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:49:00-04:00] Created branch                                Branch Name=refs/heads/mkow-test2 Repo=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:49:00-04:00]                                               Repo=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:49:01-04:00] Executing command against local clone of repo...  Command="[***updateServiceInfra.sh]" Directory=***/workspace/git-xargs-cplus-messaging-application1237588432 Repo=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:49:31-04:00] Output of command [***/scripts/updateServiceInfra.sh] for repo cplus-messaging-application in directory ***/workspace/git-xargs-cplus-messaging-application1237588432:
[git-xargs] DEBU[2022-06-23T14:49:31-04:00] Local repository worktree no longer clean, will stage and add new files and commit changes  Repo=cplus-messaging-application
[git-xargs] DEBU[2022-06-23T14:49:32-04:00] Successfully pushed local branch to remote origin  Repo=cplus-messaging-application
[git-xargs] INFO[2022-06-23T14:49:32-04:00] Repository successfully processed             Repo name=cplus-messaging-application


*****************************************************************
  GIT-XARGS RUN SUMMARY @ 2022-06-23 18:49:32.233789 +0000 UTC
  Runtime in seconds: 32
*****************************************************************


COMMAND SUPPLIED

[/***/updateServiceInfra.sh]

REPO SELECTION METHOD USED FOR THIS RUN - (see README.md for more information)

repos-file
 REPOS SUPPLIED VIA --repos FILE FLAG
│───────────────────│─────────────────────────────│
│ ORGANIZATION NAME │ URL                         │
│───────────────────│─────────────────────────────│
│ ***      │ cplus-messaging-application │
│───────────────────│──────────────────[git-xargs] DEBU[2022-06-23T14:49:32-04:00] pullRequestWorker received pull request job. Delay: 0 seconds. Retries: 0 for repo: cplus-messaging-application on branch: refs/heads/mkow-test2
[git-xargs] DEBU[2022-06-23T14:49:32-04:00] openPullRequest received job with retries: 0. Config max retries for this run: 3
───────────│

 ALL REPOS THAT WERE TARGETED FOR PROCESSING AFTER FILTERING MISSING / MALFORMED REPOS
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ REPO NAME                   │ REPO URL                                                    │
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ cplus-messaging-application │ https://***/cplus-messaging-application │
│─────────────────────────────│─────────────────────────────────────────────────────────────│


 REPOS THAT WERE SUCCESSFULLY CLONED TO THE LOCAL FILESYSTEM
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ REPO NAME                   │ REPO URL                                                    │
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ cplus-messaging-application │ https://***/cplus-messaging-application │
│─────────────────────────────│─────────────────────────────────────────────────────────────│


 REPOS THAT SHOWED FILE CHANGES TO THEIR WORKING DIRECTORY FOLLOWING COMMAND EXECUTION
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ REPO NAME                   │ REPO URL                                                    │
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ cplus-messaging-application │ https://***/cplus-messaging-application │
│─────────────────────────────│─────────────────────────────────────────────────────────────│


 REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ REPO NAME                   │ REPO URL                                                    │
│─────────────────────────────│─────────────────────────────────────────────────────────────│
│ cplus-messaging-application │ https://***/cplus-messaging-application │
│─────────────────────────────│─────────────────────────────────────────────────────────────│

Note the async log output from the PR creation thread in the report.

Describe the bug
Using git-xargs with 10+ repositories, I end up with only 10 PRs (consistently between runs, I tried running the command multiple times)

To Reproduce
Steps to reproduce the behavior including the relevant Terraform/Terragrunt/Packer version number and any code snippets and module inputs you used.
Running the git-xargs command with default settings produces only 10 PRs.
For example I ran the following:

git-xargs \
  --repos $PWD/tools/git-xargs/dbt_v1/repos \
  --branch-name branch-name-here \
  --loglevel DEBUG \
  --commit-message "My Commit Message" \
  --pull-request-description "DESCRIPTION HERE" \
  --draft \
  --dry-run \
  -- $PWD/tools/git-xargs/dbt_v1/script.sh

The output looks like this:

REPOS AGAINST WHICH PULL REQUESTS FAILED TO BE OPENED
│──────────────────────────────│────────────────────────────────────────────────────────────│
│ REPO NAME │ REPO URL │
│──────────────────────────────│────────────────────────────────────────────────────────────│
│ my-11th-repo │ my-11th-repo-url │
│──────────────────────────────│────────────────────────────────────────────────────────────│

REPOS WHOSE SPECIFIED BRANCHES DID NOT EXIST ON THE REMOTE, AND SO WERE FIRST CREATED LOCALLY
│──────────────────────────────────────│────────────────────────────────────────────────────────────────────│
│ REPO NAME (11) │ REPO URL │
│──────────────────────────────────────│────────────────────────────────────────────────────────────────────│
...I omitted content here...
│──────────────────────────────────────│────────────────────────────────────────────────────────────────────│

PULL REQUESTS OPENED

│──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│
│ REPO NAME (10) │ PR URL │
│──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│
...I omitted content here...
│──────────────────────────────────────│────────────────────────────────────────────────────────────────────────────│

// paste code snippets here

Expected behavior
I'm expecting 10+ PRs to be created, however only 10 were created. I can see that on the 11th repository a branch was created with the right content, however no PR.

Can it be related to the tool itself or a policy I have in my organization?

Nice to have

• Terminal output
• Screenshots

Additional context
Add any other context about the problem here.

THANKS!!!
Eyal

This looks useful to us if it supported bitbucket.

Thank you!

Describe the bug
When I try to apply a "command" to a list of reporitories it works perfectly on all of them except for one that return the error in attached.

Error="worktree contains unstaged changes"

To Reproduce

I can't reproduce it manually, I try to clone the project and checkout a new branch with this command git checkout -b "core/update" but I can't reproduce the error "manually".

Expected behavior

Normal behavior, so checkout and apply the command.

git-xargs version
What does git-xargs --version show?

git-xargs version v0.0.11

Describe the bug
git-xargs is not respecting .gitignore

To Reproduce
Steps to reproduce the behavior:

#!/usr/bin/env bash

git-xargs \
    --commit-message "Update Yarn Berry to latest canary" \
    --repos ../repos.txt \
    --branch-name chore/update-yarn-berry-canary \
    --loglevel debug \
    "$(pwd)/update-yarn-berry-canary.sh"

#!/usr/bin/env bash

# update-yarn-berry-canary.sh

yarn set version canary
yarn

That results in files being added to node_modules when, in my .gitignore in those projects, I have node_modules/ listed.

Expected behavior

Respect the .gitignore (in this case, node_modules/)

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Docs seem to suggest that you can skip pull request and commit straight to master. --skip-pull-requests with branch name as master give the following error

Error="a branch named \"refs/heads/master\" already exists"

Hello,

found something which is probably a bug or at least unexpected behavior.

If, for some reason the changes I made with git-xargs exists already in the branch on github, the tool prints the following errors and does not open a PR.

[git-xargs] DEBU[2021-04-27T08:45:41+02:00] Error pushing new branch to remote origin     Error="already up-to-date" Repo=room-management-example
[git-xargs] DEBU[2021-04-27T08:45:41+02:00] Error encountered while processing repo       Error="already up-to-date" Repo name=room-management-example

I run into this error, because I run my git-xargs command two times. The first time I run into an error on opening the PR. After fixing this, I run the command a second time and failed on the update. I had to delete the remote branch first.
I think it the tool should not break on an "already up-to-date" message, because the state of the repository is like it should be, only the way to achived was different.

Think this is an edge case and nothing urgent.

git-xargs version
v0.0.11

Describe the bug
If both --github-org and --repo are specified, git-xargs is still cloning all the repos in the org.

To Reproduce

$ git-xargs --branch-name=test-branch --github-org=git-xargs-test  --repo=test-repo-4  touch git-xargs-is-awesome.txt
[git-xargs] INFO[2021-09-21T15:43:26+02:00] git-xargs running...
[git-xargs] INFO[2021-09-21T15:43:27+02:00] Repository successfully processed             Repo name=test-repo-3
[git-xargs] INFO[2021-09-21T15:43:27+02:00] Repository successfully processed             Repo name=test-repo-2
[git-xargs] INFO[2021-09-21T15:43:27+02:00] Repository successfully processed             Repo name=test-repo-1


*****************************************************************
  GIT-XARGS RUN SUMMARY @ 2021-09-21 13:43:27.755367 +0000 UTC
  Runtime in seconds: 1
*****************************************************************


COMMAND SUPPLIED

[touch git-xargs-is-awesome.txt]


 REPOS SUCCESSFULLY FETCHED VIA GITHUB API
│────────────────│───────────────────────────────────────────────│
│ REPO NAME (4)  │ REPO URL                                      │
│────────────────│───────────────────────────────────────────────│
│ test-repo-1    │ https://github.com/git-xargs-test/test-repo-1 │
│ test-repo-2    │ https://github.com/git-xargs-test/test-repo-2 │
│ test-repo-3    │ https://github.com/git-xargs-test/test-repo-3 │
│ test-repo-4    │ https://github.com/git-xargs-test/test-repo-4 │
│────────────────│───────────────────────────────────────────────│


 ALL REPOS THAT WERE TARGETED FOR PROCESSING AFTER FILTERING MISSING / MALFORMED REPOS
│────────────────│───────────────────────────────────────────────│
│ REPO NAME (4)  │ REPO URL                                      │
│────────────────│───────────────────────────────────────────────│
│ test-repo-1    │ https://github.com/git-xargs-test/test-repo-1 │
│ test-repo-2    │ https://github.com/git-xargs-test/test-repo-2 │
│ test-repo-3    │ https://github.com/git-xargs-test/test-repo-3 │
│ test-repo-4    │ https://github.com/git-xargs-test/test-repo-4 │
│────────────────│───────────────────────────────────────────────│


 REPOS THAT WERE SUCCESSFULLY CLONED TO THE LOCAL FILESYSTEM
│─────────────│───────────────────────────────────────────────│
│ REPO NAME   │ REPO URL                                      │
│─────────────│───────────────────────────────────────────────│
│ test-repo-3 │ https://github.com/git-xargs-test/test-repo-3 │
│ test-repo-2 │ https://github.com/git-xargs-test/test-repo-2 │
│ test-repo-1 │ https://github.com/git-xargs-test/test-repo-1 │
│─────────────│───────────────────────────────────────────────│


 REPOS THAT WERE UNABLE TO BE CLONED TO THE LOCAL FILESYSTEM
│─────────────│───────────────────────────────────────────────│
│ REPO NAME   │ REPO URL                                      │
│─────────────│───────────────────────────────────────────────│
│ test-repo-4 │ https://github.com/git-xargs-test/test-repo-4 │
│─────────────│───────────────────────────────────────────────│


 REPOS THAT SHOWED NO FILE CHANGES TO THEIR WORKING DIRECTORY FOLLOWING COMMAND EXECUTION
│─────────────│───────────────────────────────────────────────│
│ REPO NAME   │ REPO URL                                      │
│─────────────│───────────────────────────────────────────────│
│ test-repo-3 │ https://github.com/git-xargs-test/test-repo-3 │
│ test-repo-2 │ https://github.com/git-xargs-test/test-repo-2 │
│ test-repo-1 │ https://github.com/git-xargs-test/test-repo-1 │
│─────────────│───────────────────────────────────────────────│

Expected behavior
Only repos supplied with the --repo flag get cloned/processed, even if the --github-org flag is used as well.

Consider adding an option which will parse a commit and compile a list changes to apply on all supplied repos.

For example, a list containing file modifications, simple word substitutions, commit message, branch name, and similar information could be inferred from the commit change(s).

Motivation: This abstraction could be applied in CI/CD pipelines to automatically propagate simple, predictable changes across many repos on a regular basis.

Describe the bug

When using the git-xargs with a very big commit message, that includes \n characters (to create the description) the name the PR seems to ignore the \n.

I think the title should be only the name of the commit, and the description should be out of the title.

To Reproduce

Run the command of git-xargs with a very big commit message, that includes \n.

git-xargs --skip-archived-repos --loglevel DEBUG --dry-run --repo=foo/bar --branch-name \"branch/change_x_y_z\" --commit-message "Add x y z\n**X:** is to allow this\n **Y:** is to allow that\n **Z:** idk\n> OK this comment shouldn't appear in the PR name" python main.py

Expected behavior

The title of the PR for the command above should be: "Add x y z"

Nice to have

Jim's thinking:

"Should this tool do more to be "unix philosophy friendly"? E.g., Normally, you pipe things into the real xargs. Does that make sense to do here?

I'm just thinking out loud if there is some pattern like:

git-xargs list-repos gruntwork-io | xargs <SOME SCRIPT TO RUN IN EACH REPO> | git-xargs open-prs

In the Unix Philosophy spirit of “do one thing and do it well” and handling output via stdin/stdout, would it make sense to have:
A CLI tool called something like multi-git which can perform Git operations on multiple repos.
You run multi-git clone-to-temp and give it a GitHub org or list of repos to clone. It clones them to temp folders and writes the folder paths to stdout.
Pipe the output from multi-git into plain old xargs to execute some script in each of those folders.
Pipe the output from that—actually, I’m not sure what xargs produces as stdout, but let’s assume it’s the same list of temp folder paths—to multi-git commit-and-pr-if-necessary.

Advantages:
clone-to-temp and commit-and-pr-if-necessary might be useful in and of themselves in a variety of contexts.
More Unix-like approach to solving this problem.
Drawbacks:
Much harder to get started with this
Prob doesn’t work on Windows
Prob harder to handle certain corner cases

Describe the solution you'd like
When specifying the --pull-request-title and --pull-request-description it isn't clear to me that you can do anything other than copy or paste in a string of your choosing. It would be useful to allow a file to be specified, and beyond that, a template that can be given with prompts and/or options to enter variables to fill in the template for it.

Describe alternatives you've considered
I have a script to generate the git-xargs command runs, but it lacks some of the features that a multi-line description would offer, as well as just making it hard to parse the command.

Additional context

Related: gruntwork-io/fetch#43

Describe the solution you'd like
I was using git-xargs for a set of small changes on a number of repositories, but discovered a few of them have different behavior than anticipated, and I wanted to re-run a modified script against only those failed runs. Is that possible?

Describe alternatives you've considered
Manually editing a repos.txt file with the repos and re-running seems to be the answer, but it's clunky (imo).

Additional context

Is your feature request related to a problem? Please describe.

Simple install experience via brew install git-xargs.

Describe the solution you'd like

Using goreleaser this should be easy to configure.

Example: https://github.com/estahn/cloudping/blob/master/.goreleaser.yml

Describe the solution you'd like
Add functionality to create a new random branch name if one was not provided via the --branch-name command. This could make it easier to use git-xargs because many new users get tripped up on needing to iterate against new branch names when testing out their jobs.

The branch name generation could use a UUID-like string algorithm to avoid collisions with existing branch names.

Is your feature request related to a problem? Please describe.
We'd like the safety and communication benefits of using Pull Requests while reducing the burden as much as possible

Describe the solution you'd like
I'd like a switch when calling git-xargs that will turn on the GitHub auto-merge PR feature on the PRs that it creates.

Describe alternatives you've considered
I guess we could use a GitHub action that does it somehow but then we'd need to roll that out everywhere too (there is obviously a nice tool that can do that :))

Additional context
Unfortunately the GitHub v3 API doesn't appear to expose the auto-merge feature, only the GraphQL v4 API does, so you'd need to use a library like https://github.com/shurcooL/githubv4 since go-github doesn't support v4 API.

git-xargs version
v0.0.12

To Reproduce
This fails:
git-xargs --loglevel=DEBUG --dry-run --branch-name test --repo gruntwork-io/git-xargs "touch a && touch b"

and so does even this:
git-xargs --loglevel=DEBUG --dry-run --branch-name test --repo gruntwork-io/git-xargs "touch a"

while this works
git-xargs --loglevel=DEBUG --dry-run --branch-name test --repo gruntwork-io/git-xargs touch a

So it seems that quotes are the problem, but they are necessary to construct complex commands.

Follow-up from #146

Per Jim's recommendation:

stdin: alternatively, git-xargs can read the list of repos, one per line, from stdin. This is similar to xargs itself.

I was thinking more along the lines of git-xargs getting the list of repos (but not the command to run) via either --repo / --repos flags:

git-xargs --repo gruntwork-io/foo --repo gruntwork-io/bar touch license.txt

Or via stdin:

echo -e "gruntwork-io/foo\ngruntwork-io/bar" | git-xargs touch license.txt

Is your feature request related to a problem? Please describe.
GitHub's secondary rate limits prevent the creation of many resources in a short time frame. For example, creating PRs is capped in a period and GitHub's limits kick in. git-xargs will report an error and failure to create a PR when this happens with a HTTP 503 error.

git-xargs should respect the rate-limiting headers that GitHub returns and pause/sleep for that amount of time before continuing. The secondary rate limiting prevents the use of git-xargs against large amounts of repos (dozens to hundreds) because of these limits. The update scripts need to add a long pause that kills any performance (i.e. 30s) and concurrency when dealing with many repos.

https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits

Describe the solution you'd like
It would be great if git-xargs respected the GitHub X-RateLimit-* headers and automatically slept for that duration before continuing with the API requests.

https://docs.github.com/en/rest/overview/resources-in-the-rest-api#secondary-rate-limits

Describe alternatives you've considered
The workaround is to put a fixed pause in the update script, up to 30s to avoid being throttled and having the batch job fail.

Additional context

To assist with debugging why new or complex scripts aren't working as expected, we need better visibility into STDOUT and STDERR for all commands run.