Handle this output:

bash-3.2$ bash <(curl -sL get.comby.dev)
comby-0.9.1-x86_64-macos: truncated gzip input
tar: Error exit delayed from previous errors.

I was trying out comby and wanted to do the following refactoring:

dict(foo=bar,baz=qux)

=>

{"foo": bar, "baz": qux}

but I can't find a way to repeatedly apply a pattern within a particular scope. Am I missing how to do this, or is this not currently possible?

This issue tracks the build instructions or changes for packaging comby in nixpkgs. This would be nice to have--related PRs and work in progress are in #136, #137, #138. This issue tracks the discussion to for getting things to work with NixOS.

cc thanks @strager, who prompted this.

It would be great to have consistent, automatic formatting.

I've found ocamlformat to work well for me and it's integrated with dune, all you need is a .ocamlformat in the root of the project.

I'm using this config for my projects but really don't care what is used here as long as it is consistent:

profile=janestreet
let-binding-spacing=compact

Is your feature request related to a problem? Please describe.
After seeing the presentation at Strange Loop, I was going to demonstrate comby to my colleagues. I sold it as "more expressive than search and replace with regular expressions". The examples I can find would all be easily implemented as regular expressions, e.g. with sed.

Describe the solution you'd like
The README should list a simple example that is not expressible as a regular expression, or would result in a very complex regular expression.

Thank you!

Describe the bug
Comby reports wrong range is the line ending is \r\n (CRLF) instead of \n (LF)

Reproducing

• Create a file with \r\n as line ending
• Use comby -json-lines to get output in json
• See that the range reported is wrong

Expected behavior
Correct range is reported

/usr/local/bin/comby: error while loading shared libraries: libpcre.so.3: cannot open shared object file: No such file or directory

archlinux:

1 core/pcre 8.43-1 [installed]
    A library that implements Perl 5-style regular expressions
2 core/pcre2 10.33-1 [installed]
    A library that implements Perl 5-style regular expressions.
    2nd version

https://github.com/comby-tools/comby/blob/master/src/server.ml#L194

Otherwise, we might confuse comment parens as real ones (see OCaml): https://bit.ly/2Xo0Dch

comby -templates ~/tmp/combytmpl .go

will report the following, trying to parse the arguments as anonymous ones:

Error parsing command line.  Run with -help for usage information.
missing anonymous argument: REWRITE_TEMPLATE

Anonymous argument parsing should be overridden by -templates.

Describe the bug
Possibly pilot error, but failing to complete "Build from source" instructions.

Reproducing
Assuming an appropriate Ubuntu 19.10 installation:

1. Install opam (either via apt or from opam's repository using the make cold-install approach)
2. opam init
3. eval $(opam env)
4. opam switch create 4.09.0 4.09.0
5. eval $(opam env)
6. sudo apt-get install pkg-config libpcre3-dev
7. git clone https://github.com/comby-tools/comby
8. cd comby && opam install . --deps-only -y
9. eval $(opam env)
10. make

Observe error output ending in:

/usr/bin/ld: cannot find -lhp
collect2: error: ld returned 1 exit status
File "caml_startup", line 1:
Error: Error during linking
make: *** [Makefile:4: build] Error 1

See attached output below for more details.

Expected behavior
The make step completing without error.

Additional context
Output from make step should be attached.

make-output.txt

For the version of opam installed via apt:

$ opam --version
2.0.5

After some searching I'm guessing that -lhp refers to hack_parallel. Does that seem right?

I also tried installation on an Arch-derivative distribution with similar results.

Your software expects the use of rewrite templates for source code transformations so far.
Would you like to support the expression of possible changes by the means of unified diff files where plus and minus characters can be used at the beginning of lines for marking desired modifications?

For layout or indentation-sensitive languages like Python, enforce the constraint that matching bodies should be (relatively) indented for, e.g., def, if, class, etc.

Describe the bug
I do not manage to match a method block in ruby

Reproducing
Code :

def foo
  bar.each do end
end

and my matching code is :

def :[m]
end

Expected behavior
The full method body to be matched but here it matches only until the end that closes the block.

Maybe we can have an extension for VSCode to provide a useful text replace tool.

Describe the bug
Dependencies missing in the readme/mac part, namely bisect_ppx & patdiff

Reproducing
Follow the readme on a machine with a fresh install of opam

Expected behavior
Exepected to build comby when following the readme ... and it doesn't

Additional context
I don't use ocaml in my everyday life, so I've just blindly followed the readme. I may have missed some points.

Fix
Used brew install opam && opam init ... followed the readme ... added opam install bisect_ppx patdiff ... Now make does the job.

(Thanks for the tool anyway :) )

I see a use for this tool and was going to give it a try, to only discover there is currently no release for the OS I use.

Any plans to provide Windows releases? Any roadblocks?

Describe the bug
The reason langage is indicated in the documentation.
.

But it is not not present on the live.

Reproducing
Go to https://comby.live and note

Expected behavior
Reason langage should be present on the combi.live.

Describe the bug

The v0.8.0 binary does not work under macOS, because it is linked against a non-existing /usr/local/opt/pcre/lib/libpcre.1.dylib.

The suggested brew install pcre does not resolve this, because brew installs it as /usr/local/brew/lib/libpcre.1.dylib.

Reproducing

$ comby
dyld: Library not loaded: /usr/local/opt/pcre/lib/libpcre.1.dylib
  Referenced from: /usr/local/bin/comby
  Reason: image not found
$ brew install pcre
...
Warning: pcre 8.43 is already installed and up-to-date
To reinstall 8.43, run `brew reinstall pcre`
$ brew reinstall pcre
...
$ comby
dyld: Library not loaded: /usr/local/opt/pcre/lib/libpcre.1.dylib
  Referenced from: /usr/local/bin/comby
  Reason: image not found
$ find /usr/local/brew -name libpcre.1.dylib
/usr/local/brew/lib/libpcre.1.dylib
/usr/local/brew/Cellar/pcre/8.43/lib/libpcre.1.dylib

Describe the bug

% comby -match-only 'if :[hole] = :[hole] { [:whatever] }' .go

is trying to look at non-go files also.

Reproducing

% cd /Users/foo/bar
% comby -match-only 'if :[hole] = :[hole] { [:whatever] }' .go
Timeout for input: Path: /Users/foo/bar/baz/worker.py!
...

and so on. Have to resort to manually specifying the files like so

% comby  -match-only 'if :[hole] = :[hole] { [:whatever] }' **/*.go

to not waste time and get the right behavior.

Expected behavior
It should only be looking at *.go files

Additional context
Add any other context about the problem here.

A few restrictions are supported for “holes” already.
A search for alphanumeric characters and punctuation can be specified by a syntax which is slightly different from other filter possibilities.

I imagine that advanced filters could be specified directly after the match identifier.

• Character class
• Regular expression
• Predefined constraint

Hey 👋

first of all comby is a really amazing project and the concept behind blow my mind away. My use case is quite advanced and goes beyond substitution so i'm curious what you think about it. Let's imagine the following JavaScript code.

function foo() {}

and let's say I want to be able to rewrite this to

function oof() {}

or even more advanced to

function oOf() {}

I was thinking, wouldn't it be neat if comby would be able to call out another sub program for each match? In the example above the match foo gets passed to another script e.g. bash script which does something with the string and returns the final value.

Concret example: I want to use comby to add i18n to an existing code base. Therefore I need to first find all the texts (which is quite easy with comby) and replace the text with a i18n-key and also write this key/value pair to a dictionary.

How does this sound and does this match with the project vision?

Describe the bug
This is not a bug with Comby per se, but I am having trouble building it on OpenBSD 6.5 because it cannot find <pcre.h> which lives here: /usr/local/include/pcre.h.

The build fails on the opam install -y pcre step.

I know nothing about OCaml, so I'm at a loss as to how to add include paths, opam doesn't seem to be looking in /usr/local/include?

Reproducing
Follow README build instructions on OpenBSD 6.5.

Expected behavior
With the correct dependency installed, I would expect the build to successfully complete.

Additional context
Here is the full error.

Any help would be much appreciated.

I am seeing Comby omit matches/replacements on some files in a directory specified with -directory. It finds them if the file is specified explicitly. See the repro script below.

$ comby -version
0.7.0
$ uname -a
Linux xq 4.18.0-25-generic #26-Ubuntu SMP Mon Jun 24 09:32:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Repro script

#!/bin/bash

set -eu

tmpdir=$(mktemp -d)
cd $tmpdir
curl -SL https://github.com/sourcegraph/sourcegraph/archive/ff38872f93cc0b5b9a21f56b919621e02e7dc800.tar.gz | tar xzf -
cd sourcegraph-ff38872f93cc0b5b9a21f56b919621e02e7dc800

echo
echo '### Running against the whole directory produces some matches:'
echo

comby -extensions go -directory cmd/frontend/graphqlbackend/ ':[[name]]At() string { return :[v].:[[name]]At.Format(time.RFC3339) }' ':[name]At() DateTime { return DateTime{:[v].:[name]At} }'

echo
echo '### But running it against a specific file in the directory produces matches in that file that were not produced (as expected) by the command above'
echo

comby ':[[name]]At() string { return :[v].:[[name]]At.Format(time.RFC3339) }' ':[name]At() DateTime { return DateTime{:[v].:[name]At} }' cmd/frontend/graphqlbackend/user.go

Output of repro script

xq:~ $ bash ~/tmp/comby-repro.bash 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   165    0   165    0     0    435      0 --:--:-- --:--:-- --:--:--   435
100 10.4M  100 10.4M    0     0  8958k      0  0:00:01  0:00:01 --:--:-- 21.2M
/tmp/tmp.vCLqQUXHYa/sourcegraph-ff38872f93cc0b5b9a21f56b919621e02e7dc800

### Running against the whole directory produces some matches:

------ cmd/frontend/graphqlbackend/access_token.go
++++++ cmd/frontend/graphqlbackend/access_token.go
@|-60,9 +60,7 ============================================================
 |	return UserByIDInt32(ctx, r.accessToken.CreatorUserID)
 |}
 |
-|func (r *accessTokenResolver) CreatedAt() string {
-|	return r.accessToken.CreatedAt.Format(time.RFC3339)
-|}
+|func (r *accessTokenResolver) CreatedAt() DateTime { return DateTime{r.accessToken.CreatedAt} }
 |
 |func (r *accessTokenResolver) LastUsedAt() *string {
 |	if r.accessToken.LastUsedAt == nil {
------ cmd/frontend/graphqlbackend/org_members.go
++++++ cmd/frontend/graphqlbackend/org_members.go
@|-45,10 +45,6 ============================================================
 |	return UserByIDInt32(ctx, r.membership.UserID)
 |}
 |
-|func (r *organizationMembershipResolver) CreatedAt() string {
-|	return r.membership.CreatedAt.Format(time.RFC3339)
-|}
+|func (r *organizationMembershipResolver) CreatedAt() DateTime { return DateTime{r.membership.CreatedAt} }
-|
-|func (r *organizationMembershipResolver) UpdatedAt() string {
-|	return r.membership.UpdatedAt.Format(time.RFC3339)
-|}
+|
+|func (r *organizationMembershipResolver) UpdatedAt() DateTime { return DateTime{r.membership.UpdatedAt} }
------ cmd/frontend/graphqlbackend/org.go
++++++ cmd/frontend/graphqlbackend/org.go
@|-80,7 +80,7 ============================================================
 |
 |func (o *OrgResolver) SettingsURL() *string { return strptr(o.URL() + "/settings") }
 |
-|func (o *OrgResolver) CreatedAt() string { return o.org.CreatedAt.Format(time.RFC3339) }
+|func (o *OrgResolver) CreatedAt() DateTime { return DateTime{o.org.CreatedAt} }
 |
 |func (o *OrgResolver) Members(ctx context.Context) (*staticUserConnectionResolver, error) {
 |	// 🚨 SECURITY: Only org members can list the org members.
------ cmd/frontend/graphqlbackend/external_service.go
++++++ cmd/frontend/graphqlbackend/external_service.go
@|-68,13 +68,9 ============================================================
 |	return r.externalService.Config
 |}
 |
-|func (r *externalServiceResolver) CreatedAt() string {
-|	return r.externalService.CreatedAt.Format(time.RFC3339)
-|}
+|func (r *externalServiceResolver) CreatedAt() DateTime { return DateTime{r.externalService.CreatedAt} }
-|
-|func (r *externalServiceResolver) UpdatedAt() string {
-|	return r.externalService.UpdatedAt.Format(time.RFC3339)
-|}
+|
+|func (r *externalServiceResolver) UpdatedAt() DateTime { return DateTime{r.externalService.UpdatedAt} }
 |
 |func (r *externalServiceResolver) Warning() *string {
 |	if r.warning == "" {
------ cmd/frontend/graphqlbackend/survey_response.go
++++++ cmd/frontend/graphqlbackend/survey_response.go
@|-46,9 +46,7 ============================================================
 |	return s.surveyResponse.Better
 |}
 |
-|func (s *surveyResponseResolver) CreatedAt() string {
-|	return s.surveyResponse.CreatedAt.Format(time.RFC3339)
-|}
+|func (s *surveyResponseResolver) CreatedAt() DateTime { return DateTime{s.surveyResponse.CreatedAt} }
 |
 |// SurveySubmissionInput contains a satisfaction (NPS) survey response.
 |type SurveySubmissionInput struct {
------ cmd/frontend/graphqlbackend/org_invitation.go
++++++ cmd/frontend/graphqlbackend/org_invitation.go
@|-52,7 +52,7 ============================================================
 |func (r *organizationInvitationResolver) Recipient(ctx context.Context) (*UserResolver, error) {
 |	return UserByIDInt32(ctx, r.v.RecipientUserID)
 |}
-|func (r *organizationInvitationResolver) CreatedAt() string { return r.v.CreatedAt.Format(time.RFC3339) }
+|func (r *organizationInvitationResolver) CreatedAt() DateTime { return DateTime{r.v.CreatedAt} }
 |func (r *organizationInvitationResolver) NotifiedAt() *string {
 |	if r.v.NotifiedAt == nil {
 |		return nil

### But running it against a specific file in the directory produces matches in that file that were not produced (as expected) by the command above

------ /tmp/tmp.vCLqQUXHYa/sourcegraph-ff38872f93cc0b5b9a21f56b919621e02e7dc800/cmd/frontend/graphqlbackend/user.go
++++++ /tmp/tmp.vCLqQUXHYa/sourcegraph-ff38872f93cc0b5b9a21f56b919621e02e7dc800/cmd/frontend/graphqlbackend/user.go
@|-124,9 +124,7 ============================================================
 |
 |func (r *UserResolver) SettingsURL() *string { return strptr(r.URL() + "/settings") }
 |
-|func (r *UserResolver) CreatedAt() string {
-|	return r.user.CreatedAt.Format(time.RFC3339)
-|}
+|func (r *UserResolver) CreatedAt() DateTime { return DateTime{r.user.CreatedAt} }
 |
 |func (r *UserResolver) UpdatedAt() *string {
 |	t := r.user.UpdatedAt.Format(time.RFC3339) // ISO

Describe the bug
Comby will time out with most input files, even small ones, with a simple pattern :

:[x] = __builtin_annot_intval("symb_int", :[x]);

I've been using comby a lot recently and it works wonderfully. In particular the following pattern works without any issue :

int :[x] = __builtin_annot_intval("symb_int", :[x]);

Reproducing
Here's a link to the reproduced bug.

If I'm correct, matching should happen.
In particular, if you replace the wholes with va, it matches one of the elements.

Is your feature request related to a problem? Please describe.

I'm always frustrated when I type in a query and it takes a while to return results.

Describe the solution you'd like

It'd be nice if performance was optimized. For example, the following command:

comby -o -matcher .c -jobs 8 ':[_] << :[_] << endl' . 2>/dev/null $(rg -H '<<|endl' | cut -d: -f 1 | uniq)

is way faster (30s) than this command:

comby -o -matcher .c -jobs 8 ':[_] << :[_] << endl' . 2>/dev/null

(3min).

I'm sure there's other clever ways to improve performance as well.

Additional context
Add any other context or screenshots about the feature request here.

Match rules currently fire after matches are found (after shifting completes). Ideally, match rules should fire as soon as a match is found, and then decide how much to shift.

Describe the bug
Attempting to build from source fails during the make step, specifically when processing lib/matchers/omega.ml, lib/language/omega_rule.ml, and lib/rewriter/rewrite_template.ml.

Reproducing
The following is an invocation of make and its output:

$ make
dune build --profile dev
File "lib/matchers/omega.ml", line 333, characters 12-19:
333 |           | Error _ ->
                  ^^^^^^^
Error: This pattern matches values of type ('b, 'c) result
       but a pattern was expected which matches values of type
         consume:Consume.t -> (Omega.production * 'a, string) result
File "lib/rewriter/rewrite_template.ml", line 22, characters 4-12:
22 |   | Ok label -> List.find_map label ~f:ident
         ^^^^^^^^
Error: This pattern matches values of type ('a, 'b) result
       but a pattern was expected which matches values of type
         consume:Consume.t -> (string option list, string) result
File "lib/language/omega_rule.ml", line 242, characters 4-11:
242 |   | Ok rule -> Or_error.return rule
          ^^^^^^^
Error: This pattern matches values of type ('a, 'b) Stdlib.result
       but a pattern was expected which matches values of type
         consume:Consume.t -> (expression list, string) Stdlib.result
make: *** [Makefile:4: build] Error 1

Expected behavior
I was expected the building from source to succeed.

Additional context
I'm using Ubuntu 19.10.

Tried a second time, but first removing ~/.opam and beginning at the first step of https://github.com/comby-tools/comby#build-from-source got similar results.

The steps I followed were:

• rm -rf ~/.opam
• log out
• log in
• opam init
• edit ~/.profile and ~/.bashrc for opam settings (I have no ~/.bash_profile)
• eval $(opam env)
• opam switch create 4.09.0 4.09.0
• eval $(opam env)
• git clone https://github.com/comby-tools/comby
• cd comby
• opam install . --deps-only -y
• eval $(opam env)
• make

I'm always frustrated when I need to supply several different file paths as input.

Describe the solution you'd like

Self-containing .comby file format that contains match template, rewrite template and example in one file.

if (:[cond])

---[rewrite]-----
if (1)

---[example]-----
if (fgets(line, 128, file_pointer) == Null) // 1) if (...) returns 0
      return 0;
...
if (scanf("%d) %d", &x, &y) == 2) // 2) if (scanf("%d) %d", &x, &y) == 2) returns 0
      return 0; 

Additional context
Or it could be a simplate - https://simplates.org/

Recently I've started using https://github.com/entangled/entangled to write literate style documentation with actual implementation embedded in "```" markdown code blocks.

This is pretty neat, but some filetypes (like JSON) do not have comments and can not be supported this way. It would also sometimes be interesting to represent code or data, in actual markdown format, for example tables.

I see you're already using Comby to match Markdown and output HTML

Could I use Comby today to parse Markdown tables, output package.json dependencies, and the other way around: generate a table of project dependencies from package.json? Is this something you have considered?

Describe the bug
Followed the instructions, but build failed on Mac.

$ make
dune build --profile dev
Info: Creating file dune-project with this contents:
| (lang dune 1.11)
| (name comby)
File "lib/dune", line 8, characters 3-10:
8 |    mparser
       ^^^^^^^
Error: Library "mparser" not found.
Hint: try: dune external-lib-deps --missing --profile dev @@default
File "lib/configuration/dune", line 5, characters 101-108:
5 |   (libraries camlzip comby.statistics comby.parsers comby.match comby.language ppxlib core core.uuid mparser mparser.pcre yojson ppx_deriving_yojson hack_parallel))
                                                                                                         ^^^^^^^
Error: Library "mparser" not found.
Hint: try: dune external-lib-deps --missing --profile dev @@default
File "lib/parsers/dune", line 5, characters 34-41:
5 |   (libraries angstrom ppxlib core mparser mparser.pcre))
                                      ^^^^^^^
Error: Library "mparser" not found.
Hint: try: dune external-lib-deps --missing --profile dev @@default
File "lib/matchers/dune", line 8, characters 70-77:
8 |   (libraries comby.parsers comby.match angstrom ppxlib core core.uuid mparser mparser.pcre yojson ppx_deriving_yojson))
                                                                          ^^^^^^^
Error: Library "mparser" not found.
Hint: try: dune external-lib-deps --missing --profile dev @@default
make: *** [build] Error 1

After I did opam install mparser (mparser.1.2.3), several errors went away, but still get one about Unbound value Lwt_unix.i_am_the_captain_now:

$ make
dune build --profile dev
File "lib/matchers/alpha/matcher.ml", line 751, characters 10-23:
Error: Unbound value parse_string'
Hint: Did you mean parse_string?
File "lib/pipeline/pipeline.ml", line 384, characters 6-35:
384 |       Lwt_unix.i_am_the_captain_now ();
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Unbound value Lwt_unix.i_am_the_captain_now
make: *** [build] Error 1

Reproducing
Follow build from source instructions on a MacOS

• If this is not about a matcher, please describe the bug:

Expected behavior
make and make test with no errors.

As the lib supports OCaml I think would make sense to support ReasonML as well.
I can try to help

Describe the bug

Not 100% sure if this is really a bug, but the output is unexpected.
It looks like the fist child node in HTML does not match.

Reproducing

• If this looks like a bug in the matcher, go to https://comby.live and paste a shared link below:
  Link

Expected behavior

<div>__
  <span>__Foo__</span>
  <span>__
    <span>__Bar Baz__</span>
  </span>__
__</div>

Additional context

In my particular use case, a matcher like <:[opener]>:[[val]]</:[closer]> would help as a workaround, but this does not take \s into account.

Warn on templates that start with :[hole], since it matches everything including newlines up to its suffix.

Reference: #145

As found in, e.g., SGML/HTML/XML

Hi, long time codemod user here.
One thing that has always annoyed me in the workflow of hunk-by-hunk interactively accepting changes, is that afterwards, I still have to add all the changes to the index using git add -p which is also interactive. (I could do git add . but i often have changes i don't want to commit)

consider this a low-priority request, but it would be nice that whenever i interactively accept a change, it (and only that particular change i just agreed to apply) could also be added to the git index.

I can't connect to it.

The -match-only flag currently supports only JSON output--add an option to print out matches by file and line.

Dynamic linking against PCRE in the binary distributions is not great. This issue tracks alternative solutions.

I need to matching something top-level, but i need the context (right hand side) up to the next newline. Sometimes this is empty.

Examples:

 val x = Some(42)
 val Some(x) = y

I need to be able to match something like Some(:[param]):[rhs\n], because i need to examine the RHS in the where clause.

Describe the solution you'd like
Either modify https://github.com/comby-tools/comby/blob/master/lib/matchers/matcher.ml#L525 to be many instead of many1, or add a new hole syntax form (perhaps :[id?\n] to make it zero-or-more instead of one-or-more.

Contents can be found in “holes”.
Would you like to support that data which were found in source files can be exported to well-known file formats like CSV, JSON or XML?

I sketched up an example that I can run on my code: bit.ly/2n0kqP8
Then I tried to apply it to this file: https://github.com/RevolutionRobotics/RevvyFirmware/blob/master/main.c

And the resulting diff is the attached image.

Comby 0.9.1 was freshly installed on an Ubuntu that runs on top of WSL.

Hey, I'm trying to match Haskell line comments, which start with --, but I'm getting the following error:

$ comby "-- " "--- " ./file
Error parsing command line.  Run with -help for usage information.
unknown flag --

Is there a way around it?