Annoying task, but the functions should capture the R seed and pass it through to rust, so that the results are perfectly replicable.

At present, a user who wants to block on the basis of more than one column has to combine these columns themselves. Should be simple enough to convert these under-the-hood either via concatenation or hashing and pass to the rust core.

Sorry for bothering. My system is Windows 11 and R version are 4.30 and 4.23. I used the two versions to install the package but both failed. For the RUST I used stable-x86-64-windows-msvc version 1.7.00 with toolchain gnu installed. The error message writes

undefined reference to `NtCreateFile'

which could be the factor of error. Really appreciate for your help.

E:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: ./rust/target/x86_64-pc-windows-gnu/release/libzoomerjoin.a(std-ca5208825e97b4ba.std.687851ba-cgu.0.rcgu.o): in function std::sys::windows::fs::open_link_no_reparse': /rustc/90c541806f23a127002de5b4038be731ba1458ca/library\std\src\sys\windows/fs.rs:800: undefined reference to NtCreateFile'
E:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: /rustc/90c541806f23a127002de5b4038be731ba1458ca/library\std\src\sys\windows/fs.rs:829: undefined reference to RtlNtStatusToDosError' E:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: ./rust/target/x86_64-pc-windows-gnu/release/libzoomerjoin.a(std-ca5208825e97b4ba.std.687851ba-cgu.0.rcgu.o): in function std::sys::windows::handle::Handle::synchronous_read':
/rustc/90c541806f23a127002de5b4038be731ba1458ca/library\std\src\sys\windows/handle.rs:241: undefined reference to NtReadFile' E:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: /rustc/90c541806f23a127002de5b4038be731ba1458ca/library\std\src\sys\windows/handle.rs:272: undefined reference to RtlNtStatusToDosError'
E:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: ./rust/target/x86_64-pc-windows-gnu/release/libzoomerjoin.a(std-ca5208825e97b4ba.std.687851ba-cgu.0.rcgu.o): in function std::sys::windows::handle::Handle::synchronous_write': /rustc/90c541806f23a127002de5b4038be731ba1458ca/library\std\src\sys\windows/handle.rs:290: undefined reference to NtWriteFile'
E:\rtools42\x86_64-w64-mingw32.static.posix\bin/ld.exe: /rustc/90c541806f23a127002de5b4038be731ba1458ca/library\std\src\sys\windows/handle.rs:318: undefined reference to `RtlNtStatusToDosError'
collect2.exe: error: ld returned 1 exit status
無法產生 DLL 檔案 # Unable to produce DLL file
ERROR: compilation failed for package 'zoomerjoin'

• removing 'C:/Users/user/AppData/Local/Temp/RtmpW6qdAm/pkg-lib69c458dd7b3f/zoomerjoin'

Describe the bug
I am running Windows 11 with rust 1.71.1 and wsl, when I try to install the package this is my console with an error message:

> library(zoomerjoin)
Error in library(zoomerjoin) : there is no package called ‘zoomerjoin’
> ## Install with devtools
> # install.packages("devtools")
> devtools::install_github("beniaminogreen/zoomerjoin")
Downloading GitHub repo beniaminogreen/zoomerjoin@HEAD
These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All                          
2: CRAN packages only           
3: None                         
4: vctrs (0.6.2 -> 0.6.3) [CRAN]

Enter one or more numbers, or an empty line to skip updates: 3
── R CMD build ───────────────────────────────────────────────────────────────────────────────────────────────────────────
✔  checking for file 'C:\Users\winco\AppData\Local\Temp\RtmpCwBEaE\remotes522c7688750f\beniaminogreen-zoomerjoin-f359f28/DESCRIPTION' ...
─  preparing 'zoomerjoin':
✔  checking DESCRIPTION meta-information ... 
─  cleaning src
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building 'zoomerjoin_0.0.0.9000.tar.gz'
   
Installing package into ‘C:/Users/winco/AppData/Local/R/win-library/4.3’
(as ‘lib’ is unspecified)
* installing *source* package 'zoomerjoin' ...
** using staged installation
** libs
using C compiler: 'gcc.exe (GCC) 12.2.0'
rm -Rf zoomerjoin.dll ./rust/target/x86_64-pc-windows-gnu/release/libzoomerjoin.a entrypoint.o
gcc  -I"C:/PROGRA~1/R/R-43~1.0/include" -DNDEBUG     -I"C:/RBuildTools/4.3/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c entrypoint.c -o entrypoint.o
mkdir -p ./rust/target/libgcc_mock
cd ./rust/target/libgcc_mock && \
	touch gcc_mock.c && \
	gcc -c gcc_mock.c -o gcc_mock.o && \
	ar -r libgcc_eh.a gcc_mock.o && \
	cp libgcc_eh.a libgcc_s.a
C:\RBuildTools\4.3\x86_64-w64-mingw32.static.posix\bin\ar.exe: creating libgcc_eh.a
# CARGO_LINKER is provided in Makevars.ucrt for R >= 4.2
export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_LINKER="x86_64-w64-mingw32.static.posix-gcc.exe" && \
	export LIBRARY_PATH="${LIBRARY_PATH};/c/Users/winco/AppData/Local/Temp/Rtmpctc8EF/R.INSTALL1c04338dae1/zoomerjoin/src/./rust/target/libgcc_mock" && \
	cargo +stable-gnu build --target=x86_64-pc-windows-gnu --lib --release --manifest-path=./rust/Cargo.toml --target-dir ./rust/target
error: toolchain 'stable-x86_64-pc-windows-gnu' is not installed
make: *** [Makevars.win:19: rust/target/x86_64-pc-windows-gnu/release/libzoomerjoin.a] Error 1
ERROR: compilation failed for package 'zoomerjoin'
* removing 'C:/Users/winco/AppData/Local/R/win-library/4.3/zoomerjoin'
Warning message:
In i.p(...) :
  installation of package ‘C:/Users/winco/AppData/Local/Temp/RtmpCwBEaE/file522c5d5356f/zoomerjoin_0.0.0.9000.tar.gz’ had non-zero exit status

I found this guidance rust-lang/rustup#1793 and tried to follow along but the package is still not installing.

This is the result of rustup show for me:

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/wincowger/.rustup

stable-x86_64-unknown-linux-gnu (default)
rustc 1.71.1 (eb26296b5 2023-08-03)

To Reproduce
Steps to reproduce the behavior:
Follow user guidelines on Windows 11 machine with WSL.

Expected behavior
A clear and concise description of what you expected to happen.
Installation to proceed.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 11

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.

This package is a great example of using R and Rust together and serves to fill a performance gap in the R ecosystem. The R community would benefit greatly from the publication of the library on CRAN.

Have you considered publishing to CRAN?

Is your feature request related to a problem? Please describe.
It is nice to be able to use block_by to filter out some comparisons before computing the string similarity. Currently, it is limited to equality conditions (e.g rows must have the same year to be considered for matching). I have a setting in which I don't want to compare rows if year.y is larger than year.x, i.e I'd like to block matching by the condition year.x > year.y.

I don't know how hard that would be to implement. I could track the usage of block_by (renamed salt in the internals) until the method new() for the Shingleset struct in Rust, but I don't know how that works exactly.

Describe the solution you'd like
Not sure of the syntax this should take. It could be something similar to dplyr::join_by(), so that one could do block_by = block_by(year.x > year.y). Also, that only works if the variable has a different name in each dataset.

Describe alternatives you've considered
Currently I don't use block_by. Instead I match on the full data and filter out by my condition afterwards.

Additional context
/

Now that the package is getting more mature, it would be nice to add support for other distance metrics (specifically, hamming and cosine distances). These should be relatively easy to implement following these notes, and will also provide the opportunity to refine some of the code. In the future, I would like the hash families to implement an lsh trait, which would allow us to reuse some of the amplifying code.

As per JOSS requirements, there should be community guidelines "Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support". I looked in the readme and the manuscript but couldn't find these points.

Look into adding the option to combine LSH joining on a character field with widowed joining on a continuous variable. The fastlink package has a great implementation I can look at.

Prepare for release:

• git pull
• Check current CRAN check results
• Polish NEWS
• usethis::use_github_links()
• urlchecker::url_check()
• devtools::build_readme()
• devtools::check(remote = TRUE, manual = TRUE)
• devtools::check_win_devel()
• revdepcheck::revdep_check(num_workers = 4)
• Update cran-comments.md
• git push

Submit to CRAN:

• usethis::use_version('patch')
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• usethis::use_github_release()
• usethis::use_dev_version(push = TRUE)

Write vignette for record joining + benchmark vignette.

The CRAN installation line in the README (install.packages(zoomerjoin)) should probably be have quotes around zoomerjoin, i.e. install.packages('zoomerjoin').

The CRAN builder is currently showing the following issues when building zoomerjoin:

Version: 0.1.4
Check: compiled code
Result: NOTE
File ‘zoomerjoin/libs/zoomerjoin.so’:
Found non-API calls to R: ‘ENVFLAGS’, ‘FRAME’, ‘HASHTAB’, ‘PRCODE’,
‘PRENV’, ‘PRSEEN’, ‘PRVALUE’, ‘Rf_findVarInFrame3’, ‘SET_BODY’,
‘SET_CLOENV’, ‘SET_ENCLOS’, ‘SET_ENVFLAGS’, ‘SET_FORMALS’,
‘SET_PRCODE’, ‘SET_PRENV’, ‘SET_PRVALUE’, ‘STRING_PTR’, ‘SYMVALUE’,
‘XLENGTH_EX’
Compiled code should not call non-API entry points in R.
See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual.

Unsure why this is the case + why CRAN is suddenly giving NOTEs when it was not before (could this be an rextendr issue?). Opening this issue to track my work as I investigate what's going on + to solicit help if anyone is familiar with this issue.

Is your feature request related to a problem? Please describe.
The join_by() helper function was introduced in dplyr 1.1.0. Given that functions in zoomerjoin are designed to be drop-in replacements of dplyr functions, it would be nice to support this syntax so that we don't have to manually change the syntax to a named vector.

Describe the solution you'd like
Support for join_by() syntax, so that the example below works:

library(babynames)
library(zoomerjoin)
library(dplyr, warn.conflicts = FALSE)

baby_names <- data.frame(name = tolower(unique(babynames$name)))
baby_names_sans_vowels <- data.frame(
  name_wo_vowels =gsub("[aeiouy]","", baby_names$name)
)

# dplyr
joined_names <- inner_join(
  baby_names,
  baby_names_sans_vowels,
  by = join_by(name == name_wo_vowels)
)

# zoomerjoin
joined_names <- jaccard_inner_join(
  baby_names,
  baby_names_sans_vowels,
  by = join_by(name == name_wo_vowels)
)
#> Warning in jaccard_join(a, b, mode = "inner", by = by, salt_by = block_by, : A pair of records at the threshold (0.7) have only a 93% chance of being compared.
#> Please consider changing `n_bands` and `band_width`.
#> Error in simple_by_validate(a, b, by): by_a %in% names(a) are not all TRUE

Describe alternatives you've considered
This is a minor feature request, only for convenience. Using a named vector works very well, it is simply superseded by the join_by() syntax.

Additional context
/

Thanks for this amazing package, it's so nice to match names that fast

See dgrtwo/fuzzyjoin#54 for the rationale. Should be easy enough to support this, as rust has some very developed kd-tree packages that I can just wrap in R code.

Hi, I just found this package, it looks cool and super useful!

One thing that I noted in the benchmarks is how low the memory usage is. I know that using Rust is more efficient in speed and memory usage, but I also think the numbers reported about memory might be inaccurate. From ?profmem:

[...] nearly all memory allocations done in R are logged. Neither memory deallocations nor garbage collection events are logged. Furthermore, allocations done by non-R native libraries or R packages that use native code Calloc() / Free() for internal objects are also not logged.

I suspect that a lot of memory allocations are not done in R but in Rust and that the memory usage is actually higher than reported. I run into the same thing when I benchmark polars and tidypolars, so I'm interested if you find a workaround 😉

Just to give you an example: in polars, when I take the mean of a column with 100_000, 1_000_000, 10_000_000, or 100_000_000 rows, R reports the same (tiny) memory usage but I clearly see a peak in Windows task manager:

library(polars)

bench::press(
  rows = c(1e5, 1e6, 1e7, 1e8),
  {
    dat <- pl$DataFrame(
      a = rnorm(rows),
      b = rnorm(rows),
      c = rnorm(rows)
    )
    bench::mark(
      dat$with_columns(y = pl$col("a")$mean())
    )
  }
)
#> # A tibble: 4 × 7
#>   expression                  rows     min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr>                 <dbl> <bch:t> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 "dat$with_columns(y = pl$…   1e5 557.8µs  739.5µs   1218.      3.81KB        0
#> 2 "dat$with_columns(y = pl$…   1e6   3.4ms   3.95ms    123.      3.81KB        0
#> 3 "dat$with_columns(y = pl$…   1e7    38ms  42.06ms     14.7     3.81KB        0
#> 4 "dat$with_columns(y = pl$…   1e8   526ms 526.01ms      1.90    3.81KB        0

I was told there's a linux tool to make more accurate benchmarks when calling other languages from R but I don't remember the name, I'll update this post if I find it.

Currently, the package only supports joining on two distance functions, the Euclidean and Jaccard distances. Implementing LSH schemes for new distances is tough and time consuming. A future goal for the package could be to provide an implementation of Hierarchically-Navigable Small Worlds, which would allow users to join datasets in linearithmic time. This is worse than the scaling of LSH, but has the advantage that it works for any distance functions.

Figuring out how to create a performant implementation of HNSW in Rust will likely take a lot of time, but could have huge benefits.

Hello again!

Installing zoomerjoin requires Rust and building the source locally. It would be very convenient to provide R binaries, either via R-universe of via Github releases (as polars does for example) so that one can just install the package in a couple of seconds without needing Rust.

Is your feature request related to a problem? Please describe.
If I'm not mistaken, zoomerjoin uses all threads available on the laptop, which explains in part its great performance. It would be nice to be able to configure the number of threads, so that I can use a part of the CPU for other tasks (I'm surprised that CRAN accepted the package given they don't want more than 2 threads to be used by tests and examples).

Describe the solution you'd like
Either a function or an option to specify the number of threads that can be used by zoomerjoin.

Describe alternatives you've considered
/

Additional context
/

Thanks again for this great package!

Hello @beniaminogreen, the following lines take quite some time to run as the output becomes larger.

I made a quick test with collapse::%!iin% instead of !%in% and found ~10% decrease in time and 15% decrease in memory used when both inputs have 300k rows:

library(bench) 
library(dplyr)

out <- cross::run(
  pkgs = c("beniaminogreen/zoomerjoin", "etiennebacher/zoomerjoin@more-speedup"), 
  ~{
    library(zoomerjoin)
    
    ### Setup -----
    
    corpus_1 <- data.table::rbindlist(rep(list(dime_data), 300))
    names(corpus_1) <- c("a", "field")
    
    corpus_2 <- data.table::rbindlist(rep(list(dime_data), 300))
    names(corpus_2) <- c("b", "field")
    
    ### Benchmark -----
    
    bench::mark(
      jaccard_left_join(corpus_1, corpus_2, 
                        by = "field", n_gram_width=6,
                        n_bands=20, band_width=6, threshold = .8)
    )
  }
)

tidyr::unnest(out, result) |> 
  select(pkg, median, mem_alloc)
#> # A tibble: 2 × 3
#>   pkg                                     median mem_alloc
#>   <chr>                                 <bch:tm> <bch:byt>
#> 1 beniaminogreen/zoomerjoin                3.73m    18.5GB
#> 2 etiennebacher/zoomerjoin@more-speedup    3.28m    15.1GB

I don't know whether you'd be open to using collapse as a dependency (it only has Rcpp as dependency), but I think it'd be worth it as the performance gain should increase with the size of the data. Happy to make a PR if you're fine with that.

When you use zoomer join to join grouped tables, the groups are lost. I know this is not how the dpylr joins handle it

• look into how dplyr preserves groups when joining

Prepare for release:

• git pull
• Check current CRAN check results
• Polish NEWS
• usethis::use_github_links()
• urlchecker::url_check()
• devtools::build_readme()
• devtools::check(remote = TRUE, manual = TRUE)
• devtools::check_win_devel()
• revdepcheck::revdep_check(num_workers = 4)
• Update cran-comments.md
• git push

Submit to CRAN:

• usethis::use_version('patch')
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• usethis::use_github_release()
• usethis::use_dev_version(push = TRUE)

Additional review comments for openjournals/joss-reviews#5693

Usages in the readme should have consistent a naming convention for the functions: https://github.com/beniaminogreen/zoomerjoin#usage

The jaccard_full_join() has parentheses but euclidean_full_join doesn't which makes it appear like the euclidean functions are actually objects.

Title says it all. Add blocking / salting (which is supported in the backend) to the user-facing API.

Title says it all. The iteration counter can be annoying when you are working with smaller datasets

Hi Beniamino,

Thanks for the zoomerjoin PKG.

Tried your very clear
" Political Donors " data EXAMPLE,
using the
jaccard_inner_join( ) Fx.

Really, really fast!. :-)

Could you please show
an R code Example
with the =same= " Political Donors " data,
but using the
euclidean_inner_join( ) Fx ?.

Thanks - very highly appreciated...

SFd99
San Francisco
latest zoomerjoin PKG v0.1.0
latest R, Rstudio, Ubuntu Linux 20.04

Hello,

first congrats to this amazing package. I am simply stunned by its performance.

I was wondering whether there is any way to expand the set of functions to include regex matches.
I am currently using fuzzyjoin::regex_left_join, but unfortunately, in my use case it's simply too slow.

Unfortunately, I don't know Rust, so I can't contribute anything in this regard. In any case, thanks
again for this powerful package which should be known much more widely.

Roland