kuwisdelu / matter Goto Github PK

View Code? Open in Web Editor NEW

53.0 53.0 4.0 3.11 MB

Out-of-core statistical computing and signal processing

Home Page: http://www.cardinalmsi.org

R 71.50% C++ 27.86% C 0.64%

matter's People

Contributors

Stargazers

Watchers

Forkers

michbur hoxo-m woosah stevenmayher

matter's Issues

Installation fails with version 2.2.0, from Bioconductor and from github

Hello!

I struggle to install matter on my arch-linux machine. I tried with BiocManager::install("matter") (Version 3.17 as I am running R 4.3), and I tried with remotes::install_github("https://github.com/kuwisdelu/matter/"), but both gave me the same error message:

Bioconductor version 3.17 (BiocManager 1.30.21), R 4.3.0 (2023-04-21)
Installing package(s) 'matter'
trying URL 'https://bioconductor.org/packages/3.17/bioc/src/contrib/matter_2.2.0.tar.gz'
Content type 'application/x-gzip' length 360010 bytes (351 KB)
==================================================
downloaded 351 KB

* installing *source* package ‘matter’ ...
** using staged installation
** libs
using C++ compiler: ‘g++ (GCC) 13.1.1 20230429’
g++ -std=gnu++17 -I"/usr/include/R/" -DNDEBUG   -I/usr/local/include    -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/r/src=/usr/src/debug/r -flto=auto -ffat-lto-objects  -c altrep.cpp -o altrep.o
In file included from atoms.h:8,
                 from matter.h:5,
                 from altrep.cpp:3:
coerce.h:71:19: error: ‘uint16_t’ was not declared in this scope
   71 | Rbyte coerce_cast(uint16_t x) {
      |                   ^~~~~~~~
coerce.h:1:1: note: ‘uint16_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
  +++ |+#include <cstdint>
    1 | #ifndef COERCE
coerce.h:71:7: warning: too many template headers for ‘coerce_cast’ (should be 0)
   71 | Rbyte coerce_cast(uint16_t x) {
      |       ^~~~~~~~~~~
coerce.h:71:30: error: expected ‘;’ before ‘{’ token
   71 | Rbyte coerce_cast(uint16_t x) {
      |                              ^~
      |                              ;
coerce.h:94:19: error: ‘uint32_t’ was not declared in this scope
   94 | Rbyte coerce_cast(uint32_t x) {
      |                   ^~~~~~~~
coerce.h:94:19: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:94:7: warning: too many template headers for ‘coerce_cast’ (should be 0)
   94 | Rbyte coerce_cast(uint32_t x) {
      |       ^~~~~~~~~~~
coerce.h:94:30: error: expected ‘;’ before ‘{’ token
   94 | Rbyte coerce_cast(uint32_t x) {
      |                              ^~
      |                              ;
In file included from altrep.cpp:2:
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = unsigned char; Tin = long int]’:
matterDefines.h:115:17: error: ‘INT64_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  115 | #define NA_LONG INT64_MIN
      |                 ^~~~~~~~~
coerce.h:107:27: note: in expansion of macro ‘NA_LONG’
  107 |                 if ( x == NA_LONG )
      |                           ^~~~~~~
coerce.h: At global scope:
coerce.h:118:19: error: ‘uint64_t’ was not declared in this scope
  118 | Rbyte coerce_cast(uint64_t x) {
      |                   ^~~~~~~~
coerce.h:118:19: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:118:7: warning: too many template headers for ‘coerce_cast’ (should be 0)
  118 | Rbyte coerce_cast(uint64_t x) {
      |       ^~~~~~~~~~~
coerce.h:118:30: error: expected ‘;’ before ‘{’ token
  118 | Rbyte coerce_cast(uint64_t x) {
      |                              ^~
      |                              ;
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = short int; Tin = int]’:
matterDefines.h:101:24: error: ‘INT16_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  101 | #define R_SHORT_MIN (1+INT16_MIN)
      |                        ^~~~~~~~~
coerce.h:155:18: note: in expansion of macro ‘R_SHORT_MIN’
  155 |         if ( x < R_SHORT_MIN || x > R_SHORT_MAX )
      |                  ^~~~~~~~~~~
matterDefines.h:102:21: error: ‘INT16_MAX’ was not declared in this scope
  102 | #define R_SHORT_MAX INT16_MAX
      |                     ^~~~~~~~~
coerce.h:155:37: note: in expansion of macro ‘R_SHORT_MAX’
  155 |         if ( x < R_SHORT_MIN || x > R_SHORT_MAX )
      |                                     ^~~~~~~~~~~
matterDefines.h:102:21: note: ‘INT16_MAX’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
  102 | #define R_SHORT_MAX INT16_MAX
      |                     ^~~~~~~~~
coerce.h:155:37: note: in expansion of macro ‘R_SHORT_MAX’
  155 |         if ( x < R_SHORT_MIN || x > R_SHORT_MAX )
      |                                     ^~~~~~~~~~~
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = short int; Tin = double]’:
matterDefines.h:101:24: error: ‘INT16_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  101 | #define R_SHORT_MIN (1+INT16_MIN)
      |                        ^~~~~~~~~
coerce.h:166:18: note: in expansion of macro ‘R_SHORT_MIN’
  166 |         if ( x < R_SHORT_MIN || x > R_SHORT_MAX || !R_FINITE(x) )
      |                  ^~~~~~~~~~~
matterDefines.h:102:21: error: ‘INT16_MAX’ was not declared in this scope
  102 | #define R_SHORT_MAX INT16_MAX
      |                     ^~~~~~~~~
coerce.h:166:37: note: in expansion of macro ‘R_SHORT_MAX’
  166 |         if ( x < R_SHORT_MIN || x > R_SHORT_MAX || !R_FINITE(x) )
      |                                     ^~~~~~~~~~~
matterDefines.h:102:21: note: ‘INT16_MAX’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
  102 | #define R_SHORT_MAX INT16_MAX
      |                     ^~~~~~~~~
coerce.h:166:37: note: in expansion of macro ‘R_SHORT_MAX’
  166 |         if ( x < R_SHORT_MIN || x > R_SHORT_MAX || !R_FINITE(x) )
      |                                     ^~~~~~~~~~~
coerce.h: At global scope:
coerce.h:178:1: error: ‘uint16_t’ does not name a type
  178 | uint16_t coerce_cast(int32_t x) {
      | ^~~~~~~~
coerce.h:178:1: note: ‘uint16_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:191:1: error: ‘uint16_t’ does not name a type
  191 | uint16_t coerce_cast(double x) {
      | ^~~~~~~~
coerce.h:191:1: note: ‘uint16_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = int; Tin = short int]’:
matterDefines.h:100:18: error: ‘INT16_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  100 | #define NA_SHORT INT16_MIN
      |                  ^~~~~~~~~
coerce.h:215:19: note: in expansion of macro ‘NA_SHORT’
  215 |         if ( x == NA_SHORT )
      |                   ^~~~~~~~
coerce.h: At global scope:
coerce.h:222:21: error: ‘uint32_t’ was not declared in this scope
  222 | int32_t coerce_cast(uint32_t x) {
      |                     ^~~~~~~~
coerce.h:222:21: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:222:9: warning: too many template headers for ‘coerce_cast’ (should be 0)
  222 | int32_t coerce_cast(uint32_t x) {
      |         ^~~~~~~~~~~
coerce.h:222:32: error: expected ‘;’ before ‘{’ token
  222 | int32_t coerce_cast(uint32_t x) {
      |                                ^~
      |                                ;
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = int; Tin = long int]’:
matterDefines.h:104:22: error: ‘INT32_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  104 | #define R_INT_MIN (1+INT32_MIN)
      |                      ^~~~~~~~~
coerce.h:233:18: note: in expansion of macro ‘R_INT_MIN’
  233 |         if ( x < R_INT_MIN || x > R_INT_MAX )
      |                  ^~~~~~~~~
matterDefines.h:105:19: error: ‘INT32_MAX’ was not declared in this scope
  105 | #define R_INT_MAX INT32_MAX
      |                   ^~~~~~~~~
coerce.h:233:35: note: in expansion of macro ‘R_INT_MAX’
  233 |         if ( x < R_INT_MIN || x > R_INT_MAX )
      |                                   ^~~~~~~~~
matterDefines.h:105:19: note: ‘INT32_MAX’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
  105 | #define R_INT_MAX INT32_MAX
      |                   ^~~~~~~~~
coerce.h:233:35: note: in expansion of macro ‘R_INT_MAX’
  233 |         if ( x < R_INT_MIN || x > R_INT_MAX )
      |                                   ^~~~~~~~~
matterDefines.h:115:17: error: ‘INT64_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  115 | #define NA_LONG INT64_MIN
      |                 ^~~~~~~~~
coerce.h:235:27: note: in expansion of macro ‘NA_LONG’
  235 |                 if ( x != NA_LONG )
      |                           ^~~~~~~
coerce.h: At global scope:
coerce.h:244:21: error: ‘uint64_t’ was not declared in this scope
  244 | int32_t coerce_cast(uint64_t x) {
      |                     ^~~~~~~~
coerce.h:244:21: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:244:9: warning: too many template headers for ‘coerce_cast’ (should be 0)
  244 | int32_t coerce_cast(uint64_t x) {
      |         ^~~~~~~~~~~
coerce.h:244:32: error: expected ‘;’ before ‘{’ token
  244 | int32_t coerce_cast(uint64_t x) {
      |                                ^~
      |                                ;
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = int; Tin = double]’:
matterDefines.h:104:22: error: ‘INT32_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  104 | #define R_INT_MIN (1+INT32_MIN)
      |                      ^~~~~~~~~
coerce.h:263:18: note: in expansion of macro ‘R_INT_MIN’
  263 |         if ( x < R_INT_MIN || x > R_INT_MAX || !R_FINITE(x) )
      |                  ^~~~~~~~~
matterDefines.h:105:19: error: ‘INT32_MAX’ was not declared in this scope
  105 | #define R_INT_MAX INT32_MAX
      |                   ^~~~~~~~~
coerce.h:263:35: note: in expansion of macro ‘R_INT_MAX’
  263 |         if ( x < R_INT_MIN || x > R_INT_MAX || !R_FINITE(x) )
      |                                   ^~~~~~~~~
matterDefines.h:105:19: note: ‘INT32_MAX’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
  105 | #define R_INT_MAX INT32_MAX
      |                   ^~~~~~~~~
coerce.h:263:35: note: in expansion of macro ‘R_INT_MAX’
  263 |         if ( x < R_INT_MIN || x > R_INT_MAX || !R_FINITE(x) )
      |                                   ^~~~~~~~~
coerce.h: At global scope:
coerce.h:275:1: error: ‘uint32_t’ does not name a type
  275 | uint32_t coerce_cast(int32_t x) {
      | ^~~~~~~~
coerce.h:275:1: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:288:1: error: ‘uint32_t’ does not name a type
  288 | uint32_t coerce_cast(double x) {
      | ^~~~~~~~
coerce.h:288:1: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = long int; Tin = int]’:
matterDefines.h:115:17: error: ‘INT64_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  115 | #define NA_LONG INT64_MIN
      |                 ^~~~~~~~~
coerce.h:305:24: note: in expansion of macro ‘NA_LONG’
  305 |                 return NA_LONG;
      |                        ^~~~~~~
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = long int; Tin = double]’:
matterDefines.h:115:17: error: ‘INT64_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  115 | #define NA_LONG INT64_MIN
      |                 ^~~~~~~~~
coerce.h:316:24: note: in expansion of macro ‘NA_LONG’
  316 |                 return NA_LONG;
      |                        ^~~~~~~
coerce.h: At global scope:
coerce.h:324:1: error: ‘uint64_t’ does not name a type
  324 | uint64_t coerce_cast(int32_t x) {
      | ^~~~~~~~
coerce.h:324:1: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h:338:1: error: ‘uint64_t’ does not name a type
  338 | uint64_t coerce_cast(double x) {
      | ^~~~~~~~
coerce.h:338:1: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = double; Tin = short int]’:
matterDefines.h:100:18: error: ‘INT16_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  100 | #define NA_SHORT INT16_MIN
      |                  ^~~~~~~~~
coerce.h:364:19: note: in expansion of macro ‘NA_SHORT’
  364 |         if ( x == NA_SHORT )
      |                   ^~~~~~~~
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = double; Tin = long int]’:
matterDefines.h:115:17: error: ‘INT64_MIN’ was not declared in this scope; did you mean ‘INT_MIN’?
  115 | #define NA_LONG INT64_MIN
      |                 ^~~~~~~~~
coerce.h:380:19: note: in expansion of macro ‘NA_LONG’
  380 |         if ( x == NA_LONG )
      |                   ^~~~~~~
atoms.h: In member function ‘size_t Atoms::get_atom(T*, int, index_t, size_t, int)’:
atoms.h:324:58: error: ‘uint16_t’ was not declared in this scope
  324 |                                         return read_atom<uint16_t,T>(ptr, atom, pos, size, stride);
      |                                                          ^~~~~~~~
atoms.h:10:1: note: ‘uint16_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
    9 | #include "drle.h"
  +++ |+#include <cstdint>
   10 | 
atoms.h:328:58: error: ‘uint32_t’ was not declared in this scope
  328 |                                         return read_atom<uint32_t,T>(ptr, atom, pos, size, stride);
      |                                                          ^~~~~~~~
atoms.h:328:58: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
atoms.h:332:58: error: ‘uint64_t’ was not declared in this scope
  332 |                                         return read_atom<uint64_t,T>(ptr, atom, pos, size, stride);
      |                                                          ^~~~~~~~
atoms.h:332:58: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
atoms.h: In member function ‘size_t Atoms::set_atom(T*, int, index_t, size_t, int)’:
atoms.h:353:61: error: ‘uint16_t’ was not declared in this scope
  353 |                                         return write_atom<T,uint16_t>(ptr, atom, pos, size, stride);
      |                                                             ^~~~~~~~
atoms.h:353:61: note: ‘uint16_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
atoms.h:357:61: error: ‘uint32_t’ was not declared in this scope
  357 |                                         return write_atom<T,uint32_t>(ptr, atom, pos, size, stride);
      |                                                             ^~~~~~~~
atoms.h:357:61: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
atoms.h:361:61: error: ‘uint64_t’ was not declared in this scope
  361 |                                         return write_atom<T,uint64_t>(ptr, atom, pos, size, stride);
      |                                                             ^~~~~~~~
atoms.h:361:61: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = int; Tin = short int]’:
coerce.h:219:1: warning: control reaches end of non-void function [-Wreturn-type]
  219 | }
      | ^
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = int; Tin = long int]’:
coerce.h:241:1: warning: control reaches end of non-void function [-Wreturn-type]
  241 | }
      | ^
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = double; Tin = short int]’:
coerce.h:368:1: warning: control reaches end of non-void function [-Wreturn-type]
  368 | }
      | ^
coerce.h: In function ‘Tout coerce_cast(Tin) [with Tout = double; Tin = long int]’:
coerce.h:384:1: warning: control reaches end of non-void function [-Wreturn-type]
  384 | }
      | ^
make: *** [/usr/lib64/R/etc/Makeconf:200: altrep.o] Error 1
ERROR: compilation failed for package ‘matter’
* removing ‘/home/user/R/x86_64-pc-linux-gnu-library/4.3/matter’

The downloaded source packages are in
	‘/tmp/Rtmp86H3Mo/downloaded_packages’
Warning message:
In install.packages(...) :
  installation of package ‘matter’ had non-zero exit status

Now I am wondering if my machine is missing some package as a dependency (which is most often the case when I run into such problems), or if anything else is the culprit. All my other packages are up to date, (e.g., Matrix 1.5-4.1), but the final error seems to occure while compiling altrep.cpp in line 200, which, as I can see, is the // ALTINTEGER definition. And to me that seems to be somewhat unlikely...

Do you have an idea?

Thanks a lot!

matter::sparse_mat path

I am probably missing something, but I could not find a way to add the path to the constructor for matter::sparse_mat. Can you let me know how this is done?

And, thanks so much for the great work on this package!

On the operation of sparse matrices in high-throughput data

Do you have any insight on how to find the maximum value of each element of a sparse matrix? Because the PMAX function does not support sparse matrix operations which is not friendly to high throughput operations. Thank you.

`nchunks` calculated wrongly from `chunksize` if `depends != NULL`

For the new chunked classes, if depends != NULL (or if we add a way of specifying a custom index for the iteration, e.g., with the planned permute argument), then the constructors may calculate the wrong number of chunks to use when the chunksize option is specified.

The effective chunk size may be too large in this case.

This is because when depends is specified, the same elements may be included in multiple chunks, and we currently only use the realized object size in the calculation.

argument data is missing with no default

Hi, there,
As I was loading the Imzml data,
I found this:

		mz <- matter_list(path=file,
			type=Ctypeof(mz.ibdtype),
			offset=mzData(info)[["external offset"]],
			extent=mzData(info)[["external array length"]])
		intensity <- matter_list(path=file,
			type=Ctypeof(intensity.ibdtype),
			offset=intensityData(info)[["external offset"]],
			extent=intensityData(info)[["external array length"]])

will trigger the error message: argument "data" is missing with no default

Best,
George

Add noise estimation for multidimensional signals

We need robust noise estimation in for N-D signals.

We have x(t) = f(t) + n(t) and want to determine n(t).

We can use a filter to estimate f^(t) from x(t), then estimate n^(t) = x(t) - f^(t).

Then we can summarize the noise n^(t) using sd(), mad(), quantile(), etc.

What filter to use for estimating f^(t)? Gaussian? Mean? Median?

[Bug]: NAs not correctly propagated for summary statistics of integer matter_mat

Hi Kylie,

The summary statistics for matter objects (colSums(), etc.) return incorrect results when NA are present and the datamode of the matter object is "integer":

suppressPackageStartupMessages(library(matter))

y <- matrix(c(1:8, NA_integer_), ncol = 3)
Y_integer <- matter_mat(y, nrow = nrow(y), ncol = ncol(y), datamode = "integer")

colSums(y)
#> [1]  6 15 NA
colSums(Y_integer)
#> [1]           6          15 -2147483633

colSums(y, na.rm = TRUE)
#> [1]  6 15 15
colSums(Y_integer, na.rm = TRUE)
#> [1]           6          15 -2147483633

Everything seems to be fine if the datamode of the matter object is "double":

suppressPackageStartupMessages(library(matter))

y <- matrix(c(1:8, NA_integer_), ncol = 3)
Y_double <- matter_mat(y, nrow = nrow(y), ncol = ncol(y), datamode = "double")

colSums(y)
#> [1]  6 15 NA
colSums(Y_double)
#> [1]  6 15 NA

colSums(y, na.rm = TRUE)
#> [1]  6 15 15
colSums(Y_double, na.rm = TRUE)
#> [1]  6 15 15

Cheers,
Pete

Simulating images

Should we add a simple function for simulating (non-spectral) images for examples and testing purposes?

Add `serialize = TRUE` option to chunk apply functions

When TRUE, we serialize the data from matter objects to workers.

When FALSE, we only serialize the metadata and the data is loaded locally on the workers. (Note: This requires the host and workers see the same paths to the data files!)

Supporting linear indexing

Would it be possible to support linear indexing of matter objects, as supported by base::array. E.g.,

suppressPackageStartupMessages(library(matter))
z <- matrix(1:10, ncol = 2)
# base: Extracts the 3rd element
z[3]
#> [1] 3
# matter: Extracts the 3rd row
Z <- matter_mat(z)
Z[3]
#> [1] 3 8

z2 <- array(z, dim = c(dim(z), 1))
dim(z2)
#> [1] 5 2 1
# base: Extracts the 3rd element
z2[3]
#> [1] 3
Z2 <- matter_arr(z2)
# matter: errors
Z2[3]
#> Error in Z2[3]: incorrect number of dimensions

Estimate noise functions should ignore peak windows

For the summary-based estnoise_*() functions, can we ignore the samples that are within proximity of peak candidates?

This may best be done by having locmax() return a vector of pure noise indices (as an attribute), i.e., indices of samples that are not included in the sliding windows of detected local maxima.

Add support for complete Ops group generic for `matter_arr`

Outdated documentation says we support all Ops (Arith, Compare, Logic), but only Arith is currently supported. Add support for Compare and Logic group generics.

Support multiple channels for image processing functions

This is already done for trans2d, but should be done for the various smoothing and contrast enhancement functions too.

binding of matters from different files doesnt work properly

Page six of this vignette shows how to cbind vectors that holds data from different files, but seems that it works not as intended, at least cbind(x2, y3)[] and cbind(x[,2], y[,3]) produces different results while they are expected to be the same.

Small inconsistency between `[<-` methods for matter_arr and array

Hi Kylie,

The [<- replacement method seems to work the same for base::matrix and matter_mat objects, but not so for base::array and matter_arr objects. An example is shown below. Is this fixable? I hit this when using programatically constructed indices passed to the [<- method for the matter_arr class via a do.call().

suppressPackageStartupMessages(library(matter))

m <- matrix(1:6, nrow = 2, ncol = 3)
M <- matter_mat(m)
M[]
#>      [,1] [,2] [,3]
#> [1,]    1    3    5
#> [2,]    2    4    6
# works
M[] <- matrix(6:1, nrow = 2, ncol = 3) 
M[]
#>      [,1] [,2] [,3]
#> [1,]    6    4    2
#> [2,]    5    3    1
# works
M[, ] <- matrix(-(1:6), nrow = 2, ncol = 3)
M[]
#>      [,1] [,2] [,3]
#> [1,]   -1   -3   -5
#> [2,]   -2   -4   -6

a <- array(1:8, dim = c(2, 2, 2))
A <- matter_arr(a)
A[]
#> , , 1
#> 
#>      [,1] [,2]
#> [1,]    1    3
#> [2,]    2    4
#> 
#> , , 2
#> 
#>      [,1] [,2]
#> [1,]    5    7
#> [2,]    6    8
# works
A[] <- array(8:1, dim = c(2, 2, 2))
A[]
#> , , 1
#> 
#>      [,1] [,2]
#> [1,]    8    6
#> [2,]    7    5
#> 
#> , , 2
#> 
#>      [,1] [,2]
#> [1,]    4    2
#> [2,]    3    1
# errors for matter_arr (whereas works for base::array)
A[, , ] <- array(-(1:8), dim = c(2, 2, 2))
#> Error in `[<-`(`*tmp*`, , , , value = structure(-1:-8, .Dim = c(2L, 2L, : argument is missing, with no default
a[, , ] <- array(-(1:8), dim = c(2, 2, 2))
a
#> , , 1
#> 
#>      [,1] [,2]
#> [1,]   -1   -3
#> [2,]   -2   -4
#> 
#> , , 2
#> 
#>      [,1] [,2]
#> [1,]   -5   -7
#> [2,]   -6   -8

Add support for Ops generics for `drle`

Should be straightforward to add support for Ops generics for drle classes (by decoding the compressed vector and calling the base method).

R session hangs when `data` is missing if `paths` is supplied to matter_mat()

Hi Kyle,

This is a corner case, but it'd be nice if it resulted in an error rather than hanging the R session (similar to your suggestion in #2 (comment))

library(matter)

x <- matter_mat(nrow = 5L, ncol = 2L, datamode = "integer")
# Works despite no `data` being supplied (I think because default values are filled in)
x[]
# Can update `x`
x[] <- matrix(1:10, ncol = 2)

paths <- tempfile(fileext = ".bin")
file.create(paths)
y <- matter_mat(nrow = 5L, ncol = 2L, paths = paths)
# Hangs because no data is created (I think because no default values are filled in)
y[]
y[] <- matrix(1:10, ncol = 2)

Thanks,
Pete

`if` missing closing '}'

Oh! There is an obvious error in your latest version that prevents me from installing it!

https://github.com/kuwisdelu/matter/blob/devel/R/sgmix.R

        if ( is.matrix(vals) ) {
		ans <- chunkApply(vals, margin, sgmix_int,
			x=x, y=y, r=r, k=k, ...,
        #  Line 371                                ^
	if ( is.list(vals) ) {
		ans <- chunkLapply(vals, sgmix_int,
			x=x, y=y, r=r, k=k, ...,
			nchunks=nchunks, verbose=verbose,
			RNG=TRUE, BPPARAM=BPPARAM)
	}

Adaptive bilateral filter is too adaptive

Investigate cases where filt1_adapt and filt2_adapt end up adapting to the noise rather than the local signal.

`to_raster()` fails for non-gridded character rasters

This needs to work.

Problems using filemode = "rb"

Hi Kylie,

I've just started exploring matter, it looks really cool! I hit an issue trying to specify filemode = "rb" when creating a matter matrix. Am I doing something wrong?

Thanks

library(matter)
#> Loading required package: biglm
#> Loading required package: DBI
#> 
#> Attaching package: 'matter'
#> The following objects are masked from 'package:base':
#> 
#>     apply, scale

# Using the default filemode = "rb+" works fine
x <- matter_mat(1:100, nrow=10, ncol=10)
x
#> An object of class 'matter_matc'
#>   <10 row, 10 column> on-disk matrix
#>     sources: 1 
#>     datamode: numeric 
#>     12.1 KB in-memory
#>     800 bytes on-disk

# Specifying filemode = "rb" causes R to hang (killed after 30 seconds)
x <- matter_mat(1:100, nrow=10, ncol=10, filemode = "rb")

sessionInfo()
#>R version 3.4.0 (2017-04-21)
#>Platform: x86_64-apple-darwin15.6.0 (64-bit)
#>Running under: macOS Sierra 10.12.5
#>
#>Matrix products: default
#>BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
#>LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
#>
#>locale:
#>[1] en_AU.UTF-8/en_AU.UTF-8/en_AU.UTF-8/C/en_AU.UTF-8/en_AU.UTF-8
#>
#>attached base packages:
#>[1] stats     graphics  grDevices utils     datasets  methods   base
#>
#>other attached packages:
#>[1] matter_1.3.3 biglm_0.9-1  DBI_0.6-1

#>loaded via a namespace (and not attached):
#>[1] compiler_3.4.0      Matrix_1.2-10       parallel_3.4.0
#>[4] grid_3.4.0          BiocGenerics_0.23.0 irlba_2.2.1
#>[7] lattice_0.20-35

Looking forward to a vignette

Is this like a SAS-killer?

Have you heard of disk.frame? See http://diskframe.com

Is there a way to load csv

I just heard your talk and tried the package. It looks very good. But somehow I couldn't find a method to load data from a csv. All the examples mentioned in vignette were random numbers too...

Please let me know if there is a way to load csv data in matter.

cbind not working when binding named matrices with different number of columns

As in title. Here is reproducible example

library(matter)
vec1 <- sample(0:9, 1000, TRUE, prob = c(81, rep(1, 9)))
vec2 <- sample(0:9, 500, TRUE, prob = c(81, rep(1, 9)))
rows <- paste0("row", 1:50)
cols <- paste0("col", 1:30)
mat1 <- matrix(vec1, nrow = 50, ncol = 20, dimnames = list(rows, cols[1:20]))
mat2 <- matrix(vec2, nrow = 50, ncol = 10, dimnames = list(rows, cols[21:30]))

matt1 <- as.sparse(mat1)
matt2 <- as.sparse(mat2)
matt3 <- cbind(matt1, matt2)

Error message:

 Error in validObject(.Object) : 
  invalid class “sparse_matc” object: length of 'dimnames' [2] not equal to array extent

I have also found the solution. Problem is in matter:::combine_colnames. Buggy part is like this:

	} else if ( is.null(dimnames(y)[[2]]) ) {
		colnames <- c(dimnames(x)[[2]], character(dim(x)[2]))
	} else {
		colnames <- c(dimnames(x)[[2]], dimnames(x)[[2]])

Should be like this. There is change from x to y in second arguments of c():

	} else if ( is.null(dimnames(y)[[2]]) ) {
		colnames <- c(dimnames(x)[[2]], character(dim(y)[2]))
	} else {
		colnames <- c(dimnames(x)[[2]], dimnames(y)[[2]])

Same problem in the similar part of code is also in matter:::combine_rownames.

Thanks,
Daniel

[Suggestion]: automatic propagation of dimnames

Hi Kylie,

Would you consider making some (or all of) matter(), matter_*(), and as.matter() support automatic propagation of dimnames?

library(matter)
#> Loading required package: biglm
#> Loading required package: DBI
#> 
#> Attaching package: 'matter'
#> The following objects are masked from 'package:base':
#> 
#>     apply, scale
x <- matrix(1:10, ncol = 2, dimnames = list(NULL, c("A", "B")))
dimnames(x)
#> [[1]]
#> NULL
#> 
#> [[2]]
#> [1] "A" "B"
dimnames(as.matter(x))
#> NULL
dimnames(matter(x))
#> NULL
dimnames(matter_mat(x))
#> NULL

Thanks,
Pete

Add local maxima detection for multidimensional signals

Can we implement locmax2() and locmaxn() for multidimensional local maxima?

For locmax2(), assuming a gridded 2D image, we can use the same idea: use a sliding window and a local maxima must be greater than all other samples in the window.

For locmaxn(), for the general case of nonuniformly sampled N-D signals, we can rely on knnsearch().

Add unit tests for visualization

We can verify the vizi_plot and vizi_facets objects.

Might be best to just have a gallery vignette stand-in as the test for the actual plotting?

Parallel functionals in `rowDists()` and `rowStats()` are leaky

Need to protect them with local() like the other parallel functionals.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.