Should be able to render docs and view docs for functions in the REPL.

Installing on Mac went fine. On linux, after installing TileDB via conda as described here:

https://docs.tiledb.io/en/stable/installation.html#installation

R CMD INSTALL TileDB-R

give me

** testing if installed package can be loaded
Error: package or namespace load failed for ‘tiledb’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/gnet/is2/p01/home/phaverty/Rlib/3.4.3-Bioc-3.6-current/tiledb/libs/tiledb.so':
/gnet/is2/p01/home/phaverty/Rlib/3.4.3-Bioc-3.6-current/tiledb/libs/tiledb.so: undefined symbol: tiledb_config_iter_alloc
Error: loading failed

I think I've got all the library paths set correctly since it compiled. This may be a legit error in the C++ code in TileDB-R. Have you seen this before? tiledb.h seems to export this symbol. Stumped.

We will have to build the package from source, document it in the README.

In the quickstart example for a sparse array

write_array <- function() {
I <- c(1L, 2L, 2L)
J <- c(1L, 4L, 3L)
data <- c(1L, 2L, 3L)
# Open the array and write to it.
ctx <- tiledb_ctx()
A <- tiledb_sparse(ctx, uri = array_name)
A[I, J] <- data
}

The desired effect for a standard R matrix would be obtained by passing a two-column matrix as a single argument to the "[" function, i.e.

A[ cbind(I,J) ] <- data

This is a bit counter-intuitive, but that's how it works. As it is written, A[I,J] <- data should give

m
[,1] [,2] [,3] [,4]
[1,] 1 NA 1 1
[2,] 3 NA 3 3
[3,] NA NA NA NA
[4,] NA NA NA NA

Imagine that it is asking for a matrix with all the rows specified by I, all the columns specified by J and then repeating data down the columns until it fills up this matrix. Weird, but true.

I have a tiledb array constructed from tiff file using

gdal_translate -OF TileDB input.tif output.tdb

Input file size is 37307, 29883

Now I want to read it with:

A <- tiledb_dense(uri = "output.tdb")
chunk  = A[, , 1:2]

And I'm getting:

Error in libtiledb_query_set_subarray(qry, as.integer(subarray)) :
Static type (INT32) does not match expected type (UINT64)

In the README, tiledb::libtiledb_version() should use the triple colon as libtiledb_version is not exported. (Or was it meant to be exported?) I'm on tiledb 0.1

Get the package in good enough shape to submit to CRAN

Requires windows support and better dependency packaging, as well as api stabilization and a better understanding of CRAN submission guidelines.

Can tiledb_dense and tiledb_sparse inherit from an abstract type like tiledb? I think some of the code for those specific types can be made methods on the abstract parent class.

libtiledb.cpp:227:24: error: ‘tiledb::Version’ has not been declared
     auto ver = tiledb::Version::version();
                        ^
libtiledb.cpp: In function ‘Rcpp::List tiledb_array_schema_attributes(Rcpp::XPtr<tiledb::ArraySchema>)’:
libtiledb.cpp:847:30: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (unsigned int i=0; i < nattr; i++) {
                              ^
libtiledb.cpp: In function ‘std::__cxx11::string _query_status_to_string(tiledb::Query::Status)’:
libtiledb.cpp:1073:10: error: ‘UNDEF’ is not a member of ‘tiledb::Query::Status’
     case tiledb::Query::Status::UNDEF:

TiltDB lib was built from source from dev branch

/TileDB$ git remote -v
origin	https://github.com/TileDB-Inc/TileDB (fetch)
origin	https://github.com/TileDB-Inc/TileDB (push)
wjiang2@rhino2:~/rglab/workspace/TileDB$ git branch 
* dev

Currently the R package is only built / tested on Linux and OSX

Todo's:

• Setup CI testing for windows with TileDB installed from source #8
• Adapt package configure / install scripts to use Cmake to build and install TileDB as a binary dep of the R package (similar TileDB's pip package).
• Setup conda packaging for TileDB on Windows
• Setup conda packaging for TileDB-R on Windows

In the readme,

devtools::install_github("TileDB-Inc/TileDB-R@latest")

does not work, but

devtools::install_github("TileDB-Inc/TileDB-R@master") or devtools::install_github("TileDB-Inc/TileDB-R") does

Dense Arrays don't support sparse writes due to the implementation of the [ <- function. This shouldn't be too hard to support, just need to support passing in the coordinates for the write.

Do config and tiledb_is_supported_fs need to be S4 generics with methods? They each have only one method, so a regular function would do. You might want to hide the lookup of the object@ptr in a ptr accessor, though.

https://www.rdocumentation.org/packages/Rcpp/versions/0.12.15/topics/Rcpp.package.skeleton

TileDB type support is limited to scalars, and fixed length attribute types.

Need a simplified constructor(s) with lots of optional parameters to simplify saving an R array / vector to a TileDB array in as few lines as possible.

ctx <- tiledb_ctx()

appears in many places in the R API. Does it change over time? Maybe create ctx at load time and just have the functions that need it grab it from the package namespace. That might simplify the code a bit.

R package do stuff at load time by calling a function, which you can define called ".onLoad()" typically this goes in R/zzz.R so it will evaluate last. It might look like this:

.onLoad <- function(libname, pkgName) {
ns <- asNamespace(pkgName)
delayedAssign("ctx", tiledb_ctx(),
eval.env = ns, assign.env = ns)
namespaceExport(ns, "ctx")
}

and then you can use it in your code with tiledb::ctx

Advantages:

• light weight, fewer dependencies
• does the right thing and installs tests with package by default allowing tests to (re-)run once installed
• greater familiarity (that's a personal preference)

See https://cloud.r-project.org/web/packages/tinytest/index.html

Two is just barbaric

related to TileDB-Inc/TileDB#675, which seems to be supported now in TileDB core lib?

EDITED:
Basically would like to do non-contiguous subscript , e.g.

tmp <- tempfile()
dir.create(tmp)
d1  <- tiledb_dim("d1", domain = c(1L, 5L))
d2  <- tiledb_dim("d2", domain = c(1L, 5L))
dom <- tiledb_domain(c(d1, d2))
val <- tiledb_attr("val", type = "FLOAT64")
sch <- tiledb_array_schema(dom, c(val))
tiledb_array_create(tmp, sch)

dat <- matrix(rnorm(25), 5, 5)
arr <- tiledb_dense(tmp, as.data.frame=FALSE)

arr[] <- dat

> expect_equal(arr[list(c(3,5), c(3,5))],
+              dat[c(3,5), c(3,5)])
 Error in domain_subarray(dom, index = index) : 
  non-contiguous subscript ranges are not supported 

It should be possible to return the (major, minor, rev) tuple (or whatever the R equiv. is

Follow convention in the Python API and have the context as the last argument with default.

We need to query the array schema to check if the array is sparse and set the default layout from COL_MAJOR to GLOBAL_ORDER

It looks like INT32 and FLOAT64 types are supported. I think TileDB can do variable length things, including strings, right?
I'm interested in storing rownames and colnames next to a matrix.

See if it is possible for the R package to download and statically link TileDB when building the Rcpp extension (similar to the the pip python package).

When subsetting, R drops single-element dimensions. For example, pulling a column from a matrix gives a vector, rather than a one-column matrix. TileDB-R should probably follow this as users will be confused by the difference from standard arrays.

Sadly, CRAN is still using a very old version Windows 2008 (a.k.a. Vista) server to test packages. Some unit test are failing on the server due to outdated/missing crypto algorithms on Windows 2008.

This does not affect users in any way, the binaries itself are fine, and will work for users. You just need to skip a few tests on the CRAN winbuilder (but of course still run those tests on your Windows CI). Testthat has a handy function skip_on_cran() for this, I don't know how tinytest does this.

The easiest way to test this is submitting to winbulder. The package succeeds to build and load, which is the important part, but the examples fail with:

> ## Don't show: 
> ctx <- tiledb_ctx(limitTileDBCores())
> ## End(Don't show)
> uri <- tempfile()
> ## turn factor into character
> irisdf <- within(iris, Species <- as.character(Species))
> fromDataFrame(irisdf, uri)
Error in libtiledb_array_open(ctx@ptr, uri, query_type) : 
  [TileDB::IO] Error: Failed to list directory.
Calls: fromDataFrame -> tiledb_dense -> libtiledb_array_open

And the tests fail with with Win32CNG (which is the Windows crypto library):

Error in tiledb:::libtiledb_array_create_with_key(uri, schema@ptr, key) : 
    [TileDB::Encryption] Error: Win32CNG error; error importing key blob.
  Calls: <Anonymous> ... run_test_dir -> lapply -> FUN -> eval -> eval -> <Anonymous>

Again, these tests only fail on very old, no longer supported versions of Windows. I verified that I get the same errors when running in a Windows Vista vm, but on Windows 7 and up, everything is 👍 so nothing to worry about.

R provides a C api to create custom connection objects (although it is not stable). We can use this with Rcpp to provide a TileDB R connection object. Hopefully that would enable much more functionality "for free" from R using the VFS backends.

Simplify mapping of R to TileDB datatypes by supporting storage.mode type strings as valid inputs when a tiledb type string is required (ex. tiledb_attr)

To disambiguate type tag strings, we should probably deprecate the current type strings and use TILEDB_<typename> instead.

Suggested change:
#48 (comment)

Pre-allocating a vector using "vector" in attribute_buffers is about 15pct of the time for big reads. On a circa 2015 macbook, this is 0.12s for a 30592 row by 1095 column dense matrix. R fills in the vector with NAs for you but I can't imagine what is doing that takes that long. You might consider moving attribute_buffers, maybe the whole pre-allocate and fill operation down to Rcpp, where I imagine the allocation would be many orders faster. A 15% speed boost could help as HDF5 approaches parity for big matrices. Here I compare HDF5, bigmemory (a mmap approach) and TileDB-R. (Obviously the second example should be done with sparse matrices, but I'm still figuring that out.)

## Timings on vst matrix,  30592 rows by 1095 columns dense numeric (double) matrix                                                                                          

system.time({ bm = BigMatrix(vst, "vst.bm") }) # 4.1s                                                                                                                        
system.time({ bm2 = bm[] }) # 0.3s                                                                                                                                           

system.time({ h5write(vst, "vst.h5", "vst") }) # 8.3s                                                                                                                        
system.time({ h5 <- h5read("vst.h5", "vst") }) # 1.0s                                                                                                                        

system.time( { tdb = tiledb_array(vst, "vst")}) # 2.9s                                                                                                                       
system.time( { tdb2 = tdb[]}) # 0.3s                    

Timings on x matrix, 30592 rows by 1095 columns dense numeric (double) matrix

97pct zeros, try with sparse later ...

system.time({ h5write(x, "hdf5_test.h5", "foo") }) # 48s
system.time({ h5read("hdf5_test.h5", "foo", index=list(NULL,100)) }) # 7.5s
system.time({ x <- h5read("hdf5_test.h5", "foo") }) # 13.5s

system.time({ bm = BigMatrix(x, "x.bm") }) # 39s
system.time({ bm2 = bm[,100] }) # 0.02s
system.time({ bm2 = bm[] }) # 3s

system.time( { tdb = tiledb_array(x, "x")}) # 31s
system.time( { junk = tdb[,100]}) # 7s
system.time( { junk = tdb[]}) # 13s

Rprof shows where the time goes

rp
$by.self
self.time self.pct total.time total.pct
"libtiledb_query_submit" 0.62 81.58 0.62 81.58
"vector" 0.12 15.79 0.12 15.79
"libtiledb_query_set_buffer" 0.02 2.63 0.02 2.63

$by.total
total.time total.pct self.time self.pct
"[" 0.76 100.00 0.00 0.00
"tryCatch" 0.76 100.00 0.00 0.00
"tryCatchList" 0.76 100.00 0.00 0.00
"libtiledb_query_submit" 0.62 81.58 0.62 81.58
"vector" 0.12 15.79 0.12 15.79
"attribute_buffers" 0.12 15.79 0.00 0.00
"libtiledb_query_set_buffer" 0.02 2.63 0.02 2.63

For arrays with multiple attributes, you might want to use the list API. For the case of two attributes that are each matrices, you could write

x$a[1:4,1:2]
or
x[["a"]][1:4,1:2]

You might want to do this in a lazy fashion so that x$a didn't pull the whole of matrix "a". Rather you'd want to index into "a" and the apply the rows and column indices.

Hello,

v0.9.0 of the package is inconsistent with reordering values and has incorrect mapping between indices and values

Script to store a matrix as a tiledb object

library(tiledb)

uri <- "~/test10.tdb"

# Generate a matrix
mat <- matrix(1:20, nrow=5, ncol=4)

dom <- tiledb::tiledb_domain(dims = 
 c(tiledb::tiledb_dim("rows", c(1L, nrow(mat) * ncol(mat)), nrow(mat)),
   tiledb::tiledb_dim("cols", c(1L, nrow(mat) * ncol(mat)), ncol(mat)))

schema <- tiledb::tiledb_array_schema(dom, 
  attrs=c(tiledb::tiledb_attr("vals", type=ifelse(all(is.integer(mat)), "INT32", "FLOAT64"))))

tiledb::tiledb_array_create(uri, schema)

arr <- tiledb::tiledb_array(uri, query_type="WRITE", is.sparse=FALSE)

mat_df <- data.frame(rows=as.vector(row(mat)), 
        cols=as.vector(col(mat)), 
        vals=as.vector(mat))

arr[] <- mat_df

What we are adding to the tiledb object

> mat_df
   rows cols vals
1     1    1    1
2     2    1    2
3     3    1    3
4     4    1    4
5     5    1    5
6     1    2    6
7     2    2    7
8     3    2    8
9     4    2    9
10    5    2   10
11    1    3   11
12    2    3   12
13    3    3   13
14    4    3   14
15    5    3   15
16    1    4   16
17    2    4   17
18    3    4   18
19    4    4   19
20    5    4   20

v0.9.0

reorders the rows and cols but not the values.

> arr[]
$rows
 [1] 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5

$cols
 [1] 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

$vals
 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20

v0.8.2

reorders values as well

> arr[]
$rows
 [1] 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5

$cols
 [1] 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

$vals
 [1]  1  6 11 16  2  7 12 17  3  8 13 18  4  9 14 19  5 10 15 20

Running CMD check hangs after running the examples or tests (probably during gc() cleanup). I suspect something in your finalizers in causing the process to freeze on exit, preventing R from quitting.

This happens with TileDB 2.0.8 both in 32-bit and 64-bit.

command line docs are needed

LGLSXP are just INT32 arrays under the hood. They only use 0, 1 and INT32_MIN. We could just support them by treating them as INT32 for now. Maybe later we could teach tiledb the difference with TILEDB_BOOL or something. It probably is not worth actually casting the 32 bit ints to something smaller.

Running examples in 32-bit fails here:

> ### Name: tiledb_array_schema
> ### Title: Constructs a 'tiledb_array_schema' object
> ### Aliases: tiledb_array_schema
> 
> ### ** Examples
> 
> ## Don't show: 
> ctx <- tiledb_ctx(limitTileDBCores())
Error in libtiledb_ctx(config@ptr) : 
  [TileDB::C++API] Error: Failed to create context
Calls: tiledb_ctx -> libtiledb_ctx
Execution halted

This does not seem to happen in the cran version 0.7.0 of the R package, only in the dev version.

TileDB-R today attempts to download the latest TileDB Embedded (core c++ library) for use in the installation. Downloading the latest version has cause problems, first a minor issue with TileDB 2.0.6 and external libz linkage, and second with TileDB 2.0.8 when additional assets were added that broken the download script. As a result of this breakage downloading the latest version was revisited.

It has been decided to switch TileDB-R to use a pinned version of TileDB. This means for downloading the prebuilt binary and for compiling from source a fixed version of TileDB will be used. The advantages of this are the TileDB-R package version will now offer reproducible builds. We also will have the full TileDB-R test suite run via CI when we look to update the TileDB version in the package. There are of course some potential downsides, namely CRAN does not allow for frequent package updates so it is likely the at the CRAN package will have an older version of TileDB than the latest patch release. We’ll monitor the effects of this, and re-evaluate as needed.

TileDB-R will be switched to pinning TileDB 2.0.8, which will take effect with the next TileDB-R release (0.8.0?)

Need more research into which packages are used in practice and what functionality would be useful to support.

Figure out the best way to test R packages

Current functionality is limited to RW (read / write, tiledb metadata re-loaded every access).

Setup Conda packaging for OSX and Linux

I'll use the example generated by ex_1.R. (Note that I needed to add names to the list at:

Otherwise it complains Error: No column names supplied.)

Anyway, once ex_1/ is generated, we can run the following code in 0.9.1 (8cca43b):

> library(tiledb)
> obj <- tiledb_array("ex_1/", as.data.frame=TRUE)
> head(obj[])
  rows cols  a     b c
1    1    1  1 101.0 a
2    1    2 11 106.0 k
3    1    3 21 111.0 u
4    1    4 31 116.0 C
5    1    5 41 121.0 M
6    2    1  2 101.5 b
> obj <- tiledb_array("ex_1/", attrs="a", as.data.frame=TRUE)
> head(obj[])
[1]  1 11 21 31 41  2
> obj <- tiledb_array("ex_1/", attrs=c("a", "b"), as.data.frame=TRUE)
> head(obj[])
   a     b
1  1 101.0
2 11 106.0
3 21 111.0
4 31 116.0
5 41 121.0
6  2 101.5

You can see how specifying attrs= causes the rows and cols columns to be removed, and eventually for the data frame structure to be dropped altogether if attrs is of length 1. This is pretty surprising for downstream code that expects to see the rows and cols columns for further manipulation of the indices.

For comparison, this is what I see on 0.9.0 (20e775c):

> obj <- tiledb_array("ex_1/", as.data.frame=TRUE)
> head(obj[])
  rows cols  a     b c
1    1    1  1 101.0 a
2    1    2 11 106.0 k
3    1    3 21 111.0 u
4    1    4 31 116.0 C
5    1    5 41 121.0 M
6    2    1  2 101.5 b
> obj <- tiledb_array("ex_1/", attrs="a", as.data.frame=TRUE)
> head(obj[])
  rows cols  a
1    1    1  1
2    1    2 11
3    1    3 21
4    1    4 31
5    1    5 41
6    2    1  2
> obj <- tiledb_array("ex_1/", attrs=c("a", "b"), as.data.frame=TRUE)
> head(obj[])
  rows cols  a     b
1    1    1  1 101.0
2    1    2 11 106.0
3    1    3 21 111.0
4    1    4 31 116.0
5    1    5 41 121.0
6    2    1  2 101.5

Found more than one class "Array" in cache; using the first, from namespace 'tiledb'
Also defined by ‘DelayedArray’

It can produced by loading package https://github.com/jakebolewski/singleCell/tree/tiledb

I know it is really the issue of R language itself that should have enforced the namespace to be explicitly typed for any API usage. But unfortunately it is the way it is now.

Since DelayedArray has been around for quite some time and used by dozens of other R packages.
I would recommend to rename tiledb::Array to something else to avoid such conflict.

From the changelog:

The deprecated legacy support of make macros such as CXX1X has been removed: use the CXX11 forms instead.

Took me a while to figure out why my configuration kept on failing. This will represent the state of affairs in R 3.6.0, so probably best to switch sooner rather than later.

In one unit test, the file path is prefixed with file:/ which does not seem to work on Windows:

On Windows, test_file_path here is for example

file://C:\Users\jeroen\AppData\Local\Temp\2\RtmpINkT04\filec14164d60e8/test_file

And that gives an error:

> tiledb:::libtiledb_vfs_is_file(vfs, test_file_path)
Error in tiledb:::libtiledb_vfs_is_file(vfs, test_file_path) : 
  [TileDB::VFS] Error: Unsupported URI scheme: 

@ihnorton did this not appear with your MSVC builds?

Hi @eddelbuettel I'm using a combination of scripts from my earlier issue and from this page - https://docs.tiledb.com/main/solutions/tiledb-embedded/api-usage/writing-arrays/writing-in-dense-subarrays

Hopefully I'm doing something wrong here

library(tiledb)

uri <- "test.tdb"
if (dir.exists(uri)) unlink(uri, recursive=TRUE, force=TRUE)

## Generate a matrix
n <- 5L
k <- 4L
mat <- matrix(1:(n*k) * 0.12345, nrow=n, ncol=k)
print(mat)

dom <- tiledb_domain(dims = c(tiledb_dim("rows", c(1L, n), n, "INT32"),
                              tiledb_dim("cols", c(1L, k), k, "INT32")))
schema <- tiledb_array_schema(dom, attrs=tiledb_attr("vals", type="FLOAT64"))
tiledb_array_create(uri, schema)

arr <- tiledb_array(uri, is.sparse=FALSE)
I <- c(1:3)
J <- c(2:3)

print(mat[I, J])
arr[I, J] <- mat[I, J]

Fails to write in chunks and expects full matrix

> arr[I, J] <- mat[I, J]
Error in libtiledb_query_submit(qryptr) : 
  [TileDB::Writer] Error: Buffer sizes check failed; Invalid number of cells given for attribute 'vals' (6 != 20)

CI Testing should be enabled on Windows

I can't seem to find a way to set or pass the new config settings for creating array to s3 path.
Here is what I am doing

#configure s3
config <- tiledb_config()
cred <- aws.signature::read_credentials()
config["vfs.s3.aws_access_key_id"] <- cred$default$AWS_ACCESS_KEY_ID
config["vfs.s3.aws_secret_access_key"] <- cred$default$AWS_SECRET_ACCESS_KEY
config["vfs.s3.region"] <- "us-west-1"

d1  <- tiledb_dim("d1", domain = c(1L, 5L))
d2  <- tiledb_dim("d2", domain = c(1L, 5L))
dom <- tiledb_domain(c(d1, d2))
val <- tiledb_attr("val", type = "FLOAT64")
sch <- tiledb_array_schema(dom, c(val))

url <- "s3://mike-h5/tiledb-test"
tiledb_array_create(url, sch)
 Error in libtiledb_array_create(uri, schema@ptr) : 
  [TileDB::S3] Error: Failed to create multipart request for object '/tiledb-test/__array_schema.tdb
Exception:  AuthorizationHeaderMalformed
Error message:  Unable to parse ExceptionName: AuthorizationHeaderMalformed Message: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-1' with address : 52.219.112.178 

As shown, my config object seems to be just a copy of tiledb config settings and its value has no effect on the rest operations.