Override/stop remapping if already decided by a former rule.

The algorithm currently working on certain types of data. Definitely, It is yet not rendering the expected output on several well-known data. Add support for those.

As of now supported. (CRAN V0.2.1)

• Example data as provided in vignette and README
• DBIE data (need to test with multiple data)
• • Few Examples as included in the package
• • Multiple data from DBIE #3
• ABS Data
• • Should work on examples given in tidyABS
• • Explore further ABS data
• Examples in unpivotr and related packages (or works)
• • Examples in inst
  • • enron [Part of test]
  • • purpose [More complex version in package]. Need to check for raw file.
  • • Other files.
• • "Worked examples on GitHub." by Duncan Garmonsway
• • smungs examples [examples given in Spreadsheet Munging Strategies by Duncan Garmonsway]
• Examples and datasets cited in jailbreakr
• • EUSES corpus (related paper)
• • Spreadsheets at Enron (Ref 2)
• • Explore OpenRefine #6
• Example medntioned in unheadr
• Example mentioned in hypoparsr
• • Demo mentioned here and in readme
• • Other test files here

Make it available to the user for other types of data.frames

Got it archived. Devlopment cycle need to fix urgently.

Add this to addinslist

Due to personal reasons (completely unavoidable) I'm not working on it for time being.

Hopefully, I'll be able to smile again and will gather enough strength mentally to start working on it again.

Absolutely sorry for such situation.

To handle missing data.

This is for

This may be implemented later if required.

Thanks to this SO

Following code works well :

library(rJava)
.jinit()
toolkit <- J("java.awt.Toolkit")
default_toolkit <- .jrcall(toolkit, "getDefaultToolkit")
dim <- .jrcall(default_toolkit, "getScreenSize")
height <- .jcall(dim, "D", "getHeight")
width <- .jcall(dim, "D", "getWidth")

Manually done data_blocks and not considered in analyze_cells.

Adopt to the method described /tested in bedantaguru/covrChecks#1

This should reduce Gap in two coverages

The commit 3ddcdf6 was a quick local tested attempt. The same need to be included properly.

Implement in Nightly.

This should fix/patch for

• rstudio/shinytest#157
• r-lib/covr#277
• rstudio/shinytest#270

Tracking

• Implemented in Nightly
• Add documentation for general-purpose shinytest and covr

These are the planned activity for 0.3.0
These are in progress in nightly
Few of these may be dropped (as all of these are experimental)

• Complete Heuristic maturation phase 1 : complete #5
• Introduce grammar
• %<^% kind of symbols for NNE etc.
• Add shinytest prototype as described in #13
• Compatibility of cell traceback with external operations (like {unpivotr} etc)
• Multi routes tests
• Optional packages availability based test
• Mark cell-analysis for remodelling into a state (maybe in later version)
• {DT} based table plots - check #16
• reveal.js – Presentation for various topics
• {pkgdown} tweaks [like all tables to DT etc]
• Explore Github Actions as guided in this. Also check #18
• Fix all issues tagged as bug like ( #20 )

• As of 0.2.x, I'm doing this Adhoc basis.
• I need to do in for 0.3.x

All dependency checks should route through is_available and a functionalities database to be maintained for checking whether a specific functionality is working.

Please be more specific than "diverse sources" when describing the package. Suggest "diverse sources such as Excel, Word, XML and HTML" rather than just "diverse sources" or add the others if there are others. I couldn't initially figure out if it applied to my problem without reading more than I thought was reasonable.

I think few of them can be transferred to base without changing much of the codebase

Like

• select
• rename

Need to adopt {Rcpp} whenever possible.

• in LCS
• in as_cell_df
• in direction attachment (enhead alternative)

Numerous attempt made earlier by many other developers. Few projects are really groundbreaking. These may be not in the domain of #rstats but worth connecting with these projects and check where tidycells stands.

Check out these projects

• OpenRefine
• Wikidata

Following is the prototype

nev <- new.env()

get_new_var <- function(){

  attach(nev, pos = 2L, name = "pkg:internals")
}


nev$`%+%` <- function(x, y){
  x*y
}

Note that the above needs to be tuned for the fact that it does not detach on the unload of the package.

Try to be independent of {unpivotr}

Already added a prototype/working version in cae10c0

This is to test read functions capabilities for various file types.

A broader framework can be adopted here to make almost all tests available to the user through appropriate channels.

The idea is that a test will be conditional when there is a heavy dependency on the system or third-party library. But user should have an option to run these tests for debugging/knowing machine issues. A guide also can be made for raising the issue in Github for the same.

This can be implemented by adopting tests that were already written.

As of now, a separate function is written for a test.

make sure primitive types and S3 inheritance is maintained correctly.

• This is done loosely in 0.2.x but need
• to do more robustly in 0.3.x

Explore Sources Mentioned in

• Ref 1
• Ref 2
• Ref 3

Plan a light (poc) implementation in v 0.3.0

This is for the smooth functioning of analyze_cells

• for ggplot2 add optional grid argument
• for DT add option to have numeric column (instead of A, B, C --> 1, 2, 3)

Both DT and ggplot2 has to be tuned and shiny modules have to be resynced

Originally posted by @bedantaguru in #28 (comment)

Modify read_cells to optionally opt for "heuristic detection" and "structural assignment". The proposed function read_it (should be a wrapper for read_cells with appropriate arguments) can do the following things

• Read data as it is. Required for already structured data.
• Try to find out whether reading as it is is ok or not.
• Try to combine similar tables in a document based on headers.
• Try to alert the user about what else to be done in order to read the data from the selected file. If the file type is not supported or any additional package is required to inspect.
• Detect column type after (on-demand basis) (if skipped in read_it)
• Should support standard R objects too.

Do for

• string : the current version
• numeric
• dates and times

KFL
See
#6
#30

Here is the prototype

require(magrittr)
#> Loading required package: magrittr

select.df <- function(.x, ...) {
  pos <- tidyselect::eval_select(rlang::expr(c(...)), .x)
  rlang::set_names(.x[pos], names(pos))
}
rename.df <- function(.x, ...) {
  pos <- tidyselect::eval_rename(rlang::expr(c(...)), .x)
  names(.x)[pos] <- names(pos)
  .x
}


col_contains <- function(str, dat = tidyselect::peek_data(fn = "col_contains"), .ignore_case = T) {
  if(.ignore_case){
    str <- tolower(str)
    dat %>% 
      purrr::map_lgl(~.x %>% tolower() %>% stringr::str_detect(str) %>% any) %>% 
      colnames(dat)[.]
  }else{
    dat %>% 
      purrr::map_lgl(~stringr::str_detect(.x, str) %>% any) %>% 
      colnames(dat)[.]
  }
  
}


d <- iris
class(d)<- c( "df", class(d))


d %>% dplyr::rename(tst = col_contains("seto")) %>% head()
#>   Sepal.Length Sepal.Width Petal.Length Petal.Width    tst
#> 1          5.1         3.5          1.4         0.2 setosa
#> 2          4.9         3.0          1.4         0.2 setosa
#> 3          4.7         3.2          1.3         0.2 setosa
#> 4          4.6         3.1          1.5         0.2 setosa
#> 5          5.0         3.6          1.4         0.2 setosa
#> 6          5.4         3.9          1.7         0.4 setosa
d %>% dplyr::select(col_contains("ginica")) %>% head()
#>   Species
#> 1  setosa
#> 2  setosa
#> 3  setosa
#> 4  setosa
#> 5  setosa
#> 6  setosa

^{Created on 2020-04-03 by the reprex package (v0.3.0)}

This is a trackable version of

• Test it in r-hub
• Test for optional shiny modules (series of visual_* functions)
• Write more tests (increase coverage)
• Write collate_columns function to deal with similar columns in composed data.frame
• Making a pkgdown site
• Releasing this package to CRAN
• Make doc test skip on CRAN.
• Make possibility for purrr like formula, e.g. ~ .x for tidycells::value_attribute_classify
• A compatibility function for the "Heuristic Maturation" process (after CRAN)
• Write blog + add it to R blogger and other sites
• Send it to the r-packages mailing list
• Explore options to add this in CRAN Task Views (not possible now)
• make a cheatsheet
• Explore SDMX Converter possibility
• Explore other formats (containing unorganised tables) possibility. Check out unoconv.
• Write more vignettes on other topics
• Making cell analysis little faster

cell_df additional columns dropping in numeric_values_classifier
Similarly while coming from R-object to cell_df columns are getting dropped.
Need to fix this.

At least for Proof of Concept add a simple assistant

• This should work automatically in Rstudio only.
• It should work when the package is loaded.
• Basically it is an overview in the viewer

Create a broader framework if found to be useful.

Check #26
Remove {Rcpp} dependency from tidycells_nightly

Ref #26 (comment)

Minor changes should be done in Logo. Make sure to follow a standard structure.

• Put the logo in man folder (check corrr). It should come in an interactive package help page help("tidycells").
• Try to make is SVG (as the clarity is not great now) and test in various system.

Use DT whenever possible.
May remove {plotly} dependency if this is working fine.
This is one sub-item in #14

Explore several options to enhance documentation

R Under development (unstable) (2019-12-14 r77569)
r-devel-linux-x86_64-debian-gcc

Error occurs at

and

Either skip these on CRAN (noting a mark here) or fix in a whole lot before December 2019.

Need to find the exact reason.

Once read_cells supports reading data as it is (see #7), extend it's the support for other types of files and objects.

• If possible or required, depend on wand. For magic number based file type detection.
• Support for all types mentioned in Data Import Cheat Sheet - by RStudio
• xls : Already supported through readxl and xlsx
• xlsx : Already supported through tidyxl
• doc : Already supported through docxtractr but need a better alternative.
• docx : Already supported through docxtractr
• pdf : Already supported through tabulizer but need either better alternative or more clear guide to the user.
• html, xml : Already supported through XML possibly need more clear guide and examples
• All kind of delimited files : Few supported through read.csv and readr. Need to support other possiblities.
• All haven types : Check all possible types under haven
  • SPSS
  • Stata
  • SAS

The flow of the structure

1. File Field (folder or zip files)
2. Table Field Container (File, like excel with multiple sheets)
3. Table Field (cell-df)
4. Table

Now below a table

4. information block (containing major attributes and data blocks)

Which contains

5. data-blocks (or value blocks) as used in analyze_cells
6. major attribute-blocks

So the difference between a table and an information block is that :

• The table is a superset of an information block
• The table is containing information block + minor attributes

So the minor attribute gets attached to each data-blocks in the information-block.

Default was
take_col_names = F which was made take_col_names = T
But it needs to be as prior I guess i.e. take_col_names = F

Check out dev-notes

The build is failing for system dependency.

• need to skip the test in CRAN. (to be marked for next release)
• try to test it in Travis
• need to manually inspect on Fedora with LibreOffice for possible issues.

Enable following CIs

• Circle CI [Ref : 1 2 3 4]
• Github Actions

Check multiple data from DBIE

For each composed columns (base case use LCS)

• infer from data
• use LCS
• use wikidata (see #6) : This is #31