Suggested syntax:

dt0.append(dt1, dt2, ..., force=False)

This modifies datatable dt0 by appending datatables dt1, dt2, ... from below. The force argument controls the behavior in case the datatables have incompatible columns: if force is False then column mismatch will raise an error. If the parameter is True, then columns that are missing in one of the sources will be filled with NAs.

Instead of referencing a column by number, we should just share the pointer between different datatables. We will need to explicitly keep track of reference counts, and then whichever datatable reduces this refcount to 0 is responsible for freeing the memory.

typedef struct Column {
    void   *data;
    MType   mtype;
    SType   stype;
    void   *meta;
    size_t  alloc_size;
    int refcount;
} Column;

This design will also solve the problem when a column that is being referenced in a view is removed from the source datatable.
The MT_VIEW type will be eliminated, and it would not be possible to have a datatable that mixes "data" and "view" columns (which is much saner anyways).

With this change, the field DataTable.source becomes obsolete and should be removed as well. The field DataTable.rowmapping remains, and applies to all columns. It is no longer possible to mix "view" and "data" columns (unless we add a parallel to columns bit-array inside the DataTable).

dt.DataTable([-1])

Any array starting with -1 will cause the freeze. Any other negative numbers seem to be ok

as witnessed by .verify_integrity() function.

Currently even such a simple query as dt(0) would create an evalution graph, generate a C module, JIT-compile that module, and then execute the code to produce the result.
For sufficiently simple queries this adds a lot of unnecessary overhead, instead we should be able to create the result immediately for such simple cases.

(some of the code is duplicate)

import datatable as dt
f = dt.open("~/userdata")
f[lambda f: f.col1 > f.col2, :]

should not select rows where f.col1 is NA

• Appends/rbind (#41, #45, #63, #65, #66, #108, #109, #116, #121, #123)
• Merge/cbind (#132)
• Calculation of mean, median, mode by column (#276)
• Imputation with mean, median, mode
• Column-and-row-based updates, including setting to missing
• Column-based updates, including setting to missing
• Row-based updates, including setting to missing
• Column drop (#38)
• Column rename (#52)
• Creating new frames and adding rows and columns to new frames
• Row and column subsetting (#21)
• Sorts (https://github.com/h2oai/datatable/projects/2)
• Inner and left Joins

This will allow us to reliably check that no OOB access occurs when the same column is shared across multiple datatables. This property should be checked in datatable_check.c. In particular, see #50 for some of the checks that had to be removed when view columns were merged with data columns.

Right now True/False are interpreted as 1/0, and select second/first row or column. This is very counter-intuitive... Perhaps it's better to forbid such selectors for now

dt0[True]
dt0[False]
dt0[True, :]
dt0[False, :]

The alternative would be to interpret them as selecting all/nothing, however usefulness/obviousness of such constructs is questionable (we have other means of selecting all/nothing).

For example f[3:6] selects 0 columns, f[3:7] a single column, f[3:10] -- 4 columns, etc. Seems like start gets subtracted from the end before being interpreted as a slice?

• input / file: file name to read, or input as a string
• sep: field separator
• nrows: maximum number of rows to read
• header: does the first data line contain column names
• na.strings: list of strings to be interpreted as NA values #95
• stringsAsFactors: convert all strings to categoricals -- cannot support until we have categorical type
• verbose: debugging output
• skip: how many rows to skip, or skip to row containing this string #86
• select / drop: Keep some and drop other columns in the file #83
• colClasses: override column types #83
• dec: decimal separator (for numbers)
• col.names: override column names #83
• check.names: check column names are valid and have no duplicates -- always on
• encoding
• quote: dubious, need to rethink. Perhaps this should be quoteRule instead?
• strip.white: strip leading/trailing whitespace from unquoted fields
• fill: allow ragged columns
• blank.lines.skip: ignore blank lines
• showProgress: display progress bar
• nThread: number of threads to use.

The function should return True iff the underlying value is NA.

On Ubuntu, performing any datatable manipulation causes the python process to abort with error message

LLVM ERROR: Program used external function 'pydatatable_assemble_view' which could not be resolved!

The only remedy appears to be to link the external symbols manually on affected platforms (Windows might be having same problems).

This will improve efficiency and allow us to restore warning -Wpadded.
Also it is advised to arrange the members of a struct in the order of their access frequency (per each cache line).

import datatable as dt
z = dt.open("~/userdata")
z(lambda f: f.col3 > 10.55, verbose=True)

Looking at the verbose output, that the actual filter being applied is col3 > 1055 !

Suggested changes are:

• Add single byte 0xFF at the very beginning
  The code should just define dataptr = column->data - 1 and then everything works as-is.
• Add element equal to 1 as the very first element in the offsets region. The length of the offsets region correspondingly becomes nrows + 1. This is needed to avoid unnecessary branching in code at the very beginning of the array (eg: start_offset = i == 0? 1 : abs(offsets[i-1])).

Spec says that string data is stored in the buffer with the following structure:
[String data] [padding] [offsets]
where padding should be filled with 0xFF bytes.

Suggested syntax:

del dt["colA"]
del dt[:, ("colC", "colD", "colZ")]
del dt[cols_to_drop]  # where `cols_to_drop` is any iterable (list, set, generator, etc.)

Although we want to maintain mostly C-type code, we also want to start using templates to avoid unnecessary code duplication (macros are too cumbersome, and do not get check for code coverage).

This would require, at the minimum, the following:

• All malloc functions should be replaced with dtmallocs, which should cast the result to the appropriate pointer type under C++. That is, under C malloc is called as T* ptr = malloc(size);, whereas under C++ it should become T* ptr = (T*)malloc(size);.
• Arithmetic with void* is not allowed in C++ (why???), so should be wrapped in macros that cast pointers into char*. Once implemented, the warning -Wpointer-arith should be restored (currently excluded in setup.py). (#142)

Those places where the parameter is actually unused, it should be marked with __attribute__ ((unused)) using a macro.

Also: line self._stypes = tuple() in dt.py should be removed

The problem is that columns is not allocated in this case (even though it should).

For example, this should work:

import datatable as dt
f = dt.open("~/userdata")
f[lambda f: f.col2 + f.col3]

A real-valued column is displayed as

    col6
--------
 581.654
-405.503
 382.895
 nan    
 195.676
-156.03 
 nan    
  56.936
-899.825

The nans are aligned as if they have implicit decimal dot in the end -- which is not proper. Instead nans should be always right-aligned

The pushBuffer() call happens in-order, however outside of the "ordered" section. As a consequence, it is possible that several threads will execute their pushBuffer code at a different speed and when the time comes to compute location of where to write within the output buffer, they may already write out of order. Although this remains only a theoretical possibility so far, it is still better to fix this behavior.
Suggested fix: each thread has its own output string buffer; before the "ordered" section we insert a "pre-push" callback where each threads may copy over the input into its intermediate buffer, performing any transformations as necessary (for example, unescape double-double-quotes, or backslash-escapes, or decode string into utf8, etc). Then within the "ordered" section we increment offsets of each output buffer pointer. Finally, within the pushBuffer call that happens afterwards we do a mem-copy of the intermediate buffers into the final result buffer.

import datatable as dt
f = dt.open("~/userdata")
f(lambda f: f.col3 > 100000)

produces exception ValueError: max() arg is an empty sequence in line 345 of widget.py

The following should work:

dt(select=lambda f: {"A": f.A, "B": f.B*2, "C": f.B - f.A }

This function should check whether the data in the datatable is consistent, and correct it if possible. In particular, the following checks are needed:

• all the checks already performed by the window() function
• check that boolean columns contain only values 0, 1, and NA
• check that string/categorical columns contain valid UTF-8 strings
• check that string/categorical columns have offsets that are monotonically increasing (in abs value) and do not go out-of-range
• check that string data column in a string section is padded with 0xFFs
• check validity of all rollup values
• (to be continued)

import datatable as dt
f = dt.open("~/userdata")
f["col7":"col2"]

should select columns col7, col6, col5, col4, col3 and col2.

import datatable as dt
f = dt.open("~/userdata")
f[lambda f: f.col1, :]

selects rows where f.col1 is NA.