Interest in support for automatic column mapping in RowOriented reader? about parquetsharp HOT 4 CLOSED

Hi @bmacphee,

You are correct. The row oriented API makes too many trade-offs between ease of use, flexibility and performance. It's there to make it easy for new starters to read a Parquet file, but not heavily encouraged.

Nonethless, the feature you might be looking for is the addition of a C# field attribute to explictly state that a tuple field maps to a particular column name. For example:

struct MyTuple
{
    [MapToColumnName("Index")]
    int Item1;

    [MapToColumnName("Value")]
    float Item2;
}

Orthogonally one can imagine a ColumnOrder(int) attribute to reflect the current implicit column ordering and make it explicit. Things get messy when you start mixing things together (i.e. how much flexibility do you allow?).

Perhaps as a first start, if MapToColumnName is used, then all columns need to be marked with this attribute (and throw an exception if not). I'd suggest looking at GetFieldsAndProperties() in RowOriented/ParquetFile.cs and working up from there. Afterall, the introduction of such a field attribute would only change the ordering of the resulting array.

Edit: I cannot decide whether it should be called ColumnName or MapToColumnName. The former describe an explicit column name, while the latter hints at a different ordering when reading.

from parquetsharp.

I've also noticed that GetFieldsAndProperties() has little ordering guarantees if you start mixing fields and properties, as they are listed by type (fields first, properties second).

from parquetsharp.

I did consider using attributes as a way to expose this to potential users without having to have a mode toggle argument or something. My initial proof of concept just did it by relying on what's returned from GetFieldsAndProperties being exact matches (case sensitive) for the parquet file, which was totally fine for my purposes.

I agree it would make sense to enforce the all-or-nothing approach with the column names, even if the attributes just end up echoing the field names. It's still convenient enough as you only need to write the data class once.

I don't have a diff to show right this second but what I did was add this to the ParquetRowReader construction:

	// TODO: validate types?
	// TODO: conditions for handling bare tuples for TTuple type?
	Dictionary<string, int> fileColumnIndexes = new Dictionary<string, int>();
	for (int i = 0; i < _parquetFileReader.FileMetaData.Schema.NumColumns; ++i)
	{
		fileColumnIndexes[_parquetFileReader.FileMetaData.Schema.Column(i).Name] = i;
	}
	for (int i = 0; i < fields.Length; ++i)
	{
		_objectToFileColumnMapping[i] = fileColumnIndexes[fields[i].name];
	}

and then ReadColumn just looks up the "correct" column when asked by changing the index.

I appreciate the input.

from parquetsharp.

Merging into master as part of PR #108.

from parquetsharp.