The csv from xinyi2016

Propose a solution to customize quoting behaviour

Problem

I found 6 similar issues with parsing quoted fields in Go’s built-in CSV package encoding/csv. In this proposal, I show a solution that allows encoding/csv to handle all of those cases.

Firstly, CSV inputs in the above-mentioned issues do not conform to RFC 4180 format that encoding/csv follows. Some of them seem to be bad data.

For example, Issue #56329 expects "a"b"c",123, to be parsed as [a"b"c,123,]. This allows single quotes within quoted fields. If the quoted b was a comma that we try to escape, we see the problem:

“a”,”c”,123,

The input can be interpreted as

[a, c, 123]

[a”,”c, 123]

Solution

In this proposal, I want to introduce a function parameter that allows users to modify the malformed CSV input and utilize what has been built at the same time.

The modified parser looks like this:

csv/reader.go

Lines 295 to 323 in f0ad461

 type Format func(o []byte) (n []byte) 

 func (r *Reader) readRecord(dst []string, fmts ...Format) ([]string, error) { 

 if r.Comma == r.Comment || !validDelim(r.Comma) || (r.Comment != 0 && !validDelim(r.Comment)) { 

 return nil, errInvalidDelim 

 } 

 // Read line (automatically skipping past empty lines and any comments). 

 var line []byte 

 var errRead error 

 for errRead == nil { 

 line, errRead = r.readLine() 

 if r.Comment != 0 && nextRune(line) == r.Comment { 

 line = nil 

 continue // Skip comment lines 

 } 

 if errRead == nil && len(line) == lengthNL(line) { 

 line = nil 

 continue // Skip empty lines 

 } 

 break 

 } 

 if errRead == io.EOF { 

 return nil, errRead 

 } 

 for _, applyFmt := range fmts { 

 line = applyFmt(line) 

 }

This design allows encoding/csv to handle malformed CSV inputs. It introduces a possibility for encoding/csv to support other CSV formats.

xinyi2016 / csv Goto Github PK

csv's People

Contributors

Stargazers

Watchers

csv's Issues

Propose a solution to customize quoting behaviour

Problem

Solution

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	type Format func(o []byte) (n []byte)

	func (r *Reader) readRecord(dst []string, fmts ...Format) ([]string, error) {
	if r.Comma == r.Comment \|\| !validDelim(r.Comma) \|\| (r.Comment != 0 && !validDelim(r.Comment)) {
	return nil, errInvalidDelim
	}

	// Read line (automatically skipping past empty lines and any comments).
	var line []byte
	var errRead error
	for errRead == nil {
	line, errRead = r.readLine()
	if r.Comment != 0 && nextRune(line) == r.Comment {
	line = nil
	continue // Skip comment lines
	}
	if errRead == nil && len(line) == lengthNL(line) {
	line = nil
	continue // Skip empty lines
	}
	break
	}
	if errRead == io.EOF {
	return nil, errRead
	}

	for _, applyFmt := range fmts {
	line = applyFmt(line)
	}