marklister / product-collections Goto Github PK

Would it be possible to release a scala 2.13.0-M5 compatible jar?

Need to rewrite the GPL code in the CsvParser.

Github doesn't have a forum. So ask your questions by commenting on this issue.

The scaladoc should not contain CollSeq3 through CollSeq22. Likewise CsvParser3 through CsvParser22 are just noise.

Scaladoc can limit inclusion by source file but not by class.

Anyone with ideas please add to this issue.

Is there an existing mechanism to drop a header row when parsing CSV files? If not, what do you think is the most effective option for adding this?

In many csv-s there are several delimiters (like tabs or spaces), but current CSV parsers takes only delimiter.head.

Can we add support for CSV files that use carriage returns (\r) as line separator? Currently the entire file contents are concatenated into the first row.

In addition to the separator, we need also to set the output string representing None in CsvOutput.

Thank you sir!

Vi edited files on Linux end with a \n char. This leads to an java.lang.IllegalArgumentException: 1 at line 2 / java.lang.ArrayIndexOutOfBoundsException

I have added an offending file and thow testcases here:

• https://github.com/zoosky/product-collections/blob/master/jvm/src/test/scala/CSVSpectrum.scala#L219
• https://github.com/zoosky/product-collections/blob/master/shared/src/test/scala/CsvSpec.scala#L134
• https://github.com/zoosky/product-collections/blob/master/res/csvs/vi.csv

Can you give a hint how to fix this? -> Version 1.4.3

I tested with https://github.com/zoosky/ReadCSV

IO - Standard String-> Option[Numeric] converters

I've been trying to run some unit tests on my parsers. To speed things up, I wanted to run all tests in parallel. However, if I do that, my tests become unreliable (i.e. failing inconsistently).

CollSeq((1,2,3),(2,3))

Almost kills the compiler. It seems to go through some sort of recursive type resolution and eventually (correctly) complains:

)(in method apply),T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T4(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T5(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)] <and>
  [T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply), T2(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply), T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply), T4(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)](s: Product4[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T2(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T4(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)]*)org.catch22.collections.immutable.CollSeq4[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T2(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T4(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)] <and>
  [T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply), T2, T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)](s: Product3[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T2,T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)]*)org.catch22.collections.immutable.CollSeq3[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T2,T3(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)] <and>
  [T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply), T2(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)](s: Product2[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T2(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)]*)org.catch22.collections.immutable.CollSeq2[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply),T2(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)] <and>
  [T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)](s: Product1[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)]*)org.catch22.collections.immutable.CollSeq1[T1(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)(in method apply)]
 cannot be applied to ((Int, Int, Int), (Int, Int, Int, Int))
              CollSeq((1,2,3),(2,3,4,5))

scala> Seq(1,2)._1
res36: Seq[Seq[Int]] = List(List(1, 2))

While product-collections as a name is reasonably accurate it does lack a certain 'zing' and not many people know what a 'Product' is.

Accepting suggestions on a new name for this project...

A single type CollSeqN[T,T,T...] where T is a Numeric type could convert to a Saddle Matrix.

Perhaps there is scope for some limited interoperability. Need to investigate if there's any point or if it'd be easier to just use Saddle outright.

There is no provided GeneralConverter for Option[Long], while one can be provided, this seems like an oversight.

Error:(22, 23) could not find implicit value for parameter c2: com.github.marklister.collections.io.GeneralConverter[Option[Long]]
parser.parseFile(filePath, delimiter = ",", hasHeader = true)
^

please 💯

I think this is a simple oversight, since the build.sbt appears to cross-build to 2.10, 2.11 and 2.12.

Recommend providing sibling method to CsvParser.parseFile, (e.g. CsvParser.parse) accepting a scala.io.BufferedSource (or other general input stream) instead of a file name.

How can we use this library without reading the entire CSV into memory at once?

use @implicitNotFound to provide a more helpful message.

See #21.

Test case:

package com.github.marklister.collections.io
import Utils._

object MemTest {

  def main (args:Array[String]):Unit={
    println("Attach monitor, press enter")
    readLine()
    var f = new java.io.FileWriter("/dev/null")
    def st= Stream.continually((1,2))
    st.take(10000000).writeCsv(f)
    f.close
    f=null
    System.gc
  }
}

Results:

Is there a particular reason for 1.4.3 artifacts not to be published for Scala 2.10.6? It seems to be the first release not to be backward compatible.

Sometimes you just want a Seq[Seq[String]] from a csv file. The documentation/examples do not seem to support this core functionality.

#20 and #21 exposed a probable bug in scala relating to tupled on a FunctionN. The workaround is to return Tuples in the Iterators not Products but this exposes some incorrect type assumptions made in product-collections...

In CSV files, quotation of values is optional and some implementations treat it like it. For example, if I export data from MS Excel, only some values are quoted.

It would be nice to deal with it, right now, this parser is unable to import data exported from MS Excel.

Today I needed a flatZipAll method.

As many csv files have headers it would be nice to have a version of colseq with headers support, where I will be able to get all headers and also get any column by header name (not only by number) and any cell by header name and row number.

It'd be nice to be able to map a product collection using CC.tupled. Doesn't work with Products and the bug that I filled against Scala looks like it won't gather any support therefore the path of least resistance is to drop support for ProductN directly.

To workaround a limitation in sbt-boilerplate we have a fake Tuple23 class and flatZip might return an instance off this class.

Ideally, I'd prefer not to have a flatZip method on CollSeq22 because this would cause a compile time error instead of a runtime error. I think a compile time error is always preferable to a runtime error.

The downside is that one would get a "no such method" error instead of a "Arities above 22 are not supported" error. But considering the alternatives I think compile time exceptions are still far preferable to runtime exceptions.

Prerequisite to fix this: some enhancement on sbt-boilerplate (at the moment it looks like a JSR233 solution)

We really wanted an interface which was like

case class Input(a: Int, b: BigDecimal, c: String, d: Option[String])
val p = mkParser[Input]
p.read(stream) //=> Stream[Input]

case class Output(...)
val w = mkWriter[Output]
stream.foreach(w.write)           // stream = Stream(Output(...), ...)
// OR, w.writeStream(stream)

Is there any interface in working to make this library something more like this? Or is there perhaps another library that is already more like this interface?

This might be related to #9....

The classic machine learning "Mushroom" dataset has, wouldn't you know, 23 columns (22 attributes + 1 classification). Sure, one could continually hack things to add "just one more column", but I'm wondering if there was some way of cleverly nesting CollSeqN types and defining appropriately associated implicit conversions, thereby allowing something like¹:

val p = CsvParser[CollSeq2[Integer, Character], CollSeq3[Double, String, Boolean]]

or

val p = CsvParser[Tuple2[Tuple2[Integer, Character], Tuple3[Double, String, Boolean]]

or whatever makes since from a type perspective.

So my question is, could the architecture support this sort of thing, or is the 1:1 association of CSV token counts with type parameter arity tightly bound?

Drop opencsv and use built in parser for scala-js and scala-jvm.

Blockers: parser fairly untested, a bit ugly. messy and character based.

Probably need to add type aliases for CollSeq1-22 to the collections package object. I'll investigate how TupleN works.

Workaround for now: import org.catch22.collections.immutable._

Hi Mark,

Any chance you could start publishing this to a Maven repo? (Or if you already are, document the dependency info.)

Thanks!

See #33 and this question on so