I'm trying to do Mat[T] dot Vec[Int] with a result of Mat[T] where T is a domain wrapper around an Int ( case class T(value: Int) ), but I fail to understand which type class my T must implement to support this operation.

Can you add this to the existing documentation?

I am trying to write a function that mimics R's diff, which returns the difference between each two elements in a vector. The simplest implementation would be to merge a series with it's lag-1, and subtract across rows. I came across what appears to be a bug as follows:

import org.saddle._
val s1 = org.saddle.Series(Vec(1,2,3), Index("a", "b", "c"))
val mergeShift = s1.join(s1.shift(1))
mergeShift.row("b") // Should be [2 1], but returns [2 2]

mergeShift appears as expected, but the row extraction acts as though the shift never happened. Am I doing something wrong?

Thanks!

I am doing an iterative algorithm where I must do an element-wise multiplication of a matrix an update matrix. But because the multiplication it's element-wise, I don't actually need to construct the update matrix as a whole; I can just compute single elements as needed. This gives much lower memory use, and correspondingly better locality.

If there was som some way to get the element index in a map so that I could do something like:

val m2 = m map { case (row,col,val) = val * computeElement(row,col) }

That would solve this problem, though I would still end up needing to store two matrices in memory, m and m2, when really I only need one. What I'd really like is:

m transform { case (row,col,val) = val * computeElement(row,col) }

I realize there are all sorts of issues both philosophical and practical with mutable objects, but they sure can be handy when your data is so big you can only have one copy in core.

Sonatype repository contain empty jars for version 1.2.0 and 1.3.0
Size: 267 Bytes
There are only manifest inside

With a file like this:

col1,col2,col3
1,2,3
4,5,6

This code ...

val file = CsvFile("myfile.csv")
val data = CsvParser.parse(params=CsvParams(hasHeader=true))(file)

produces a frame with three rows instead of 2 with the correct column names:

data  : org.saddle.Frame[Int,Int,String] = [3 x 3]
        |         0    1    2 
        |      ---- ---- ---- 
        | 0 -> col1 col2 col3 
        | 1 ->    1    2    3 
        | 2 ->    4    5    6 

Unless I missed something, it looks like the last commit was 9 months ago. Is this project still being actively worked on?

Here's a repl session:

scala> val strVec = Vec("string", "another string", "unrelated")
Nothing <: String?
true
String <: Product?
false
strVec: org.saddle.Vec[String] = 
[3 x 1]
        string
another string
     unrelated


scala> val intVec = vec.randi(3)
intVec: org.saddle.Vec[Int] = 
[3 x 1]
-2144970151
-1801346899
-1565693780

scala> val df = Panel(strVec, intVec)
df: org.saddle.Frame[Int,Int,Any] = 
[3 x 2]
                  0           1 
     -------------- ----------- 
0 ->         string -2144970151 
1 -> another string -1801346899 
2 ->      unrelated -1565693780 


scala> val df2 = df.rfilter(x => x.get(0).map(_.toString).getOrElse("").contains("string"))
Nothing <: String?
true
String <: Product?
false
df2: org.saddle.Frame[Int,Int,Any] = 
[2 x 2]
                  0           1 
     -------------- ----------- 
0 ->         string -2144970151 
1 -> another string -1801346899 

scala> :t df
org.saddle.Frame[Int,Int,Any]

scala> :t df2
org.saddle.Frame[Int,Int,Any]

scala> df.colType[Int]
res4: org.saddle.Frame[Int,Int,Int] = 
[3 x 1]
               1 
     ----------- 
0 -> -2144970151 
1 -> -1801346899 
2 -> -1565693780 


scala> df2.colType[Int]
res5: org.saddle.Frame[Int,Int,Int] = Empty Frame

scala> df.colType[String]
Nothing <: String?
true
String <: Product?
false
res8: org.saddle.Frame[Int,Int,String] = 
[3 x 1]
                  0 
     -------------- 
0 ->         string 
1 -> another string 
2 ->      unrelated 

scala> df2.colType[String]
Nothing <: String?
true
String <: Product?
false
res6: org.saddle.Frame[Int,Int,String] = Empty Frame

When running the H5Store Spec test I get the following set of errors at the very end of the test:

[error] ! H5Store is thread-safe
[error] FileSystemException: E:\TEMP\tmp5023cb40.h5: The process cannot access the file because it is being used by another process.
error
[error] sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
[error] sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
[error] sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
[error] sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:268)
[error] sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
[error] org.saddle.io.H5StoreSpec$$anonfun$3$$anonfun$apply$15.apply$mcZ$sp(H5StoreSpec.scala:443)
[error] org.saddle.io.H5StoreSpec$$anonfun$3$$anonfun$apply$15.apply(H5StoreSpec.scala:389)
[error] org.saddle.io.H5StoreSpec$$anonfun$3$$anonfun$apply$15.apply(H5StoreSpec.scala:389)
[error] org.saddle.io.H5StoreSpec$$anonfun$hdfTest$2.apply(H5StoreSpec.scala:68)
[error] org.saddle.io.H5StoreSpec$$anonfun$hdfTest$2.apply(H5StoreSpec.scala:67)

It looks like the code is trying to delete a shared test file before the thread pool shutdown is complete.

The problem ‘goes away’ if you insert the following line of code in between the shutdown call and the delete call,
~ lines 439 – 441 in H5StoreSpec.scala:

while( !pool.isTerminated() ) Thread.sleep( 100 ) ;

Is it possible to have interop with the above technologies?

Hi i am Trying to write the following frame, It it able to write first two rows and getting null pointer exception

NSCDL0 -> NSCDL0 LP ABHGMIMND01 010108.17B D4CBDBMJRA NSCDL0
NSCDL0 -> NSCDL0 LP ABHGMIMND01 010108.17B D4CBDBMJRA NSCDL0
SDEDL0 -> SDEDL0 LPP NA NA NA NA
SDSEFR -> SDSEFR KP NA NA NA NA

ERROR:

Exception in thread "main" java.lang.NullPointerException
at org.saddle.io.CsvImplicits$$anon$2$$anonfun$writeRows$1$1$$anonfun$5.apply(CsvImplicits.scala:155)
at org.saddle.io.CsvImplicits$$anon$2$$anonfun$writeRows$1$1$$anonfun$5.apply(CsvImplicits.scala:155)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.saddle.io.CsvImplicits$$anon$2$$anonfun$writeRows$1$1.apply(CsvImplicits.scala:155)
at org.saddle.io.CsvImplicits$$anon$2$$anonfun$writeRows$1$1.apply(CsvImplicits.scala:150)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
at org.saddle.io.CsvImplicits$$anon$2.writeRows$1(CsvImplicits.scala:150)
at org.saddle.io.CsvImplicits$$anon$2.writeCsvStream(CsvImplicits.scala:169)
at org.saddle.io.CsvImplicits$$anon$2.writeCsvFile(CsvImplicits.scala:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.att.bdcoe.capop.apps.OpprPush$.processApp(OpprPush.scala:31)
at com.att.bdcoe.capop.init.Navigate$.main(Navigate.scala:15)
at com.att.bdcoe.capop.init.Navigate.main(Navigate.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Initially the module would contain only H5Store. In my opinion this is justified to reduce the transitive dependency on a library not part of the core functionality. If that's agreeable the only issue then is the implicit object in CsvImplicits that sports an explicit "writeHdfFile" method. I don't see this used anywhere - can it be purged or does a more elegant approach need to be taken?

When I try to run the following expression in the REPL, I get the stack trace below.

Series(("a",Double.NaN))

Stacktrace:

java.util.FormatFlagsConversionMismatchException: Conversion = s, Flags = 0
  at java.util.Formatter$FormatSpecifier.failMismatch(Formatter.java:4041)
  at java.util.Formatter$FormatSpecifier.checkBadFlags(Formatter.java:2949)
  at java.util.Formatter$FormatSpecifier.checkGeneral(Formatter.java:2907)
  at java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2680)
  at java.util.Formatter.parse(Formatter.java:2528)
  at java.util.Formatter.format(Formatter.java:2469)
  at java.util.Formatter.format(Formatter.java:2423)
  at java.lang.String.format(String.java:2797)
  at scala.collection.immutable.StringLike$class.format(StringLike.scala:270)
  at scala.collection.immutable.StringOps.format(StringOps.scala:31)
  at org.saddle.Series.createVal$1(Series.scala:879)
  at org.saddle.Series$$anonfun$stringify$2.apply(Series.scala:882)
  at org.saddle.Series$$anonfun$stringify$2.apply(Series.scala:882)
  at org.saddle.util.package$.buildStr(package.scala:37)
  at org.saddle.Series.stringify(Series.scala:881)
  at org.saddle.Series.toString(Series.scala:904)

They mention data.toFrame and data.toFrameNoHeader. Now parse returns a Frame directly so this calls are not needed.

I see the point of employing fastutil.
I would possibly had taken the same design decision if I had started Saddle.

But there's another aspect to consider: 4 decades of a lot of robust code as part of BLAS/LAPACK.
Since users of Saddle are possibly interested on statistical models, etc... seems like it makes sense to integrate Saddle and Breeze somehow. At the moment, applications would have to copy data from one library to another.

What would be the pros and cons of replacing Saddle data structures by Breeze data structures?
Any idea of what would be the effort needed?

Thanks a lot :-)

Before addressing #22 the project should be converted to multi-module, in the interest of preventing merge pains caused by intermittent code changes.

Very useful when memory is at a premium (big matrices.)

This may be tricky if the underlying libraries don't support it.

Ideally, would also allow mixing of types, e.g. Mat[Double] * Vec[Float], without intermediate copies.

I know I can use rconcat with a frame, but could we provide simpler solution?

Hi,

Is it possible to provide interop and abstraction on top of other lines algebra libraries like:
Breeze
AlgeBird
TypeLevel projects

Also provide the ability to produce computable documents like IPython Notebook or Scala Notebook and iterop with them.

Would it be possible to construct a CsvFile from a File object? It is convenient to have File handle all the path separator magic rather than trusting to raw string mangling.

In the Quick Start Guide, it says

You’ll notice that if an index is not provided, a default int index is set where the index ranges between 0 and the length of the data.

which implies that you can create a Frame using

Frame(Seq(Vec(1,2), Vec(2,3)))

but there is no method

def apply[T: ST](values: Seq[Vec[T]])

only

def apply[T: ST](values: Vec[T]*)

(source)

hi thanks for this module!

i have a bug when using parsePar on the following csv:

http://data.dft.gov.uk/road-accidents-safety-data/DfTRoadSafety_Accidents_2009.zip

with separator "," (whatever the column)

the bug disappears with parse

If there was a way to instruct Saddle treat missing values as zeroes, then we could use Frames and Series to do sparse matrix and vector operations.

http://saddle.github.io/doc/quickstart.html#vec

scala> vec.randn2(2, 15) // 100 obs normally distributed with mean 2 and stdev 15

should be

scala> vec.randn2(100, 2, 15) // 100 obs normally distributed with mean 2 and stdev 15

scala> vec.randn2(2, 15)
:24: error: not enough arguments for method randn2: (sz: Int, mu: Double, sigma: Double)org.saddle.Vec[Double].
Unspecified value parameter sigma.
vec.randn2(2, 15)

scala> vec.randn2(100, 2, 15)
res18: org.saddle.Vec[Double] =
[100 x 1]
16.1250
19.2511
7.7113
3.5501
0.6810
...
-11.2612
10.4147
-24.9516
-3.0263
-0.8401

Consider the following code which prints the rolling sum of a vector.

Vec(-1,1,3,10,300).rollingSum(3)

If I understand the meaning of rollingSum correctly, it should create a sliding window of size 3, and sum up all the numbers within the window. Hence, the correct result of the statement about is

Vec(3, 14, 313)

However, when I run the code, I get the following weird result:

Vec(3, 12, 209)

After some digging, it appears to me that the rolling window always take the first element into account. For example, the actual value returned from rollingSum for the fourth element is -1 + 3 + 10, instead of 1 + 3 + 10.

Has anyone experience similar issue with the rolling* functions?

Thanks.

Interoperability with pandas may still be an "aspirational" requirement but would be critical for my use case. Generating a large Frame[DateTime,String,Double] of price data with 1644 securities over 3607 trading days from Yahoo finance (a simple use case), the resulting h5 file works correctly in and out of saddle but in

vitables reports a division by zero error in long_scalars nrowsinbuf = buffersize // rowsize

and in pandas (latest versions) shows:

ValueError: Shape of passed values is (1644, 0), indices imply (1644, 3607)

Requires Mat.takeRows(Array(0)) instead, contrary to docs

scala> val m = mat.rand(3,3)
m: org.saddle.Mat[Double] = 
[3 x 3]
-0.1587 -0.3825 -0.7195 
 0.8251  0.0543  0.9566 
 0.5578  0.1654 -0.4489 


scala> m.takeRows(0)
<console>:19: error: type mismatch;
 found   : Int(0)
 required: Array[Int]
              m.takeRows(0)
                         ^

scala> m.takeRows(Array(0))
res83: org.saddle.Mat[Double] = 
[1 x 3]
-0.1587 -0.3825 -0.7195 

The instructions to start a sbt project (http://saddle.github.com/doc/index.html#download-and-install) suggests that you use

g8 saddle/saddle.g8

but I had some trouble using this giter8 template: I had to update some of the library versions (starting with "org.scala-saddle" %% "saddle" % "1.0.1", because the template asks for a snapshot version), because sbt wasn't finding some libraries. Maybe the template should be updated...

val v = Vec(1,2,3,4,5,6,7,8,9)
val m = Mat(2,2,v)

Expected:
m.toVec == Vec(1,2,3,4)

Actual:
m.toVec == v

Hi,

This PR closed in August contains some updates we need for our deployment.

#63

Its content is currently only available in 1.3.4-SNAPSHOT. Could you please release 1.3.4 so that we can get the official jar from the official repo?

Thanks!

S

basically i can achieve with the naïve approach like
someFrame.mapVec{case v=>v.fillNA(x=>0.0)}

I find it a quite common use case so it will be great if we can just write
someFrame.fillNA(x=>0.0)

please let me know or I can submit a PR if above make sense for performance

• thanks for the nice library

The filter API is working great when I filter by a Series element-wise, but I am having problems selecting only the elements which are unique. That is, I would like to drop any elements of the series that are repeated. This requires filtering by groups. I am having trouble discovering the syntax that accomplishes this. For example:

val s  = Series('a'->1, 'b'->2, 'b'->3)
s(s.groupBy.combine(_.count).filter(_==1).index)

gives the error

<console>:16: error: type mismatch;
 found   : org.saddle.Index[Char]
 required: org.saddle.Index[Any]
Note: Char <: Any, but trait Index is invariant in type T.
You may wish to define T as +T instead. (SLS 4.5)
              s(s.groupBy.combine(_.count).filter(_==1).index)

which I understand (well enough), but what is the workaround? It seems like maybe a filter method for SeriesGrouper would do the trick?

Thanks!

Shouldn't CSV writing and reading produce idempotent results?

scala> f
res12: org.saddle.Frame[Int,Int,Double] =
[2 x 2]
           0      1
     ------- ------
0 -> -0.4969 0.0686
1 -> -0.3326 0.7883

scala> f.writeCsvFile("test.csv")

scala> val file = CsvFile("test.csv")
file: org.saddle.io.CsvFile = CsvFile(test.csv, encoding: UTF-8)

scala> val df = CsvParser.parse(file)
df: org.saddle.Frame[Int,Int,String] =
[3 x 3]
      0                    1                   2
     -- -------------------- -------------------
0 ->                       0                   1
1 ->  0  -0.4969191360767252 0.06860397763436138
2 ->  1 -0.33264686017541434  0.7882551460465188

Vec, Mat, etc.

Hi,
If this is not the place to raise this, please let me know where I can..

sbt/saddle newbie here. I tried to run sbt on the saddle project and got the following error. sbt works fine with other projects so was wondering what I was doing wrong with saddle.

Sbt version: 0.13
Scala Version: 2.9.3
Ubuntu 13.04

sbt
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
..

:: UNRESOLVED DEPENDENCIES
::�::::::::::::::::::::::::::::::::::::::::::::::�::
com.github.gseitz#sbt-release;0.7: not found�
:: com.github.mpeltonen#sbt-idea;1.1.0: not found�
�:: com.orrsella#sbt-sublime;1.0.5: not found�
:::::::::::::::::::::::::::::::::::::::::::::�

Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
com.github.gseitz:sbt-release:0.7 (sbtVersion=0.13, scalaVersion=2.10)�
com.github.mpeltonen:sbt-idea:1.1.0 (sbtVersion=0.13, scalaVersion=2.10)�
com.orrsella:sbt-sublime:1.0.5 (sbtVersion=0.13, scalaVersion=2.10)�
sbt.ResolveException: unresolved dependency:
com.github.gseitz#sbt-release;0.7: not found
unresolved dependency: com.github.mpeltonen#sbt-idea;1.1.0: not found
unresolved dependency: com.orrsella#sbt-sublime;1.0.5: not found

Regards,
Rajiv

This throws ArrayIndexOutOfBoundsException while parsing csv:

1,2,3
1,2,

This works:
1,2,3
1,,3

Is it possible to extend the current functionality to provide multi dimensional spreadsheet capabilities like in the following:

Quantrix
DADiSP (http://www.dadisp.com/)
Morphit (http://www.edge-ka.com/products/morphit)
ZigZag (http://www.xanadu.com/zigzag/, http://www.nongnu.org/gzz/gi/gi.html, http://www.dgp.utoronto.ca/~mjmcguff/research/zigzag-and-mSpace/zigzag-and-mSpace.pdf, GZigZag - http://gzigzag.sourceforge.net/nutshell.html)
Javelin Software (http://www.thecomputerarchive.com/archive/Software/Applications/Javelin%20Plus%20(1987).pdf, http://www.lynnbob.com/bob/articles/JavelinReview.htm)
Lotus Improv
Spreadsheet 2000
numberGo
Resolver One
Matrex (http://sourceforge.net/blog/matrex-blends-matrix-math-with-spreadsheets-strengths/, http://matrex.sourceforge.net/guideothers.shtml)
SciDAV (http://scidavis.sourceforge.net/)

f1 = Frame(..).sortedRowsBy...
f2 = Frame()

f1.rconcat(f2)

Concatenation is applied to unsorted version of f1, is it expected behaviour?

Neither the scaladoc nor the quick start guide explain how to update a row in a Mat or Frame. I tried the most obvious to me thing (below), but it didn't work.

I'd be happy to update the documentation if I could figure out how to do it =)

scala> val f: Frame[Int, Int, Double] = mat.zeros(3,2)
f: org.saddle.Frame[Int,Int,Double] = 
[3 x 2]
          0      1 
     ------ ------ 
0 -> 0.0000 0.0000 
1 -> 0.0000 0.0000 
2 -> 0.0000 0.0000 


scala> f.row(0)  = Vec(2.0,3)
<console>:12: error: ambiguous reference to overloaded definition,
both method row in class Frame of type (keys: Array[Int])org.saddle.Frame[Int,Int,Double]
and  method row in class Frame of type (slice: org.saddle.index.Slice[Int])org.saddle.Frame[Int,Int,Double]
match expected type ?
              f.row(0)  = Vec(2.0,3)
                ^

scala> f.rowAt(0) = Vec(2.0, 3)
<console>:12: error: ambiguous reference to overloaded definition,
both method rowAt in class Frame of type (loc: Int)org.saddle.Series[Int,Double]
and  method rowAt in class Frame of type (locs: Array[Int])org.saddle.Frame[Int,Int,Double]
match expected type ?
              f.rowAt(0) = Vec(2.0, 3)
                ^

Hi,

This is more of a feature request, that an actual issue. Is it possible to join two frames on a common column's values (say akin to the sql inner join between two tables)?. Right now the join is performed on the indexes, I'd like to do it on a common column, like the pandas "on" parameter to the DataFrame.join function. Any plans for a merge operation (like the pandas.DataFrame.merge one)? Thanks!

Cheers,
Cosmin

The following code is my attempt to multiply a Frame by a Series, which the docs suggest is possible.

val m = mat.rand(6,4)
val v = vec.rand(4)

m dot v  // works

val i = Index(-1,2,10,100)
val s = Series(v,i)
val f = Frame(m)
f.setColIndex(i)

f dot s // fails

val fv = Frame(v)
f dot fv // fails similarly

The error I get is:

<console>:23: error: No org.saddle.ops.InnerProd instance available to operate on values of type org.saddle.Frame[Int,Int,Double] and org.saddle.Series[Int,Double]
              f dot s
                ^

And similarly for frame dot frame. Am I doing something wrong here? Or simply not implemented yet?