how to work around "next on empty iterator"in chapter 9th? about aas HOT 11 CLOSED

I think you are not successfully reading the input. This indicates you don't end up producing trials to analyze. If you've modified the code, I think it's somewhere in your modification.

from aas.

but I only modify 4point:
1.class RunRisk(private val spark: SparkSession) extends java.io.Serializable
2.modify path
3.modify val formatter = DateTimeFormatter.ofPattern("d-MMM-yy",Locale.ENGLISH)
4.
import java.util.Locale
import org.apache.log4j.Logger
import org.apache.log4j.Level

@srowen
could you help me?thanks ~~~

um....my environment is:
ubuntu 18.10
spark 2.3.1
JDK 1.8.0_201
scala 2.11.8

from aas.

if I did NOT modify the above former three point ,there will be other error logs....

from aas.

I use trials.show(),the content is:
+-----+
|value|
+-----+
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
| NaN|
+-----+

from aas.

I found

for (i <- 0 until numTrials) {
  val trialFactorReturns = multivariateNormal.sample()
  val trialFeatures = featurize(trialFactorReturns)
  trialReturns(i) = trialReturn(trialFeatures, instruments)

}

this code make trialReturns(i) from 0 ->NaN

from aas.

I traced back and found totalReturn is NaN

from aas.

when CVCY.csv is in folder stocks/ then totalreturns will be NaN
is it Right?

from aas.

The return value type of readGoogleHistory is Array[(LocalDate, Double)
while the type of allStocks is Iterator[Array[(LocalDate, Double)]](auto inference by intellij with scala 2.11.8)
it seems not works well in

val allStocks = files.iterator.flatMap { file => try { Some(readGoogleHistory(file)) } catch { case e: Exception => None } }

Could you give me more tips?
@srowen
much thanks~

from aas.

The code compiles and runs fine. I'm not even clear what you're saying the problem is; allStocks is an iterator over results of readGoogleHistory.
I think you don't have the data downloaded. You need to download or unzip it from the .zip file provided. This code doesn't work out of the box any more because the original data moved, so there are extra steps here.

from aas.

stocksReturns.map(linearModel(, factorFeatures)).map(.estimateRegressionParameters()).toArray

I found this code returns a lot of NaN

and I check stocksReturns and factorFeatures:
stocksReturns is not null ,but it contains a lot of 0
factorFeatures is not null ,but it contains a lot of 0

3 files in folder factors/
and only CVCY.csv in folder stocks/.(this help me to reproduce the error faster,same error with all *.csv in folder stocks/ )

do you know how to modify it?
(of course I know it's absolutely OK in your platform)
much thanks~
@srowen

It seems that this model fails to be created.

from aas.

I check the doument of estimateRegressionParameters()

it said:
If there is no variance in y, i.e., SSTO = 0, NaN is returned.

in my stocksReturns and factorFeatures,
there's so much 0 -------->estimateRegressionParameter return NaN.
which may be the reason why ch09-code fails in my platform.

And Also,
If I do NOT use all the *.csv in folder stocks/ ,the ch09-code will succeed.
If I use spcecific *.csv(such as only CVCY.csv),it will reproduce the above error.
if I use all *.csv file in folder stocks/,it will also reproduce the above error.
@srowen

So I guess the reason of failure is due to the API defect,or the data need to be pre-preprocessed before fed into it.

from aas.