airbnb / aerosolve Goto Github PK

What does the parameter "lossOnly" mean in AdditiveModelTrainer?

While trying to use aerosolve for kaggle airbnb competition, I noticed in com.airbnb.aerosolve.core.transforms.Transformer.java
that only string features are being copied from context to examples.
Is this behavior coded on purpose ?
In general, is the differentiation between context and item features only for memory optimization or does it impact the algorithms ranking task ?
Thanks in advance for any advices to help.

missing FeatureVector.java,then project can't build

Hello:
In file： FeatureVectorGen line3，4 can‘t found example and featureVector, is this an error?
Thanks.

import com.airbnb.aerosolve.core.Example;
import com.airbnb.aerosolve.core.FeatureVector;

"/aerosolve/demo/income_prediction/sh job_runner.sh MakeTraining" returns:

/aerosolve/demo/income_prediction/build/libs/income_prediction-0.1.3-all.jar does not exist, skipping.
java.lang.ClassNotFoundException: com.airbnb.aerosolve.demo.IncomePrediction.JobRunner

income_prediction-0.1.3 is now income_prediction-0.1.6

Hi, there!

We are a research team working on third-party library analysis. We have found that some widely-used third-party libraries in your project have major/critical bugs, which will degrade the quality of your project. We highly recommend you to update those libraries to new versions.

We have attached the buggy third-party libraries and corresponding jira issue links below for you to have more detailed information. We have analyzed the api call related to the following libraries and found one library that is using the API call that might invoke buggy methods in the library of the history.

1. commons-codec commons-codec
   version: 1.4
   API call in your project:org.apache.commons.codec.binary.Base64.setInitialBuffer(byte[],int,int)

Jira issues:
Base64InputStream#read(byte[]) incorrectly returns 0 at end of any stream which is multiple of 3 bytes long
version:1.4
ArrayIndexOutOfBoundsException when doing multiple reads() on encoding Base64InputStream
version:1.4
Base64 encoding issue for larger avi files
version:1.4
org.apache.commons.codec.net.URLCodec.ESCAPE_CHAR isn't final but should be
version:1.2;1.3;1.4
org.apache.commons.codec.language.RefinedSoundex.US_ENGLISH_MAPPING should be package protected MALICIOUS_CODE
version:1.4
org.apache.commons.codec.language.Soundex.US_ENGLISH_MAPPING should be package protected MALICIOUS_CODE
version:1.4
Caverphone encodes names starting and ending with "mb" incorrectly.
version:1.4
All links to fixed bugs in the "Changes Report" http://commons.apache.org/codec/changes-report.html point nowhere; e.g. http://issues.apache.org/jira/browse/34157. Looks as if all JIRA tickets were renumbered.
version:1.1;1.2;1.3;1.4
Regression: Base64.encode(chunk=true) has bug when input length is multiple of 76
version:1.4
DigestUtils: MD5 checksum is not calculated correctly on linux64-platforms
version:1.3;1.4
new Base64().encode() appends a CRLF; and chunks results into 76 character lines
version:1.4
Base64 encode() method is no longer thread-safe; breaking clients using it as a shared BinaryEncoder
version:1.4
Base64 default constructor behaviour changed to enable chunking in 1.4
version:1.4
Base64InputStream causes NullPointerException on some input
version:1.4
Base64.encodeBase64String() shouldn't chunk
version:1.4
2. org.apache.commons commons-lang3
version: 3.4
Jira issues:
TypeUtils.ParameterizedType#equals doesn't work with wildcard types
version:3.3.2;3.4
DateUtilsTest.testLang530 fails for some timezones
version:3.4
StringUtils.stripAccents from "Ł" and "ł"
version:3.4
No release notes for version 3.4
version:3.4
JsonToStringStyle doesn't handle chars and objects correctly
version:3.4
ReflectionToStringBuilder doesn't throw IllegalArgumentException when the constructor's object param is null
version:3.4
StrLookup.systemPropertiesLookup() no longer reacts on changes on system properties
version:3.4
StringUtils#capitalize: Javadoc says toTitleCase; code uses toUpperCase
version:3.4
Multiple calls of org.apache.commons.lang3.concurrent.LazyInitializer.initialize() are possible
version:3.4;3.5
EnumUtils *BitVector issue with more than 32 values Enum
version:3.4
StringUtils#equals fails with Index OOBE on non-Strings with identical leading prefix
version:3.4
There are no tests for CharSequenceUtils.regionMatches
version:3.4
ArrayUtils.removeAll(Object array; int... indices) should do the clone; not its callers
version:3.4
TypeUtils.isAssignable throws NullPointerException when fromType has type variables and toType generic superclass specifies type variable
version:3.4
FastDateFormat does not support the week-year component (uppercase 'Y')
version:3.4
ordinalIndexOf("abc"; "ab"; 1) gives incorrect answer of -1 (correct answer should be 0)
version:3.4
Fix implementation of StringUtils.getJaroWinklerDistance()
version:3.4
parseDateStrictly does't pass specified locale
version:3.4
ClassUtils.getClass(ClassLoader; String) fails for "void"
version:3.4
NumberUtils.isNumber bug
version:3.4
FastDateFormat doesn't respect summer daylight in localized strings
version:3.4
StringUtils#normalizeSpace does not trim the string anymore
version:3.4
DiffBuilder: Add null check on fieldName when appending Object or Object[]
version:3.4
FastDatePrinter Memory allocation regression
version:3.4
SerializationUtils.ClassLoaderAwareObjectInputStream should use static initializer to initialize primitiveTypes map.
version:3.2;3.3;3.4
NumberUtils.isNumber and NumberUtils.createNumber resolve inconsistently
version:3.4
ArrayUtils.contains returns false for instances of subtypes
version:3.4
CompareToBuilder.append(Object;Object;Comparator) method is too big to be inlined
version:3.4
StrBuilder#replaceAll ArrayIndexOutOfBoundsException
version:3.2.1;3.4;3.5
NumberUtils#createNumber() returns positive BigDecimal when negative Float is expected
version:3.x

Sincerely~
FDU Software Engineering Lab
Marth 14th,2019

The README says that the Linear model "only operates on string features", but the demo using the linear model (image_impressionism), appears to use both string features and float features. Is this sentence in the Readme incorrect or am I missing something?

Hi, I am a potential house renter, I want to sell my house and buy a few apartments in my city, qinhuangdao, China. I am wondering could I pull the historical price data on airbnb in my city and analyze myself, in order to make a final decision? I would very much appreciate your help. Thank you a lot!!!!

Version in the following command should be updated:

aerosolve/demo/image_impressionism/README.md
91: spark-shell --master local --jars build/libs/image_impressionism-0.1.7-all.jar

If you go to http://airbnb.io/projects/aerosolve/ and click on the big red GitHub button, you are actually redirected to the airpal repository. I know this issue is not related to aerosolve itself, but I didn't know where to tell you best.

I got this error while trying to run sh job_runner.sh TrainModel

I ran it within the demo/income_prediction folder.
Everything was built with Gradle successfully.

sh job_runner.sh MakeTraining and sh job_runner.sh MakeTesting worked and did make outfile files

Running
scala 2.11.8
spark 2.0.1

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.rdd.RDD.coalesce$default$3(IZ)Lscala/math/Ordering;
	at com.airbnb.aerosolve.training.AdditiveModelTrainer$.sgdTrain(AdditiveModelTrainer.scala:416)
	at com.airbnb.aerosolve.training.AdditiveModelTrainer$.train(AdditiveModelTrainer.scala:262)
	at com.airbnb.aerosolve.training.AdditiveModelTrainer$.trainAndSaveToFileEarlySample(AdditiveModelTrainer.scala:786)
	at com.airbnb.aerosolve.training.AdditiveModelTrainer$.trainAndSaveToFile(AdditiveModelTrainer.scala:768)
	at com.airbnb.aerosolve.training.TrainingUtils$.trainAndSaveToFile(TrainingUtils.scala:192)
	at com.airbnb.aerosolve.demo.IncomePrediction.IncomePredictionPipeline$.trainModel(IncomePredictionPipeline.scala:93)
	at com.airbnb.aerosolve.demo.IncomePrediction.JobRunner$$anonfun$main$1.apply(JobRunner.scala:41)
	at com.airbnb.aerosolve.demo.IncomePrediction.JobRunner$$anonfun$main$1.apply(JobRunner.scala:32)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at com.airbnb.aerosolve.demo.IncomePrediction.JobRunner$.main(JobRunner.scala:32)
	at com.airbnb.aerosolve.demo.IncomePrediction.JobRunner.main(JobRunner.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala

:core:compileJava (Thread[Task worker for ':',5,main]) started.

Task :core:compileJava
Putting task artifact state for task ':core:compileJava' into context took 0.0 secs.
Executing task ':core:compileJava' (up-to-date check took 0.006 secs) due to:
Task has failed previously.
All input files are considered out-of-date for incremental task ':core:compileJava'.
Compiling with JDK Java compiler API.
/Users/maximebodereau/Documents/Projects/Ux AI/aerosolve/core/src/main/java/com/airbnb/aerosolve/core/util/Weibull.java:13: error: cannot find symbol
public WeibullBuilder defaultBuilder() {
^
symbol: class WeibullBuilder
location: class Weibull
1 error

:core:compileJava (Thread[Task worker for ':',5,main]) completed. Took 0.262 secs.

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':core:compileJava'.

java.lang.NoSuchFieldError: pid

Please

In demo/income_prediction/job_runner.sh, the version number for the income prediction jar file is old. As a result, there is a ClassNotFoundException when trying to find the JobRunner class.

Existing:
build/libs/income_prediction-0.1.3-all.jar \

Should be:
build/libs/income_prediction-0.1.6-all.jar \

Please apply this change and commit to the git repo.

Thanks!

Hi, I'm a Stanford MS student trying to run the image impressionism and income classification demos. When running gradle shadowjar --info, I get multiple errors of the following type during the execution of the task :core:compileJava:

/Users/ei5h4/Documents/aerosolve/core/build/gen-java/com/airbnb/aerosolve/core/ModelRecord.java:1075: error: method hashCode in class Object cannot be applied to given types;
      hashCode = hashCode * 8191 + org.apache.thrift.TBaseHelper.hashCode(featureWeight);
                                                                ^
  required: no arguments
  found: double
  reason: actual and formal argument lists differ in length

My thrift version is 0.10.0. I tried downloading and installing an older version of thrift (0.9.0) from source since this demo is old and might rely on an older thrift (just a hypothesis). But that turned out to have some roadblocks as well since the older thrift uses some C code namespace tr1 that is no longer supported by C++11 on my OSX El Capitan. So I couldn't verify if thrift is the issue or something else. Basically I thought the hashCode function in the error above might have a changed signature from 0.9.0 to 0.10.0.

I think anyone else attempting to build the demo will run into this issue as well. Really hope to get this running on my machine soon. Aerosolve is super exciting!

I hope that someone could help me.
I have this issue when running sh job_runner.sh DebugTransform

ERROR Runner: Exception on job DebugTransform : com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'delete_string.fields'

maybe this empty class cause this:

`package com.airbnb.aerosolve.core.transforms;

// TODO: remove this once all configs have migrated over to the new transform names
public class DeleteStringFeatureColumnTransform extends DeleteStringFeatureFamilyTransform {}
`

Thanks a lot

Hi, I'm confused about this parameter about 'node_query'. Would you please provide example data or sql? Thanks a lot

I think that it will be great to have the training module compiled with Scala 2.11, because the version 2.10 is relatively old.