Curve Fitting about xchart HOT 26 CLOSED

There are several freely available fitters and spline interpolators in the commons-math project (http://commons.apache.org/proper/commons-math/), e.g. in package fitter and package analysis.interpolation.

Disclaimer: I am a developer for commons-math and was looking for a free library to do create some examples for our user-guide.

from xchart.

Awesome! I'd hate to reinvent the wheel again, so that looks great. Around ten years ago a wrote my own n-degree polynomial curve fitting algorithm in MATLAB using the Newton method. I've recently used the commons-math3 LUDecomposition for matrix math. Are you involved in that?

from xchart.

BTW, I just released V2.2.0: http://xeiam.com/xchart_changelog.jsp

If you want to use XChart for your user guide, it currently has 3 themes you can apply to the Charts: XChart, GGPlot2, and MATLAB. You can always make your own theme as well.

I also just added the capability in the release to export charts in high-res, for cases where charts need to end up in printed material. The export method lets you specify the DPI.

from xchart.

Ok cool. I did write some code in the linear package, but not the LUDecomposition, in case you have any questions or problem just post on the commons-user mailinglist, either me or someone else will quickly response usually.

from xchart.

@netomi I'm curious if you ever used XChart to publish your user guide? If so, could you post the link here? Cheers!

from xchart.

@netomi I'm also interested in integrating curve-fitting directly into XChart. Do you have any code to contribute to jump start the effort?

from xchart.

It is not yet published, some screenshots are committed to the source tree. I will keep you updated on this issue.

@curve-fitting: you can take a look at some of the unit-tests at https://svn.apache.org/repos/asf/commons/proper/math/trunk/src/test/java/org/apache/commons/math3/fitting/ it is quite straight-forward to use, but I can try to make a contribution if you can give me a bit of a sketch of the API that you would like to see.

from xchart.

@netomi I took a look at some of the unit tests and gave this some thoughts. Imagine a chart app as shown here. It would probably be better to make the scatter more linear and perhaps create a new ScatterChart05.java class for this first example.

The idea would be to be able to very simply create a second Series to add to a chart based on another original series. So in the above example, we have

 // Series
chart.addSeries("Gaussian Blob", xData, yData);

The best thing would be to extend this as follows:

 // Series
Series series = chart.addSeries("Gaussian Blob", xData, yData);

 // Fit Series
Series series2 = chart.addSeries(series, FitType.Linear);

Behind the scenes, XChart would use org.apache.commons.math3.fitting and give the user a great fit automatically, with just the simple one-liner above.

The fit series could also be internally styled so that the Markers are turned off by default.

Linear, exponential, and n-degree polynomial fits would be the first targets.

Histograms with Gaussian fits is also on our wish list, but that would come later...

What do you think?

from xchart.

@netomi Also, I'm curious, what's the difference between the fitting and interpolation packages?

from xchart.

@fitting vs interpolation: I think a good explanation can be found here: http://siber.cankaya.edu.tr/NumericalComputations/ceng375/node71.html

I will take a look the coming days and provide you with some prototype.

from xchart.

@netomi OK, that's pretty clear. Fitting is what we want (not interpolation), right? Do all the commons.math3 fitting algos use the least squares method??

Looking forward to a prototype impl!

from xchart.

sorry for the delay, but I was quite busy.
As a starter I created a first prototype of spline support, which is a bit different to the fitting, but would also be a nice contribution.

You can see a demo in my forked branch. Examples are LineChart07 or AreaChart01.

The idea is to support an interpolation type for each Series. The default would be linear, but a user can select also Spline. There are a few things missing:

• cache the data points in the Series object to not re-calculate them all the time a Series is painted
• correctly calculate the min/max values for the axis (due to the spline interpolation, the actual min/max values can exceed the original data points)
• do not paint markers for the interpolation points
• make the number of interpolation points configurable

I wanted to get feedback from you and then I can clean up my contribution.

from xchart.

I'll take a look at this ASAP. I'm really busy at the moment.

from xchart.

On 08/30/2013 11:10 AM, Tim Molter wrote:

@netomi https://github.com/netomi I took a look at your additions and
I'm quite excited about it. There are a few things that came to mind
right away...

1. In xchart-demo, it'd be better to create an entirely new package for
   this stuff. We can call it |curvefitting| for example. This way the
   other packages and examples remain untouched.

ok

1. For all demos, it'd be good to see both the original data AND the
   fitted data.

ok

1. The org.apache.commons.math stuff should be added as a dependency in
   the pom, rather than copy and pasting the code into the project.
   Also your lineChart05 depends on org.apache.commons.math.* and not
   the classes copied into XChart. Use the dependency management XML
   tag in the parent pom to define the dependency_version. In the the
   xchart pom add the dep without the version. See my XChange project.

I was unsure if you wanted to have a dependency to another lib,
especially as commons-math is already pretty big (~ 1MB), but I would of
course prefer to have it as a dependency.

The import is an artifact as I had commons-math first as a dependency,
but forgot to clean up prior to the commit.

1. We might want to call |SeriesInterpolationStyle.LINEAR|
   |SeriesInterpolationStyle.NONE| instead.

ok.

1. I like your approach of setting the style on a series better than my
   proposed idea of creating an original series and then creating a
   second fitted series from it (see previous comment). That way the
   chart doesn't necessarily need to display the original series.

I suggest you do a pull request for this stuff. Your additions are good
so far, and it's going in the right direction. Before the pull request
though, it'd be good to at least take care of the first three comments
above. If you do that, I can clean things up, fix some inefficiencies
and address some of the issues you've mentioned in a previous comment.

Well, it was just a quick and dirty proof of concept. There are several
things I do not like and will clean up before creating a pull request.

One thing I was wondering about: is there a specific reason why you do
all calculations with BigDecimal? This will be quite slow for large
graphs and I do not think the arbitrary precision is really needed.

Thomas

from xchart.

Damn, that's a large dependency. Let's just keep it how you have it then for now. We can always change it later.

is there a specific reason why you do all calculations with BigDecimal?

Good question. I think I was looking for an Object to encapsulate both floating point numbers and integers. When representing Integers with a Double, precision or exactness gets lost of course. Also, for Dates, representing the epoch time which is a long as a double seems strange. But you could totally be right. What would you suggest?

~Tim

from xchart.

On 08/31/2013 09:05 PM, Tim Molter wrote:

Damn, that's a large dependency. Let's just keep it how you have it then
for now. We can always change it later.

is there a specific reason why you do all calculations with BigDecimal?

Good question. I think I was looking for an Object to encapsulate both
floating point numbers and integers. When representing Integers with a
Double, precision or exactness gets lost of course. Also, for Dates,
representing the epoch time which is a long as a double seems strange.
But you could totally be right. What would you suggest?

There is nothing wrong to represent an integer value with a double (as
long as your integer can be represented with full precision in a double,
see below), of course you will lose accuracy when performing
calculations, but this is totally negligible for drawing a graph where
you have to deal more with the limitations of your screen resolution.

For the y Axis, converting the data to a BigDecimal is a complete waste,
as you have to create them via Number.doubleValue(), thus your original
data is represented as double, so you gain nothing by converting them to
a BigDecimal.

For the x Axis, where the data could be Dates represented as long, it
may look at first that a double would not be good enough. But a double
has 52 significant bits = 13 significant decimal places. With this you
can perfectly represent Date objects up to the year 70000. So there is
absolutely no need to use BigDecimal. Convert the data to double and
perform all calculations with doubles, this will be much faster.

As long as you do not do process money values or do scientific computing
where precision really matters, there is no need to use BigDecimal, imho.

In my fork, I did the necessary changes in the PlotContentLineChart
class, but left the others untouched, you may decide by youself how you
want to proceed.

Thomas

from xchart.

@netomi I will get rig of the BigDecimals like you suggest. Thanks for spotting that!

I don't see any recent changes in your fork. Perhaps you forgot to push. For moving forward it would be great if you could submit a pull request against the develop branch for what you've done so far plus the minor clean up things we've talked about. I'd be happy with giving to direct write access to the project if you'd like as well.

from xchart.

On 09/03/2013 12:03 PM, Tim Molter wrote:

@netomi https://github.com/netomi I will get rig of the BigDecimals
like you suggest. Thanks for spotting that!

I don't see any recent changes in your fork. Perhaps you forgot to push.
For moving forward it would be great if you could submit a pull request
against the develop branch for what you've done so far plus the minor
clean up things we've talked about. I'd be happy with giving to direct
write access to the project if you'd like as well.

Sorry for my late reply, I currently have some visitors at home and can
not spend much time on such things.

I did not push yet as I was not really happy with my changes wrt min/max
calculation for the axis, but will work on it soon.

Thomas

from xchart.

@netomi Take your time. No rush. :)

from xchart.

@netomi I got rid of all BigDecimals and replaced them with simple doubles. Thanks for the good suggestion.

from xchart.

Great!

btw. xchart is now officially part of the commons-math userguide projects, see here https://svn.apache.org/repos/asf/commons/proper/math/trunk/src/userguide/.

Results can be seen in the userguide section on the homepage: http://commons.apache.org/proper/commons-math/userguide/distribution.html

But there are more to come, just in case you are interested to hear about it.

from xchart.

@netomi

Yes, I'm definitely interested, so thanks for sharing! I'm very pleased to see XChart being used. Let me know when there is more to see. I must say, the charts look really great on the userguide.

FYI, I recently released version 2.3.0, which includes the BigDecimal --> double refactor, as well as real-time charts.

As far as integrating curve-fitting into XChart, I think it would be nice to add org.apache.commons.math as a dependency of XChange, even is the jar is 1 MB. The dependency would only be required for when people actually want to use the curve-fitting functions. There are also tools such as ProGuard that can be used to reduce the footprint of third-party jars when deploying an app.

I was also wondering if Math could help me accomplish something... I'd like to give XChart a bunch of data, make a histogram, fit say a normalcurve to it, and make a chart. Can Math do the work of creating the binned histogram of the data? Actually, that'd be pretty easy just to do myself...

from xchart.

Adding commons-math as an optional dependency should work, and you may benefit from other tools there as well. I got a bit distracted with the curve fitting contribution but I am still willing to help with that.

Regarding the histogram, there is a nice example on this stackoverflow page (the answer with the EmpiricalDistribution):

http://stackoverflow.com/questions/10786465/how-to-generate-bins-for-histogram-using-apache-math-3-0-in-java

from xchart.

@netomi After some thinking, I cannot decide if it is wise to include curve fitting into XChart, but rather keep XChart strictly for plotting data provided from the outside. In this case, people can just create their own series with the fitting capabilities of Math and add it to the chart. On the other hand it'd be neat to say chart.addNormalFit. I think I'm leaning towards the former. At least to start. There are some higher priority things right now. I first want to use Math and figure out the curve fitting and make some examples for sure.

FYI, check out the StyleManager class. I've added a lot of detailed style tweaking capabilities such as legendSeriesLineLength. If there is anything else you need, let me know.

Did you ever consider refactoring Math to be a multi-module Maven project? Math is a perfect candidate for it. There'd be several benefits.

from xchart.

Ok thanks, I will soon update the dependency and check the new styles.

@math: it is quite unlikely that math will become a multi-module project, but if you have a good argument for this you can really bring this up on the mailinglist.

from xchart.

Decided to leave this out of XChart. People can generate their own fit data. Maybe I'll make some examples using Math

from xchart.