Embedded bubble plot visualization about cost-of-capital-calculator HOT 49 CLOSED

Welcome @DoraSzasz!

One suggestion here would be to exclude the year slider bar (point (2)). This would require significantly changing the output that is computed from B-Tax. In addition, without restructuring the computations to utilize parallelization, this would result in runtimes that exceed what we probably want to have on the web app.

In the cited visualization, we were showing how bonus depreciation is being phased out under current law. If we wanted this user to see how important bonus depreciation is in the year they specified on CCC, we could simply add an additional set of results that include current law with the exception of bonus depreciation.

But my suggestion would be to first start with the bubble plot that displays the results for current law and the user defined reform in the year that was specified by the user.

cc @rickecon @MattHJensen

from cost-of-capital-calculator.

Is there a reason for the values of $10B and $80B in the index_landscape.html?

No. Didn't even notice that in the html. It's a remnant of a previous label for the asset size bubble scale (which is hardcoded, but would be nice if it weren't).

from cost-of-capital-calculator.

@DoraSzasz I think the bubbles in the revenue vs. rating plot need to be rescaled so that the biggest bubbles are smaller. But it looks good to me. I second @jdebacker 's comments.

from cost-of-capital-calculator.

@DoraSzasz asks:

Just a quick question to suggestion 1. Do you think that a color encoding based on Corporate/Noncorporate is helpful? And thus, we eliminate the radio button for corporate/non-corporate? Or, do you think that the color encoding should be on different equipment/structures/... assets? But if we are going to do this, all the bubbles in a row will have the same color, so it is not so helpful.

I think different colors for corp/non corp would be helpful to put them on the same plot, but one needs to be careful it doesn't get too busy to the point it's difficult to read.

Since different asset groups are already on different lines, it might be less beneficial to use different colors for each group, as you suggest.

but why are they so many categories (d and e)?

You are correct that the first part of the suffice refers to corporate and non-corporate. The additional part of the suffix refer to how the investment if financed. _e denotes variables relating to investment that is financed through new equity issuance. _d denotes variables relating to investment financed through debt. Variables without an _e or _d represent investment financed with a mix of debt and equity (we call this "typically financed investment" since we end up using a mix of debt and equity that is equal to their historical proportions in the corporate or non-corporate sector).

Of course, if we are going to truncate all those 0.00x to 0, all these columns will be equivalent. Is that right?

I don't know what this means. Can you clarify?

from cost-of-capital-calculator.

@DoraSzasz Cool!

I forgot, what are the circles that are not colored in?

In the "Going, Going, Gone" plot you are working from, the circles without color were the values for the base year (e.g. 2017). Then when one moved the sliders for the year, they were their for reference. As I note in my first comment above, I would probably do away with the year slider if these plots will be displayed as part of the default output from the Cost of Capital Calculator on TaxBrain. That said, you might still have the baseline results represented in the uncolored circles when the reform results are presented in color (again, to have some frame of reference for how much things change with the reform policy).

I also observed that even if the legend shows 4 dimensions for the circles, just 3 dimensions are used. Was this the intention?

I don't think this was the intention. I thought the code created a number of quantiles for asset size and that there was one bubble size per quantile. But perhaps I'm forgetting.

Otherwise, I think these look good. And with the logic to create those drop down menu selections, you are well on your way to Step #4.

from cost-of-capital-calculator.

OK. Let's scrap the year slider bar. @jdebacker @DoraSzasz

from cost-of-capital-calculator.

@rickecon @MattHJensen
I was looking at the plot and I have some other suggestions before starting:

1. I think that is may be a good idea to use color (variants of a color) of bubbles to encode an additional variable (e.g., the industry classification). Thus, for e.g., there will be 7 colors for the Equipment asset category and 4 colors for Structures asset category. I was thinking to use variance of blue to encode the components of Equipment category and for Others to use gray. Similar for Structures category. You can see the attached screenshot (revenue vs rating) for bubble color encoding:
   

2. Also, in the plot in the screenshot I like the way they represented the dimension of the bubble (see Production cost widget). Do you think that it is a good idea to have such a representation?

3. In web-page, 50.0% appears truncated. I will fix this, probably it is a matter of plot dimensioning?

Just to make sure, is it important to have in the same plot Structures and Equipment categories? Once we will integrate the drop-down menu, we will be able to see just one at time.
I will probably have a question about the x-axis variable that will include METTR, METR, cost of capital and deprecation, but I would like to design a prototype with the solved issues from 1-3, 5, 6 from issue and the proposed suggestions.

cc @jdebacker

from cost-of-capital-calculator.

@jdebacker

Is there a reason for the values of $10B and $80B in the index_landscape.html?

Thank you!

from cost-of-capital-calculator.

@DoraSzasz re your comments above:

1. I think color-coding is a good idea.
2. I'm find with either representation of bubble size - what you end up with might depend on the number and relative sizes of the bubbles.
3. You are right re plot dimensions. Also, let me point out (though you may have noticed), we wanted to have a certain order for the types o equipment and structures and ended up needing to use two plots right next two each other (so it looked like one plot). The need for two plots arose so that we could get the vertical titles for the broad categories (equip and structures) to help with dimensions of the plot.

A final note: I think we'll want to have equipment, structures, intellectual property products (IPP), inventories, and land as the four broad asset categories to plot (at least if we want to relate everything from the results tables).

from cost-of-capital-calculator.

Just a quick question to suggestion 1. Do you think that a color encoding based on Corporate/Noncorporate is helpful? And thus, we eliminate the radio button for corporate/non-corporate? Or, do you think that the color encoding should be on different equipment/structures/... assets? But if we are going to do this, all the bubbles in a row will have the same color, so it is not so helpful.

Also, which are the columns for corporate/non-corporate variables? I can see in reform_byasset these columns: z_c, z_nc, z_c_d, z_nc_d, z_c_e, z_nc_e, but I am struggling reading them. I can figure out that "_c" stands for corporate and "_nc" for non_corporate, but why are they so many categories (d and e)? Of course, if we are going to truncate all those 0.00x to 0, all these columns will be equivalent. Is that right?

@rickecon @jdebacker

from cost-of-capital-calculator.

Hi @jdebacker
I figured out how you mapped the data in window.data_sources in the jQuery function with the ColumnDataSource objects generated with Bokeh.

I am not sure why I have different values than you for 'assets_c in equipments_mettr_c_2020:

I am getting: 'assets_c': [17413293619.0, 53550810954.0, 0.0, 9260043911.0, 12396994619.0, 0.0, 7998988409.0, 28310184525.0, 498000000000.0, 133000000000.0, 75049828223.0, 108000000000.0, 25020946205.0, 7268611167.0, 6455272413.0, 118000000000.0, 91645013950.0, 12363020746.0, 219000000000.0, 278000000000.0, 476000000000.0, 423000000000.0, 168000000000.0, 120000000000.0, 136000000000.0, 266000000000.0, 62839620240.0, 109000000000.0, 8900871527.0, 202000000000.0, 46750113838.0, 24563500689.0, 124000000000.0, 9895536360.0, 84858812351.0, 101000000000.0, 2759304252.0, 16821590673.0, 220000000000.0, 72268924053.0, 197000000000.0, 199000000000.0], 'reform': [0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.24477376000000001, 0.26949442800000001, 0.26949442800000001, 0.26949442800000001, 0.25529827500000002, 0.351383047, 0.30957759400000001, 0.21943476100000001, 0.29054870599999999, 0.33927014500000002, 0.25571471100000004, 0.23326803399999999, 0.23812403800000001, 0.28716772499999998, 0.24897213500000001, 0.24897213500000001, 0.36468244599999999, 0.16729422199999999, 0.22739439399999997, 0.17669717699999998, 0.21797037, 0.25051200499999998, 0.25051200499999998, 0.28036625500000001, 0.23378471399999998, 0.24106374100000003, 0.28537121399999998, 0.28825875400000001, 0.24253759499999999, 0.31835877000000001, 0.28256452500000001, 0.36070317700000004, 0.26291823399999997, 0.26291823399999997]

While you have: 'assets_c': [17136828130.0, 51945942737.0, 0.0, 8982230464.0, 12038787586.0, 0.0, 7784262159.0, 27434323775.0, 472000000000.0, 121000000000.0, 69095240718.0, 104000000000.0, 24247307130.0, 7063509146.0, 6100491570.0, 114000000000.0, 87096847948.0, 11782738580.0, 216000000000.0, 274000000000.0, 466000000000.0, 402000000000.0, 164000000000.0, 117000000000.0, 134000000000.0, 261000000000.0, 57680307669.0, 107000000000.0, 7949436393.0, 196000000000.0, 39839633604.0, 21976408983.0, 118000000000.0, 9665918602.0, 75105175719.0, 96845226639.0, 2665605988.0, 16094427191.0, 211000000000.0, 70369397851.0, 192000000000.0, 194000000000.0], 'reform': [0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.43505156700000003, 0.24477376000000001, 0.26949442800000001, 0.26949442800000001, 0.26949442800000001, 0.25529827500000002, 0.351383047, 0.30957759400000001, 0.21943476100000001, 0.29054870599999999, 0.33927014500000002, 0.25571471100000004, 0.23326803399999999, 0.23812403800000001, 0.28716772499999998, 0.24897213500000001, 0.24897213500000001, 0.36468244599999999, 0.16729422199999999, 0.22739439399999997, 0.17669717699999998, 0.21797037, 0.25051200499999998, 0.25051200499999998, 0.28036625500000001, 0.23378471399999998, 0.24106374100000003, 0.28537121399999998, 0.28825875400000001, 0.24253759499999999, 0.31835877000000001, 0.28256452500000001, 0.36070317700000004, 0.26291823399999997, 0.26291823399999997]

I will re-verify the codes.

Thank you,
Dora

from cost-of-capital-calculator.

@DoraSzasz The index was created as the source data (a csv file) was read in a Pandas data frame. We didn't do anything to create a special order for the index.

I will admit that I'm no expert on this and @brendancol did most of the work with JS and getting the slider working, etc. You might ping him if you have more detailed questions on the JS stuff.

from cost-of-capital-calculator.

Thanks @jdebacker .
Yes, I figured out.

I am debugging now the JS part. Probably my next big wall to pass will be the correlation between the two Bokeh figures with the actual JS code. But for the moment, it is fine, I have no questions, apart the issue with assets_c, but I can re-check. However, the other columns match perfectly to your data.

Thank you for help,
Dora

cc: @brendancol

from cost-of-capital-calculator.

@jdebacker

Here is the first version with the point #5 of this issue solved (Drop down menu to allow the user to decide which industries to display in the two asset categories on the y-axis).
Here are the comments from @rickecon:

Dora,

Those look screen shots look really good. Below are some comments from me, but I would highly recommend that you include Jason DeBacker on this thread as he is the managing contributor to the b-tax repository.

1. I would include a label for the size-of-circle legend. I think the size means how much the particular firms in the industry invested in the particular type of equipment or structure. But Jason could tell you exactly what the label should be.

2. Despite its ease of getting the meaning from the context, I think the year slider bar should have a “Year” label somewhere.

3. I would love for these interactive graphics to have a button that saves the current figure snapshot as a .png file. The button might be labeled “Save as .png”. Is that possible?

4. I forgot, what are the circles that are not colored in?

Rick

@jdebacker Any suggestions based on Rick's comments, or other suggestions? I also observed that even if the legend shows 4 dimensions for the circles, just 3 dimensions are used. Was this the intention?
Also, could you give more explanations of point 4) of Rick's question?

Thank you,
Dora

from cost-of-capital-calculator.

@jdebacker
I am trying to solve point 2. of the issue.
One question: The blue circles in the figures, represent the baseline (mettr_c_year)?

I was looking at the metr_boxplot, but I see that you have used reform_byasset.csv and the function make_reform_sources in the Python build script (should I use these sources as reform ?) . Now, the question is: how do I obtain the data from reform function of year?

I should, in this case have something similar with mettr_c_year. A baseline_mettr_c_year and a reform_mettr_c_year.

Any suggestions?

CC: @rickecon

from cost-of-capital-calculator.

I am trying to solve point 2. of the issue.
One question: The blue circles in the figures, represent the baseline (mettr_c_year)?

In the "Going, Going, Gone" plot, the blue circles denote the tax policy in the year selected by the slider. When this year does not equal the baseline year (2016), then you will also see the unshaded circles that represent the 2016 policy.

I was looking at the metr_boxplot, but I see that you have used reform_byasset.csv and the function make_reform_sources in the Python build script (should I use these sources as reform ?) . Now, the question is: how do I obtain the data from reform function of year?

I'm not sure I understand this. If you are aiming to make a bubble plot that displays the output of the Cost of Capital Calculator in a graphical (rather than tabular) format (which is what I understood the task to be), then you will have output for the baseline and reform policies only in one year. Because output is only given for one year, my suggestions above were to ignore the year slider function for now.

My suggestion would be to use as your datasource, the the output of run_btax.run_btax_with_baseline_delta. You can do this by running (from a directory with your puf.csv file):

import btax
from btax.run_btax import run_btax, run_btax_with_baseline_delta
# specify IIT reform to be passed to Tax-Calculator.  e.g., 
iit_reform = reform_all = {
    2017: {
        '_PT_rt1': [0.15],
        '_PT_rt2': [0.15],
        '_PT_rt3': [0.15],
        '_PT_rt4': [0.15],
        '_PT_rt5': [0.15],
        '_PT_rt6': [0.15],
        '_PT_rt7': [0.15]
    }
}

# First argument says if have PUF or not - say False if have access to PUF
# Second argument is start year, third IIT reform, then B-Tax reform parameters
ModelDiffs = run_btax_with_baseline_delta(False,2017,iit_reform,btax_betr_corp=0.15,btax_depr_3yr_exp=0.,
         btax_depr_5yr_exp=0.,btax_depr_7yr_exp=0.,btax_depr_10yr_exp=0.,
         btax_depr_15yr_exp=0.,btax_depr_20yr_exp=0., btax_depr_25yr_exp=0.,
         btax_depr_27_5yr_exp=0.,btax_depr_39yr_exp=0.)
# ModelDiffs is a tuple of dataframes in this order: base_output_by_asset,
#                       reform_output_by_asset,
#                       changed_output_by_asset,
#                       base_output_by_industry,
#                       reform_output_by_industry,
#                       changed_output_by_industry,
#                       row_grouping

In this way, you can integrate the plot with the B-Tax calculations, which is what you'll want if this is to be embedded in the CCC output pages.

Please let me know if anything isn't clear.

cc @DoraSzasz @rickecon

from cost-of-capital-calculator.

Yes, I see now.
I was missing the part with how to get the data sources. Your suggestion is really helpful. I was using the ones that are in /resources folder.

Ok, I will ignore the year slider function.

Thank you,
Dora

from cost-of-capital-calculator.

@jdebacker. Are you sure that you want to do away with the year slider? I like the ability to change the year. And now that I understand what they are, I like the hollow uncolored circles for the base year of 2016. Let me know what you think about including a year slider bar.

from cost-of-capital-calculator.

Yes, I also like the year option, but as @jdebacker mentioned, this will require modification of the output computed by B-Tax and maybe higher computational time.

Yes, let me know what you decide.

from cost-of-capital-calculator.

@rickecon It depends on the intent of the plot and where it'll be used. B-Tax does not generate year by year results by default, so if this plot is to be incorporated into the CCC output, then I don't think you want the year slider. Doing so would require reconfiguring B-Tax to report year by year output and I don't think there is much interest in year by year results.

from cost-of-capital-calculator.

@jdebacker I am getting this error when I try to run the script you recommended from deploy, where the puf.csv file is: IOError: Expected one of (u'./data', u'./btax/data') to exist. Change working directory or define BTAX_CUR_DIR env var

Should I have celery and flask servers running?

from cost-of-capital-calculator.

@DoraSzasz I was able to run from deploy. Do you have B-Tax installed (i.e., if you do conda list do you see it?)?

I've never run the web app locally, so don't know about running celery and flask...

from cost-of-capital-calculator.

@rickecon @jdebacker just for my curiosity: How was the data with the years generated?
Also, should I let the hollow uncolored circles for the base year of 2016 in the figure?

@jdebacker Do we still need to use JQuery for interacting with plots? I think that we can use directly bokeh for the interactions with the plots, no?

from cost-of-capital-calculator.

@jdebacker Yes, I can see it.

from cost-of-capital-calculator.

@DoraSzasz Try updating B-Tax to a more recent release. I know some things changed with the paths...

Re the by year data. Those were generated by running a series of reforms, representing tax law in 2016-2017, 2018, 2019, 2020 (bonus depreciation is set to phase out over this time). The results were then saved to a csv file, with a column for each year.

from cost-of-capital-calculator.

@jdebacker Could you, please, give me the path of the puf.csv? I updated and git pulled B-Tax, but still not able to find it. Is it in btax/data folder? I can only see in /btax/test_data a puf91taxdata.csv.

Thank you,
Dora

from cost-of-capital-calculator.

@DoraSzasz puf.csv if not available in the repo.

I think you should be able to work around this. When calling B-Tax with:

ModelDiffs = run_btax_with_baseline_delta(False,2017,iit_reform,btax_betr_corp=0.15,btax_depr_3yr_exp=0.,
         btax_depr_5yr_exp=0.,btax_depr_7yr_exp=0.,btax_depr_10yr_exp=0.,
         btax_depr_15yr_exp=0.,btax_depr_20yr_exp=0., btax_depr_25yr_exp=0.,
         btax_depr_27_5yr_exp=0.,btax_depr_39yr_exp=0.)

Change the first argument to True. So you'd have:

ModelDiffs = run_btax_with_baseline_delta(True,2017,iit_reform,btax_betr_corp=0.15,btax_depr_3yr_exp=0.,
         btax_depr_5yr_exp=0.,btax_depr_7yr_exp=0.,btax_depr_10yr_exp=0.,
         btax_depr_15yr_exp=0.,btax_depr_20yr_exp=0., btax_depr_25yr_exp=0.,
         btax_depr_27_5yr_exp=0.,btax_depr_39yr_exp=0.)

This will run the model in "test" model and not use the puf.csv file.

Let me know how this works.

from cost-of-capital-calculator.

@jdebacker It works now.
However, in the command line, the output is:
"You loaded data for 2009.
Calculator instantiation automatically extrapolated your data to 2013.
You loaded data for 2009.
Calculator instantiation automatically extrapolated your data to 2013."

The files that are generated, are: base_byasset.csv, baseline_byindustry.csv, reform_byasset.csv, and reform_byindustry.csv.

Could you, please confirm that this is correct?

from cost-of-capital-calculator.

@jdebacker Could you, please look at the previous message? Thank you!
I think that now that we eliminate the integration of the slider with year, it is much better to do not have to add additional js code and create a single figure (not two, as you design) using Python and Bokeh. I am now working at restructuring the code. Do you see any problems that may appear if I am doing this? I am now learning more details about Bokeh and Bokeh server.

Thank you,
Dora

from cost-of-capital-calculator.

@DoraSzasz Sorry - thought I had responded, but I guess it didn't post.

But, yes, you are seeing the right output.

I think that now that we eliminate the integration of the slider with year, it is much better to do not have to add additional js code and create a single figure (not two, as you design) using Python and Bokeh. I am now working at restructuring the code. Do you see any problems that may appear if I am doing this?

I think that's right. The js code is was necessary for the slider. But you might need it for the dropdown menu, I'm not sure.

With regards to the two plots, I think I recall that as a solution to getting the two vertical titles - "Structures" and "Equipment".

from cost-of-capital-calculator.

@jdebacker
Short question: Is there a way to run bokeh server through B-Tax and CCC?

from cost-of-capital-calculator.

@DoraSzasz I don't think it'd be a problem to run on your local machine. I'm not sure about how it would work on the web app.

@PeterDSteinberg can you comment on this?

from cost-of-capital-calculator.

@jdebacker @PeterDSteinberg Yes, I am interested if bokeh server will work with webapp.

from cost-of-capital-calculator.

@DoraSzasz, getting the new plots to work on bokeh server is straightforward, but integrating that into webapp is harder. It runs as a separate service, so that would require some changes in deployment. Currently webapp only uses Postgres and Django for the frontend.

Its definitely possible. It might be a better alternative to using static plots.

from cost-of-capital-calculator.

@brittainhard and @MattHJensen . This issue is important. We are committed (both philosophically and in terms of specifically earmarked funds raised) to implementing dynamic visualizations from our web apps as well as on runs that users execute locally. As we are building out more dynamic visualizations to be produced in users' browsers after running some TaxBrain product, we need the requisite capabilities to produce those visualizations on the servers to which we send these user requests. Getting Bokeh (or D3 JavaScript) visualizations to be produced at the end of a TaxBrain, Cost of Capital Calculator, or OG-USA run is really important. My first thought is that we can use Bokeh because it will play nicely with our Python base code. How big of changes to the deployment will this require? @DoraSzasz @jdebacker

from cost-of-capital-calculator.

@brittainhard Thank you a lot for your response. I found that Bokeh server is really powerful for having interactive graphs. If it is not very hard to integrate it in your deployment, it will be really useful. Otherwise, we have to design the interactivity of Bokeh plots using JS. Personally I prefer using Bokeh plots with Python and Bokeh server allows designing both the plots and the interactivity using Python.
@rickecon @MattHJensen

from cost-of-capital-calculator.

@brittainhard @DoraSzasz @rickecon Let me know if you want to set up a call tomorrow or Friday to make sure we are all on the same page with what we need to have dynamic plots included in the web-app output pages.

from cost-of-capital-calculator.

@jdebacker @brittainhard @rickecon I am available on Friday at any time. Feel free to choose the hour.

from cost-of-capital-calculator.

What sort of interactivity would need to be included in the webapp output? My understanding was that those are meant to visualize the output of CCC, and ideally they can be pasted into static documents like papers.

from cost-of-capital-calculator.

@MattHJensen I think we are going to at lest want the images to be dynamic so that one can toggle through various outputs (e.g. select "by industry" or "by asset", selection METR or METTR, etc). Though we could of course just generate a number of different static images an radio buttons on the webpage to toggle between.

No matter which of those we chose, I do agree with you that the option to download a static image in png or pdf format would be ideal.

from cost-of-capital-calculator.

@MattHJensen @jdebacker I agree with @jdebacker. Yes, I already integrated the function of saving as .png. I will discuss with @rickecon the other details and hope to publish the two graphs soon.

from cost-of-capital-calculator.

BTW, I would still like to chat about whether we need to run Bokeh server or not. I just don't know enough about it for the pros/cons in the case. I do see that we can just have CCC pass the output data frames to a script that then builds the plots - and the web app can then read in the html files that are generated form this build plots script.

from cost-of-capital-calculator.

@jdebacker I would love to chat about this. I like Bokeh server for having real-time plots and easy to make interactivity. But, if needed, I will adapt my code to work with Flask/Django, too.

Here are the other options I added to the graph: Wheel Zoom, Save as PNG, Reset. I still have to prettify it a little. Now it is generated entirely using Bokeh and Bokeh server.

from cost-of-capital-calculator.

Setting up Bokeh server for the webapp would cost a substantial amount of developer time up front and maintenance down the road. I recommend avoiding it for now.

from cost-of-capital-calculator.

@MattHJensen , What does that mean for these visualizations? What is the alternative to setting up Bokeh server (D3.js)? I thought we wanted these visualizations to be produced for users after runs of the web app. If they are not created from the web app, then where do you envision the visualizations being generated and used? @DoraSzasz @jdebacker

from cost-of-capital-calculator.

We can still integrate them with Flask/Django to produce interactivity. They will be created real-time from web app. Could you, please, confirm this, @jdebacker ?

from cost-of-capital-calculator.

@DoraSzasz is right: we can either switch from static plot to static plot using the webapp frontend, or we can embed interactive Bokeh plots that use JS for interactivity, or we can do some of both. The choice between the two really depends on how we'd like the user interface for CCC to be.

from cost-of-capital-calculator.

@MattHJensen I agree. However, we need to choose one of the ways to do the interactivity, since the code for implementing interactivity is different for each way (JS, Bokeh server, Flask/Django). We should definitely chat about this. @jdebacker

from cost-of-capital-calculator.

This has been done for B-Tax output on PolicyBrain (via the CCC). Closing here, but may reopen issue when addressing B-Tax output for local runs.

from cost-of-capital-calculator.