california-data-collaborative / ratecomparison Goto Github PK

To reproduce, set the number of months to forecast = 0 while planning is active.

Error:

Warning in new_recent_month_data$usage_ccf[1:nrow(recent_month_data)] <- tmp$usage_ccf.y :
number of items to replace is not a multiple of replacement length
Warning: Error in :: argument of length 0
Stack trace (innermost first):
70: reactive:planneddf [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#71]
59: planneddf
58: dropNulls
57: updateSliderInput
56: observerFunc [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#204]
1: shiny::runApp
Warning: Error in :: argument of length 0
Stack trace (innermost first):
106:
105: stop
104: planneddf
103: list_or_dots
102: dplyr::bind_cols
101: eval
100: eval
99: %>%
98: reactive:df_plots [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#296]
87: df_plots
86: eval
85: eval
84: %>%
83: is.data.frame
82: colSums
81: plot_fixed_revenue [R/make_plots.R#245]
80: inherits [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#742]
79: as.widget
78: func
77: origRenderFunc
76: output$fixed_revenue_barchart
1: shiny::runApp
Warning: Error in :: argument of length 0
Stack trace (innermost first):
87:
86: stop
85: df_plots
84: eval
83: eval
82: %>%
81: plot_revenue_over_time [R/make_plots.R#21]
80: inherits [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#430]
79: as.widget
78: func
77: origRenderFunc
76: output$revenue_time_series
1: shiny::runApp
Warning: Error in :: argument of length 0
Stack trace (innermost first):
89:
88: stop
87: df_plots
86: eval
85: eval
84: %>%
83: is.data.frame
82: colSums
81: plot_barchart_by_tiers [R/make_plots.R#199]
80: inherits [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#500]
79: as.widget
78: func
77: origRenderFunc
76: output$barchart_by_tiers
1: shiny::runApp
Warning: Error in :: argument of length 0
Stack trace (innermost first):
99:
98: stop
97: df_plots
96: eval
95: eval
94: %>%
93: reactive:df_change [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#523]
82: df_change
81: plot_bill_change_histogram [R/make_plots.R#102]
80: inherits [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#703]
79: as.widget
78: func
77: origRenderFunc
76: output$bill_change_histogram
1: shiny::runApp

Warning: Error in :: argument of length 0
Stack trace (innermost first):
84:
83: stop
82: df_change
81: plot_bill_change_boxplot [R/make_plots.R#153]
80: inherits [C:\Users\Christopher\Documents\CA_Data_Collab\repos\RateComparison/server.R#723]
79: as.widget
78: func
77: origRenderFunc
76: output$bill_change_boxplot
1: shiny::runApp
ERROR: [on_request_read] connection reset by peer

The fixed charge should change based on the meter size of the account. There should be new input boxes on the left that allow the user to specify the charge for each meter size.

Right now only the barchart_by_tiers changes when this toggle is switched. Eventually when demand dynamically adjusts in response to price, it will be useful for the line chart revenue_over_time, and the histogram+boxplot to respond to the toggle also.

Instead of assuming constant demand, water demand should change as price changes. These demand changes will then propagate down to revenue and equity considerations. This will depend on having estimates of price elasticity ( issue #1 )

If anyone has specific ideas about how this should be implemented, (i.e. someone with more economics knowledge than me!) please feel free to comment on this issue.

To do:

• Get Division, User Code, and Power Zone data into SCUBA
• Get data transfer with Service point / Meter ID
• Use service/meter ID as location ID?
• Set cust_loc_rate_class to align with desired groupings
• Update OWRS file with the above rate classes
• Sort out zero-budget issue in data transfer
• Compare historical calculated revenues with total sales data to check accuracy

cc @patwater

@christophertull @anudeepvanjavakam1 no urgent though would be good go through the code and see how we might speed up load times -- currently pretty slow. Part of that might be due to the transition to the analytics portal (@christophertull thoughts?)

See here for one method of analysis to profile the code itself: http://rstudio.github.io/profvis/examples.html#example-3---profiling-a-shiny-application

There might also need to be some thinking in terms of the server configuration though that's bumping up against the limits of my knowledge :P

Another option if we need to deploy this for a large number of agencies might be to get an RStudio Server account -- not sure how that would affect speed though

Right now the baseline OWRS file is loaded from a hardcoded filepath. Users should be able to select their own OWRS file using an "Upload" button.

See barchart.

Currently, in order to calculate the "baseline" bills, a rate structure must be hard-coded into the source of the tool. It would be much better if the pricing structure could be read from a text file that fully specified the structure and any data dependencies.

Most likely, this will take the form of a function calculate_bill() that could do something like the following:

total_bills <- calculate_bill( data, rate_schedule_string )

Before this can happen, a format for storing and transmitting rates needs to be devised, which is a problem that I a working on, probably in a different repo.

Goal is to have a smoothing of historical bi-monthly bills so that the forecasts don’t get so jumpy during the initial horizon. Potential solution is to convert each bill to a daily value and apply a logic to split it across months to standardize (and maybe have an option for the user to specify frequency).

@christophertull thoughts on whether this would be better handled within RC or SCUBA? Note this issue is reoccurring with other utilities so could image a smoothed monthly usage column in the monthly table. Non-urgent though something to discuss.

When adding/removing a new tier, there will be a few seconds when the number of rows in the "Tier Start" box is not the same as the number of rows in the "Tier Prices" box. During this time, the charts are all replaced with a red error message. It would be nice if this error message did not appear, or if it took longer to appear, giving the user more time to adjust the tiers.

Options for fixing as I see it:

1. I tried adding an "Update Tiers" button that was tied to only the tiers and nothing else. I think the code is still present, just commented out. This sort of worked, but prevented any of the charts from appearing initially until the "update tiers" button was pressed. It would be better if the charts appeared by default, and then any subsequent changes to the tiers required a push of the "update tiers" button.
2. It is possible that each tier could have it's own row, such that the tier start and price for a tier were both int he same box, and new boxes could be added as desired? This seems like it would be a little messier and it would also require rewriting the code that reads and parses the boxes.

Other ideas are welcome.

MNWD rate codes issues:
RW does not include RC5 (RC 5 is hydrant and should be in OTHER- they don't have a budget)
RESIDENTIAL_MULTI should have A2 included (its not in any others from what I could see (note that A2 does not have sewer for the sewer model)
COMMERCIAL should not include CM1MN or CM2MN as those are MNWD accounts and do not pay bills- delete from Commercial- could be in INSTITUTIONAL if desired to include

Users should be able to load a rate structure from a file, either to set the baseline rates or to populate the boxes in the customer class tabs. There should probably be an import button somewhere.

Before this can happen, a format for storing and transmitting rates needs to be devised, which is a problem that I a working on, probably in a different repo.

In the same way that simple fields can depend on columns, the tier starts and tier prices should be able to depend on data fields. This is already supported by RateParser and the OWRS file, but not by the UI inputs here

SMC updated their portal. It's available here: https://data.santamonica.gov/dataset/water-usage

Right now this toggle does not work, although the framework to begin is visible in the barchart_by_tiers plot. This should definitely be implemented for barchart_by_tiers, and maybe for the histogram and boxplot (e.g. one could see the percentage change in bill) as well.

@monobina ran across an error that arises here

#for generating customer class
class_proportions <- as.data.frame(prop.table(table(df$cust_class)), stringsAsFactors = FALSE)
      
class_proportions$Freq <- c(input$Commercial,input$Institutional, input$Irrigation,input$Other,input$ResidentialMulti, input$ResidentialSingle)

Note how the class_proportions are generated dynamically from the dataframe, but the inputs are hardcoded. This throws an error if there are more (or fewer?) classes in the dataframe than there are inputs. To get around this we should probably generate the account growth inputs in a loop.

The error for reference:

Stack trace (innermost first):
    84: $<-.data.frame
    83: $<- [C:\Users\mmukherjee\Desktop\MNWD\RateRP3.2.9/server.R#62]
    82: <reactive:planneddf> [C:\Users\mmukherjee\Desktop\MNWD\RateRP3.2.9/server.R#62]
    71: planneddf
    70: <reactive:DF> [C:\Users\mmukherjee\Desktop\MNWD\RateRP3.2.9/server.R#511]
    59: df_original
    58: dropNulls
    57: updateSliderInput
    56: observerFunc [R/class_graphs.R#223]
     1: runApp
Warning: Error in $<-.data.frame: replacement has 6 rows, data has 7

Without prior explanation, it can be difficult to understand what all the charts mean. It would be useful to figure out a way to show plain language explanations for each chart to aid user understanding.

Maybe another view is possible (e.g. a "Help" view) where chart explanations get overlaid on top of each chart?

It will be tricky to do this without making the interface too cluttered, but it is important that users understand what they are seeing!

Users should be able to specify a growth rate, (e.g. 1% per year, 200 accounts per year, etc) and that growth rate should get factored into water demand forecasts. This should be implemented in the scenario planning tab ( issue #5 ).

Implementation

One thought would be to randomly sample from existing customers in order to simulate growth in the future.

If random sampling is chosen, then care should be taken that the the sample is done proportionate to the expected growth in each customer class. (e.g. if only residential growth is expected, then we shouldn't be sampling from industrial)

Add a way to specify price elasticity of demand for each tier.

This could be as simple as another box next to the tiers where price elasticities can be specified.

Populate the box with reasonable default values. These could probably be pulled from the economics literature?

Right now the different UI tabs and charts are all copy-pasted and suffixed with numbers (e.g. df_plots1, df_plots2, etc). This is fine for the short term, but it will make the code more readable, and maintainable if we refactor these into Shiny Modules and then generate them in a loop.

See here for info on creating modules: http://shiny.rstudio.com/articles/modules.html

And here for generating a UI in a loop: http://shiny.rstudio.com/gallery/creating-a-ui-from-a-loop.html

After designing a suitable rate schedule, users should be able to save and export their schedule into a machine-readable format.

Before this can happen, a format for storing and transmitting rates needs to be devised, which is a problem that I a working on, probably in a different repo.

After multiple customer classes have been added (issue #3), there should be a view showing the aggregate effects of the new rate structure across all customer classes.

Anyone have any ideas about what sorts of graphs/charts should be displayed here? It could be the same as the class-specific pages but it doesn't have to be.

The Github wiki should have a detailed user guide to explain how to use the tool, how to interpret the charts (see #9 ), etc.

If scenario planning is enabled ( see issue #5 ), then instead of just looking retroactively, we should be able to forecast water use into the future.

The exact details are completely open, but some possibilities include:

1. Extrapolating past use into the future, either by copying directly or by averaging over previous years.
2. Creating a statistical model of water demand in aggregate for each customer class (care should be taken here so that factors like population growth can be factored in).
3. Creating a statistical model at the customer level and then aggregating up to each customer class. This would be the most complicated but also the most flexible.

Option 1 would probably be a good and relatively easy start, with option 3 probably being the best choice overall.

This could be a new tab on the left where optional scenario analyses can be added later.

One idea would be to have an "Enable" checkbox in the tab that, when checked, allows forecasting water use into the future instead of just looking retroactively. These forecasts could then be modified by other scenario planning options (specified in other issues).

It could be useful to have descriptive statistics available in some of the views. Some of these are available already when hovering over a chart (thanks to Plotly) but others have suggested to have more/more visible stats like counts, means, medians, etc.

Maybe add a vertical dotted line where the true data ends? or change to dashed line chart for predicted values?

Right now forecasting is done in an arithmetic manner using simple means. Eventually a more robust forecasting method might support the display of error bounds. See previous discussion here: #26

Summary stats of number of customers by rate code, meter size, etc.

Add new tabs corresponding to different customer classes.

When on budget based rates, enabling planning changes the revenue calcs from this:

to this:

After census information has been integrated, there should be charts to show:

Change in bill broken out by income/education.

Maybe scatter plot where users can choose which census demographic?

For the forecast module, the revenue by tier, fixed vs variable and bill impact should only show the future data, not the past data. What usage and et data are you using in the future too? It would be good to be able to select behavior/ external factors to model the impacts and sensitivity.

-Drew