Hi,
Could you please forward the sample test swmm model for which results are displayed in the docs of github repo
Thanks!

Remove this, or move to make more sense with swmmio.graphics.config

@aerispaha maybe I have overlooked something but is there some place where this code is documented with examples? It looks like it has plenty of nice features but I'm not entirely sure how to use them.

The Pandas DataFrames returned by the function swmmio.create_dataframeRPT always include a CFS label even if the model is run in SI units. swmmio should provide column headers consistent with the model units.

This is related to #14.

This is causing an issue in Python 2.7 Windows builds. We should either drop support for Python 2.7 or set the Pandas version explicitly to v0.24.2. This is causing a build fail in #63.

Processing pandas-0.25.0.tar.gz
Writing c:\users\appveyor\appdata\local\temp\1\easy_install-4dl70z\pandas-0.25.0\setup.cfg
Running pandas-0.25.0\setup.py -q bdist_egg --dist-dir c:\users\appveyor\appdata\local\temp\1\easy_install-4dl70z\pandas-0.25.0\egg-dist-tmp-7pcggv
Traceback (most recent call last):
  File "setup.py", line 68, in <module>
    "Programming Language :: Python :: 3.6",
  File "C:\Python27\lib\site-packages\setuptools\__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "C:\Python27\lib\distutils\core.py", line 151, in setup
    dist.run_commands()
  File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:\Python27\lib\site-packages\setuptools\command\develop.py", line 38, in run
    self.install_for_development()
  File "C:\Python27\lib\site-packages\setuptools\command\develop.py", line 154, in install_for_development
    self.process_distribution(None, self.dist, not self.no_deps)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 752, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 780, in resolve
    replace_conflicting=replace_conflicting
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 1063, in best_match
    return self.obtain(req, installer)
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 1075, in obtain
    return installer(requirement)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "C:\Python27\lib\site-packages\setuptools\command\easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 253, in run_setup
    raise
  File "C:\Python27\lib\contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
    yield
  File "C:\Python27\lib\contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 154, in save_modules
    yield saved
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
    yield
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "C:\Python27\lib\site-packages\setuptools\sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "c:\users\appveyor\appdata\local\temp\1\easy_install-4dl70z\pandas-0.25.0\setup.py", line 21, in <module>
    return version
  File "c:\users\appveyor\appdata\local\temp\1\easy_install-4dl70z\pandas-0.25.0\versioneer.py", line 1629
    print("Adding sample versioneer config to setup.cfg", file=sys.stderr)
                                                              ^
SyntaxError: invalid syntax
Command exited with code 1

Follow the convention used within pyswmm: support cleaner importing of the main classes, straighten out the config and definitions.

support these pattern:

from swmmio import Model, FloodReport, ComparisonReport, INPDiff, BuildInstructions

mymodel = Model('model.inp')
mymodel.draw(param, ...) #is there a better function word?
mymodel.conduits = modified_conduits #setters/getters to make changing a model more straightforward

#access global/config variables in a more obvious way
import swmmio 
swmmio.config.prj = 'epsg:2272'
swmmio.config.shapefile_directory = '/path/containing/shapefiles/for/basemap'

Create a new published release on PyPi using Appveyor

on_success:
  - "%PYTHON%\\python.exe -m pip install wheel"
  - "%PYTHON%\\python.exe setup.py bdist_wheel"
  - >
    IF "%APPVEYOR_REPO_BRANCH%" == "master"
    (
    IF "%APPVEYOR_REPO_TAG%" == "true"
    (
    echo SHOULD PUBLISH TO PyPi
    python -m pip install twine &&
    python -m twine upload dist/* --repository-url %REPO% -u %PYPI_USERNAME% -p %PYPI_PASSWORD%
    )
    )

Code relating to drawing is located in swmm_graphics and swmm_utils and should likely be all refactored into swm_graphics. Additionally, drawing options are clunky and could use a more streamlined approach.

When working with the create_dataframeINP function in dataframes.py, it always inserts the header (column definitions) of the Green-Ampt model in the data frame. This seems to happen since the inp_header_dict dictionary (in sectionheaders.py) only provides the parameters of the Green & Ampt model:

'[INFILTRATION]':'Subcatchment Suction HydCon IMDmax'

As a workaround, I defined a simple function that checks the infiltration model from the options section and adjusts the header definition for infiltration accordingly:

dict_inf_headers = {'GREENAMPT': ['Subcatchment','Suction','HydCon','IMDmax'], 'HORTON': ['Subcatchment','MaxRate','MinRate','Decay','DryTime',' MaxInfil]'], 'CURVE_NUMBER': ['CurveNum','','DryTime']}

Need to allow user to have control over order of things drawn. also, labels (streets) should always be drawn last.

Leverage Click for CLI tools

Hi Aerispaha,

I found that when I run SWMM in parallel, it fails randomly. For example, if I run 500 cases in parallel, 0-20 out of 500 may not getting any results and prompt 303 error: not compatible hot start file. The number of failed runs differs every try. What could be the problem please?

Replace with JSON or YAML file with more flexibility and information about each INP section (e.g. column header variation, units)

https://docs.readthedocs.io/en/latest/intro/getting-started-with-sphinx.html

(Re)move redundant / out of scope functions, pep8 compliance, consistency, review and clean up inheritance scheme

swmmio.py existing roadmap

class Model(object)

• __init__
• rpt_is_valid
• to_map
• _get_scenario
• conduits
• orifices
• weirs
• pumps
• nodes
• subcatchments
• node
• export_to_shapefile

class SWMMIOFile(object)

• __init__
• findByteRangeOfSection
• createDictionary

class rpt(SWMMIOFile)

• __init__
• createByteLocDict
• returnDataAtDTime

class inp(SWMMIOFile)

• __init__

Summary

Network analysis is often very useful in the context of water/drainage infrastructure. It would be convenient to provide a method for traversing/manipulating the swmmio.Model as a network/graph object. This could also provide a bridge to the sewergraph project.

Proposed API

m = swmmio.Model('path-to-model.inp')

# access the Networkx graph 
G = m.network

# analyze edges 
for u,v,d, in m.network.edges(data=True):
    upnode = G.node[u]
    dnnode = G.node[v]
    if dnnode['InvertElev'] > upnode['InvertElev']:
        print ('Backwards Conduit: ', d['Name'])

It looks like in swmmio.version_control, the create_combinations function doesn't correctly update the XSECTIONS part of the *.inp when a pipe already exists. The CONDUITS section is correctly updated (US/DS nodes, length, n, etc.).

I don't have hot start file defined, when run "python -m swmmio --run", it works, but when I try to run multiple cases by "python -m swmmio -sp Dir", it add hot start to the end of .inp file automatically and show error: "local variable 'simulationStart' referenced before assignment". Could you please kindly suggest?

When returning a dataframe of nodes data via swmmio.Model.nodes(), the column named MaxQ is ambiguous. Currently, it refers to MaxQ from the Node FLooding Summary table in the rpt: effectively, the max flow of flood water leaving the node.

MaxQ should probably be reserved for actual flow in the node, not the max rate of flooding.

When calling import swmmio.utils.modify_model (needed to run the function replace_inp_section()) an import exception is thrown... This seems to be related to another import in version_control.

Pandas is giving this warning due to a flavor of indexing set to change in future versions:

See the documentation here:
  http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated
    full_set = pd.concat([df1.ix[common_ids], df2.ix[common_ids]])
  C:\projects\swmmio\swmmio\version_control\inp.py:152: DeprecationWarning: 
  .ix is deprecated. Please use
  .loc for label based indexing or
  .iloc for positional indexing

Update swmmio to use the supported Pandas indexing pattern.

I have pip install swmmio and when I run "python -m swmmio ..." it prompts "from swmmio.swmmio import Model \ no module named swmmio". I tried both python 2.7 and python 3.7, what could be the problem please?

Update:
main.py
line 3
modify
from .swmmio import Model
to
from swmmio import Model
solved the problem.

Currently, there are multiple ways that models are compared within swmmio:

• in compare_models.py
• in the Change / BuildInstructions classes

Unify this process and scrap old junk code.

I am trying it and after installation in an Anaconda environmental i got this error when running the first line
from swmmio import swmmio

I got an error in the file spatial.py in line 1
from definitions import ROOT_DIR
error:
ImportError: No module named definitions

When I tried to ru sg.draw_model(mymodel) I got the following error:

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-16-c8ed5de8abd8> in <module>()
      1 from swmmio.graphics import swmm_graphics as sg
----> 2 sg.draw_model(mymodel)

C:\Users\Test\AppData\Local\Continuum\anaconda2\lib\site-packages\swmmio\graphics\swmm_graphics.pyc in draw_model(model, nodes, conduits, parcels, title, annotation, file_path, bbox, px_width)
    111         if title: annotate_title(title, draw)
    112         if annotation: annotate_details(annotation, draw)
--> 113         annotate_timestamp(draw)
    114 
    115         #SAVE IMAGE TO DISK

C:\Users\Test\AppData\Local\Continuum\anaconda2\lib\site-packages\swmmio\graphics\drawing.pyc in annotate_timestamp(draw)
    206         size = (draw.im.getbbox()[2], draw.im.getbbox()[3])
    207         scale = 1 * size[0] / 2048
--> 208         fnt = ImageFont.truetype(config.font_file, int(20 *scale))
    209 
    210         timestamp = strftime("%b-%d-%Y %H:%M:%S")

C:\Users\Test\AppData\Local\Continuum\anaconda2\lib\site-packages\PIL\ImageFont.pyc in truetype(font, size, index, encoding)
    242 
    243     try:
--> 244         return FreeTypeFont(font, size, index, encoding)
    245     except IOError:
    246         ttf_filename = os.path.basename(font)

C:\Users\Test\AppData\Local\Continuum\anaconda2\lib\site-packages\PIL\ImageFont.pyc in __init__(self, font, size, index, encoding)
    131 
    132         if isPath(font):
--> 133             self.font = core.getfont(font, size, index, encoding)
    134         else:
    135             self.font_bytes = font.read()

IOError: cannot open resource

I should say that I have no admin rights on this machine and I won't have them

Would it be cleaner to return model elements with only the attributes requested by the user, or should we continue to include as much related information as possible in the conduits() dataframe?

E.g. the conduit roughness coefficient is named "ManningN" within swmmio.

Spatial data is carried within the model element dataframes. This information would be very useful is easily exportable as a GeoJSON feature dataset.

The organizeConduitData / NodeData functions were one of the first written for this project and can most likely be reworked to use dataframe parsing functions in swmmio.utils.dataframes. I assume this will make the code more standardized, clean, and possibly faster.

Root cause seems to be the swmmio.dataframes,create_dataframeBI() function having trouble parsing the "Origin" column where file paths of parent INPs are written.

This process should probably reference a relative folder structure.

Need to run tests on Windows

not sure what it means, but i get this warning when i try to use swmmio

Warning (from warnings module):
File "C:\Python37\lib\site-packages\swmmio\swmmio.py", line 289
all_nodes = pd.concat([juncs_df, outfalls_df, storage_df])
FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=False'.

To retain the current behavior and silence the warning, pass 'sort=True'.

model.conduits() fails when the COORDINATES table in an inp file contains duplicate entries. The dataframes.get_link_coords() function fails because the dataframe.get_value(row.OutletNode, 'X') will return multiple records unexpectedly.

Hi,
when I do from swmmio import swmmio I get a syntax error coming from the definitions package:

In [1]: from swmmio import swmmio
  File "C:\Users\jeff\Anaconda2\lib\site-packages\definitions\parser.py", line 83
    yield from cls._flat_tree(element)
             ^
SyntaxError: invalid syntax

I am running python 2.7. From what I can tell, the yield from syntax is only valid in python 3. Have you had this problem or know of a work around?

When parsing models without rain gages, this warning always is presented.

header "[RAINGAGES]" not found in "*.inp"
returning empty dataframe

When generating the coords column in dataframes, some coordinates are being rounded. This causes spatial representations to be off.

Update package for Python 3 but maintain Python 2 compatibility.

Summary

All INP sections that follow the common tabular structure with unique indices as the first column can be read via the swmmio.utils.create_dataframeINP function. However, only a limited number of these sections have been added as properties in the higher-level swmmio.core.inp object.

Add coverage for read/writing all INP sections from the swmmio.core.inp object. INP sections are defined currently as properties with setter/getters in the swmmio.core module here. List of INP sections described here in the OWA SWMM repo.

INP Sections to Cover

• TITLE
• OPTIONS
• FILES
• RAINGAGES
• TEMPERATURE
• EVAPORATION
• SUBCATCHMENT
• SUBAREAS
• INFILTRATION
• AQUIFER
• GROUNDWATER
• SNOWPACK
• JUNCTIONS
• OUTFALLS
• STORAGES
• DIVIDERS
• CONDUITS
• PUMPS
• ORIFICES
• WEIRS
• OUTLETS
• XSECTIONS
• TRANSECTS
• LOSSES
• CONTROLS
• POLLUTION
• LANDUSE
• BUILDUP
• WASHOFF
• COVERAGE
• INFLOW
• DWF
• PATTERN
• RDII
• HYDROGRAPH
• LOADING
• TREATMENT
• CURVES
• TIMESERIES
• REPORT
• MAP
• COORDINATES
• VERTICES
• POLYGONS
• SYMBOLS
• LABELS
• BACKDROP
• TAGS
• PROFILES
• LID_CONTROLS
• LID_USAGE
• GWF
• ADJUSTMENT
• EVENT
• STREETS (new in SWMM 5.2)
• INLETS (new in SWMM 5.2)
• INLET_USAGE (new in SWMM 5.2)

Need to change setup.py (I think) to ensure the inp_settings files are included. Currently, this causes the run_models module to fail.

Hi @aerispaha, you have pulled together a great collection of tools here! Great work! I wanted to mention that there might be possible avenues of collaboration on some of my tools which are under development.

1. For running models we have pyswmm https://github.com/OpenWaterAnalytics/pyswmm
   This tool currently provides a pythonic interface to running SWMM. We are in the early stages of adding an Input API to SWMM (https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/tree/InputAPI) so that we can expose hydraulic and hydrologic data read in from an *.inp file. The "finished" version of this tool will enable optimization.
2. For rapidly pulling data out of the binary output file, you might consider looking at SWMMOutputAPI https://github.com/bemcdonnell/SWMMOutputAPI
   This tool provides a nice python interface to the binary output file.

There are many visualization tools that can be consolidated and/or removed. Find what is redundant and fix

Reproduce with INP with associated pump InletNodes all numeric (int) values:

pumps_df = dataframes.create_dataframeINP(inp_pth, "[PUMPS]", comment_cols=False)
coords_df = dataframes.create_dataframeINP(inp_pth, "[COORDINATES]")
verts = dataframes.create_dataframeINP(inp_pth, '[VERTICES]')


pumps_df.apply(lambda r: dataframes.get_link_coords(r, coords_df, verts), axis=1)

Inside get_link_coords the dtype of InletNode in pumps_df is assumed to be integer which causes problems here:

def get_link_coords(row, nodexys, verticies):
    """for use in an df.apply, to get coordinates of a conduit/link """
    x1 = round(nodexys.at[row.InletNode, 'X'], 4)
    y1 = round(nodexys.at[row.InletNode, 'Y'], 4)
...

This throws this error:

---> 21 pumps_df.apply(lambda r: dataframes.get_link_coords(r, coords_df, verts), axis=1)
     22 # dataframes.get_link_coords
     23 

c:\projectcode\swmmio\swmmio\utils\dataframes.py in get_link_coords(row, nodexys, verticies)
     69 def get_link_coords(row, nodexys, verticies):
     70     """for use in an df.apply, to get coordinates of a conduit/link """
---> 71     x1 = round(nodexys.at[row.InletNode, 'X'], 4)
     72     y1 = round(nodexys.at[row.InletNode, 'Y'], 4)
     73     x2 = round(nodexys.at[row.OutletNode, 'X'], 4)

~\Anaconda3\envs\emnet\lib\site-packages\pandas\core\indexing.py in __getitem__(self, key)
   1866                 raise ValueError('Invalid call for scalar access (getting)!')
   1867 
-> 1868         key = self._convert_key(key)
   1869         return self.obj._get_value(*key, takeable=self._takeable)
   1870 

~\Anaconda3\envs\emnet\lib\site-packages\pandas\core\indexing.py in _convert_key(self, key, is_setter)
   1913             else:
   1914                 if is_integer(i):
-> 1915                     raise ValueError("At based indexing on an non-integer "
   1916                                      "index can only have non-integer "
   1917                                      "indexers")

ValueError: ('At based indexing on an non-integer index can only have non-integer indexers', 'occurred at index 106829.1')

Hi again @aerispaha,
I was just using this very useful library again and had a question for you. I want to write my nodes and conduits into a shapefile. I used your spatial.py function write_shapefile and that worked, but the datatypes in the dataframe didn't transfer. Everything came across as a string. Have you thought about using Geopandas for this instead of the shapefile library? It seems like a nice fit since you are using Pandas dataframes already (which I love, btw). It also carries the datatypes into the shapefile. I'm happy to code this up and submit a PR if you think it's a good change.