We need a preferences file that creates global variables when OpenPNM is imported, which contains information such as default IO paths.

Someday, when we have an OpenPNM binary install, this will be something that gets populated during the install sequence.

Do you guys know how to do this? @putza @RodericDay

Website documentation did not accept the ":code:" text. See for example:
http://openpnm.org/src/docs/_build/html/userguide/data_storage.html

It works fine on my local doc build though.

Relating to the discussion during the telecon of Oct9th (see #44) we are considering allowing scalar pore and throat properties which will implicitly apply to all pores.

We need these for two reasons:

1. To store things like system temperature for isothermal simulations where a temperature field is not calculated, hence we have pn.pore_properties['temperature'] as a scalar
2. In cases where temperature IS calculated we want the code to behave the same way, so things like the diffusion coefficient can as a function of each pores temperature using pn.pore_properties['temperature'], which would be a vector.

Numpy would broadcast the vectorized form so the code would be identical to the scalar form.

We need to start pinning down how "physics" are handled.

For illustration, consider capillary entry pressure. At the moment my script uses the pore diameter to get entry pressure from the Washburn equation. One can easily imagine about 20 different ways to calculate entry pressure (i.e. Purcell toroid, surface evolver, effective vs. inscribed diameter, incorporation of aspect ratio in the curvature, and so on), so the user might want to select a predefined one or create their own.

One approach is to create a module called physics (PHYS). Inside this module are classes for all of the possible physical properties we might want to add (i.e. entry pressure, hydraulic conductivity, and so on), and each class contains methods that output the desired value.

I can think of 3 ways to do this:

1. For instance, OpenPNM.PHYS.ThroatEntryPressure.Washburn(pn, theta, sigma) would generate pn.throat_properties['Pc_entry']. In this scenario, users could add methods to the ThroatEntryPressure class to suit their needs. We could 'define' or 'mandate' that the output from this class is always pn.throat_properties['Pc_entry'].
2. Alternatively, we might like just OpenPNM.PHYS.Washburn(...). This means that each method is treated as a class so gets its own file. This is more inline with the GEN module, but makes it harder for users to identify which methods are available for throat entry pressure calculations. They'd have to know that Purcell was a different version of Washburn.
3. Finally, we might opt for OpenPNM.PHYS.ThroatEntryPressure(pn, 'Washburn', sigma, theta). This is similar to the first option but Washburn is an argument. This would still require users to add their option to the main ThroatEntryPressure class, so is no better than 1.

I'm just pointing out that this problem is looming and throwing out some options. We also might need to think about a fluids class like Harold uses in his prototype GUI.

I am a newbie in python. When I run the 'pn = OpenPNM.GEN.Cubic(divisions=[10,10,10],lattice_spacing=.0005,loglevel=20,btype = [0,0,0]).generate()'

The result is that pn is nonetype. Could anyone tell me what happened?

I made a list of inlet pores using the following logic:
inlets = pn.pore_properties['numbering'][pn.pore_properties['coords'][:,2]<4](This is basically all pores on the left hand side of the image)

I then called the IP algorithm with this list of inlets as follows:
OpenPNM.ALG.InvasionPercolationAlgorithm(pn, npts=100, inlets=inlets, outlets=outlets).run()

I got some sort of broadcast error:
File "OpenPNM\ALG__InvasionPercolationAlgorithm__.py", line 190, in _setup_for_IP
self.plists[clusterNumber-1] = self.NewPore
ValueError: could not broadcast input array from shape (0) into shape (1)

When I called the algorithm with a inlets = [1] it worked fine, so the problems appears to be related to lists.

Pore and throat properties "type" seem like they should be int 32, but they are float 64.
Are there any ideas for non-integer values for the type attribute? Otherwise we should swap this into int.

James will remove the old IP algorithm and upgrade the one with timing written recently.

Find a way to change the network initialization steps so that one generator can be created then used to spit out multiple 'realizations' with the same network parameters.

I pondered this last night and had a vision about how the Geometry module should work now that we're giving it more power. This combines issues #31, #37, (which I've closed) and in some ways also includes #38, which is the new thread about the 'self-protecting dictionary'.

Basically, how the Geometry should work (IMHO) is as follows:

params = { some long list of parameters for network generation }
pn1 = OpenPNM.Geometry.Cubic().generate(**params)
pn2 = OpenPNM.Geometry.Cubic().generate(**params)

(Note that we now send params to Cubic() which is a bit confusing given the new role of Geometry)

Once the networks are in memory, we may need to manipulate their geometry as follows:

OpenPNM.Geometry.Cubic().translate(pn1,displacement)

This should change pn1 directly, rather than generate a whole new network since network generation may be time consuming (and memory intensive).

In term of network stitching the following would be good:

OpenPNM.Geometry.Cubic().stitch(pn1,pn2)

Which would stitch pn2 ONTO pn1, rather than creating a pn3.

The first bug that needs to be tested with the new test folder protocol we started today is getting the test scripts to run! :-) The first line in these files is:

import OpenPNM

But this fails since it can't do the import. A quick Stackoverflow search suggests that this is a very difficult problem to fix, but maybe @RodericDay has some ideas?

In keeping with the spirit of the test suite I have added a file called test1.py that breaks.

Hi all,
Please be sure to stress test Cubic before committing it to develop.
Nobody's scripts work right now due to a bunch of little issues. I'm working on it, and should have it ready in an hour or so (by 4pm Eastern Oct 13, 2013).

make a toggle to turn off the Haine's jump logic

try the following:

import OpenPNM
import scipy as sp
pn = OpenPNM.GEN.Cubic(domain_size=[10,10,10],lattice_spacing=1,btype = [0,1,0]).generate()
sp.unique(pn.pore_properties['type'])

I get:
array([ 0.])
as a result.

At the moment, we have adjacency and incidence matrices stored as pn._adacency and pn._incidence. These are further delineated by _csr, _coo, etc to tell us the storage scheme.

The problem is that we have no idea what information is stored there! Is it 1's and 0's, is it throat numbers or what? When OrdinaryPercolation runs, it overwrites pn._adjacency_csr with 1's and 0's according to which pores are filled. When someone calls get_neighbor_pores, it doesn't check the status of this matrix.

Proposed fix: we should make an adjacency and incidence matrix dictionary with the same name as the info stored there, such as:

pn.adjmat_csr['numbering']
pn.incmat_coo['type']

pn.print_overview is trying to access the .dtype and .shape values of all condition values, even if they are just scalars.

A fix for this would be to force all pore/throat_conditions to be array scalars

I don't know how to do this. Probably something similar to what we were planning with the pore/throat_properties.

@putza @RodericDay any ideas?

[I'm going through the documentation, and debugging as I go.]

First glitch: Notice how the pore properties of pn1 change when pn2 is created:

import OpenPNM
gn1 = OpenPNM.Geometry.Cubic()
pn1 = gn1.generate()
print pn1.pore_properties['diameter'][0]
pn2 = gn1.generate()
print pn1.pore_properties['diameter'][0]
print pn2.pore_properties['diameter'][0]

yields:

3.55850909492e-05
1.27589228176e-05
1.27589228176e-05

It seems we need the generate() command of Cubic to return some sort of "deep copy" of self._net but I couldn't figure out how to do it. @putza do you know how to do this?

It is impossible to type a multiple digit number into a spinbox. I think the default object for this type of input should be a text box, so people can enter precise values

My SourceTree graph (see attached) is showing that I somehow merged master into develop. I actually do not see any files that have been affected, other than .gitignore. But it's a worrysome limb of the tree that I've tried to "undo" but received this message:

git -c diff.mnemonicprefix=false -c core.quotepath=false revert --no-edit 6ed7992
error: Commit 6ed7992 is a merge but no -m option was given.
fatal: revert failed

@putza I currently use OpenPNMbase as the parent class of all new classes, specifically for the loglevel.

However, I don't like that all of OpenPNMbase's functions (ie declare_parameters, IOpickle, IOunpickle, Sandbox_doc, set_loglevel) clutter up my new class's popup menu.

Can these be hidden eg _declare_parameters ?

Soon, I'll be bold enough to make these changes on my own.

Andreas, can you log into the openpnm server and set it up so that:

It reads the latest git repo every x minutes
Runs Sphinx every x minutes
Put a link in the var/www/openpnm folder to the Sphinx /_build folder?

This way the docs are up to date and on the site. I guess this will always be the dev docs. We will want make static folders for specific versions as we go to paste in the html files.

Also, i have been having trouble building the docs on my machine, so we could make sure the server is setup right and then just use that as our default doc generator.

• Documentation coming along.
  • Some issues with tables not rendering and such. Building on Andreas Linux box didn't help.
  • Andreas will setup the webserver to generate docs and place in a web-readable folder, JG will add links to the Wordpress sites
  • We will keep the 'matplotlib' style documentation rather than going to the more stark 'numpy' style. JG will clean up the main index page to be a bit less 'website like'
• Harold is done a first version of his elegant dictionary
  Discussion:
  • set_item will reject arrays of the wrong size
  • this is a subclass of the python dict, so we need to import it
  • alter the dict class to prevent sp.ones(10,1), but allow sp.ones(10,) and sp.ones(10,2+)
  • Let's move towards the dictionary protecting itself, then removing all set and get methods
• Rename Base to Utilities
  • Is pickle needed?
  • Can we use csv files to output/save a pore network?
  • We'll think about writing a csv writer and reader, maybe this could be housed in JH's io class.
  • A csv writing method for algorithm outputs. Put a save_csv(filename) method in OpenPNMBase, but don't implement it there. All the child classes will inherit it and subclass it as needed. (Scipy.savetxt)
  • AP demo'd some nice sphinx live generated doc with running code, he will investigate this for future use of generating 'reports' of each run.
• Added a test folder for placing test script to see if things work/break
• JG will impliment the adjacency matrix fix (use dictionaries)
• JG will rename generators to geometry

The InvasionPercolationAlgorithm file contains two version of IP. We are now using the new 'timed' version by @jhinebau. We should remove the old one from the file, change it's name to InvasionPercolation, and allow for the "timed" feature to be optional, perhaps by setting flow = inf, or some flag. I'm assigning this to @jhinebau since he'll know best how to make the timing optional.

James' Questions:

1. Who's dgupta? He/she seems legit.
2. Will IO functions ever need a specific home? If so keep it. Kill it if we can't think of a good example.
3. What classes currently have near-finished documentation that we can use for templates?
4. What .rst files should we all work on before 1.0 release?
5. Can minutes be recorded in "Issue" format for James' benefit? (6-month internship starting next week)

Does it make sense to add a kind of 'blank' step in the generic generator for people to fill up with their own custom methods? For instance, the generate() step would be:

self.generate_pores()
self.generate_throats()
...
self.generate_misc()

and the misc could collect all user defined functions as follows:

def misc()
some code and maybe calls to other custom generate methods

If the user does not need any extra steps then the misc() method in the GenericGenerator would be empty.

As I write the documentation I find myself thinking of this from the users perspective. I have found numerous little things like this.

@putza has started to cleanup the documentation, but this issue is getting on urgent now so I'm elevating it to a "task". We all need to start writing the user documentation, but it's not clear where and how to begin this. Can we have a discussion on this during the next telecon? I would also like to see a quick tutorial on how to get the most out of rst files.

Andreas has complained on numerous occasions about sp appearing among the OpenPNM methods in the Spyder popup. What if we did:

import scipy as _sp

This would make scipy a private member of all classes and it would appear lower down the autogenerated popup list.

This is nitpicky: The description of Adjacency Matrices in
http://openpnm.org/src/docs/_build/html/userguide/data_storage.html#adjacency-matrices
seems too in-depth for this part of the documentation. I imagine folks reading this page top to bottom are going to give up during this section.

I'm going to mess around with it. Feel free to undo my changes if you feel strongly about it.

@jgostick close this if it's ok.

I just realized the purpose of using the underscore. It's so that when you run:

IP = OpenPNM.ALG.InvasionPercolationAlgorithm(net=pn,inlets=inlets,outlets=outlets)
IP.run()
IP.[popup]

the IP functions in the popup list don't get overwhelmed by all the variables created and attached to self throughout the algorithm.

Hey folks, did we decide to use SI units for everything? This is causing some problems with pore and throat volumes being something like 10^-15 cubic meters, not that cubic microns makes any more sense. The problem will be solved by never using float32. It's currently being used when writing vtk files.

There are some large 3D images and other files that we had in SVN version control, and only removed AFTER migrating to Git. Git stores all versions of all file in blobs, so deleting a big file only removes it from subsequent commits, but its still there. Cloning our repo is thus much slower than it should be.

Finding them and removing them requires someone with serious Git skills, so this issue may be open for a while.

1 - Go Over Issues

Issue #41 - import scipy as _sp? We could do a mass search and replace, and also remove np while we're at it.

Issue #33 - The GUI can write VTKs using the new Visualization module, but we still need to think of some way to get data 'out' of the GUI.

Issue #39 - JTG made a lot of changes to the Geometry module. The main reason was to get the generation params out of the Geometry class call, and sent to the .generate() call instead. (See issue #39 for history of JTGs thoughts on this process). Are we all happy with these?

Issue #20 - Windows machines don't have Graphviz, so inheritance diagrams don't work. Extra incentive to get this working.

Issue #19 - SandboxDoc is gone, but other issues remain

2 - Development Discussions

• The use of staticmethods, are we overdoing objects?
• Self-protecting dictionary
• Constants dictionary
• Thermofluid properties: vector vs scalar pore_properties?
• @maghighi and JTG did a big reno of the Physic folder...BUT maybe we should revert it back to a class-less form? (See feature/PhysicsClassToModule)
• Algorithms...how should parameters be passed? What is the meaning of an algorithm object? Should we remove the explicit .run()?

3 - Documentation Updates

Lesson: please explain the correct usage of library, module, object, class, function, method, etc.

Proposal 1: All physics methods are fully documented in the docstring. For an example see OpenPNM.Physics.CapillaryPressure.Purcell()

Proposal 2: Agree to use Numpydoc style, which means we can't make up headings

Nothing has been done to the 'developer' guide yet

JTG has created an examples folder under docs, and started creating example scripts.

4 - Transport Solver Updates and Discussions

Mahmoud's work

5 - Status of the GUI

At some point we should swap out all the numpy and np in our code for scipy and sp and then forget about numpy. Is there any reason we shouldnt?

merge the two VTK writing classes and finish scaling_factor documentation and functionality

Now that we plan to use methods stored in Geometry to alter the topology and properties of networks, we need to be able to pass existing networks back to the Geometry class.

I have added to very simple functions to GenericGeometry (translate_coordinates and scale_coordinates) that expect a network object and displacement or scale vector as arguments. This fails for technical reasons beyond my skill set.

This issues pulls together #11 and #16, so we can close those, and focus developing the features we want in this 'self-protecting dictionary' class.

The dictionary should:

• Prevent writing arrays of the wrong length, so all arrays are the same size.
• Prevent arrays of (N,1), but all (N,) and (N,2 or higher)
• Somehow make sure the data type is suitable
• Prevent overwriting of existing data by accident

Please add to this list as you see fit...

The IP algorithm can take a long time to run, and the only way to know if the algorithm is actually running (and not frozen) is if the loglevel is set to debug. I realize that the end point of this algorithm is not known ahead of time, but it would be nice if there was some indication that the algorithm is in progress.

all clickables in http://openpnm.org/?page_id=56 just give me a spinning wheel loading icon.

Firefox v.23.0.1 on Windows 7

I think the IO folder is redundant. I think that the VTK functions should be moved to Visualization, and the network import function should be in Generators. Generating a network object by importing the data is still generating. And VTK under visualization seems obvious.

I think @jhinebau should do this since these are both his functions.

When setting the loglevel in calls to an algorithm the following error is received:

File "OpenPNM\ALG__OrdinaryPercolationAlgorithm__.py", line 48, in init
self._setup(**kwargs)
TypeError: _setup() got an unexpected keyword argument 'loglevel'

The Generators accept the loglevel argument without complaint.

Hi all,
The pore property 'coords' is missing from boundary pores in networks generated from Cubic because that block of code has been commented out.

We need coordinates on all pores for network consistency. Additionally, the Paraview visualization breaks when pore properties are of different lengths.

This is the second time I'm having to resuscitate this chunk of code. Please don't just comment it out and move on. If it breaks, and you need to comment it out temporarily, don't merge your feature into the develop branch until it's unbroken.

Thanks,
James

When running run_script.py, I get this:

Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 523, in runfile
execfile(filename, namespace)
File "D:\AFCC code\GitHub projects\OpenPNM\run_script.py", line 41, in
pn.throat_properties['Pc_entry'] = OpenPNM.Physics.CapillaryPressure().Washburn(pn,0.072,110)
TypeError: 'module' object is not callable

It has been discussed a few times about enforcing (or encouraging) the use of custom methods to add or edit property data. I wanted to compile the main reasons why we should be more strict about this:

• Every pore/throat list added could be forced to be the correct length.
• As I write the documentation, I think it would be helpful if we had a list of all "predefined' pore and throat properties. At some point during the network generation step we should explicitly create these, then they can be documented and enshrined.
• As per @jhinebau's issue #11 about data types, we could also define which data type each property should be.
• The code responsible for automatic generation of each property could be accompanied by a docstring that would explain the purpose of the property.

This is a temporary fix so that we can at least get some data out of the GUI

The individual cluster growth rates should be user specified, in a list of floating point numbers.

I think the Generator modules will soon start to have more than just generation functions. Debarun is coding a 'stitch' function which will also require 'translate', 'rotate' , 'flip' and so on. These are really more about geometry manipulation, so Generators is becoming a misnomer. I also need some special set_property functions for the Custom networks that aren't technically generation.

Therefore, I propose to rename the Generators module to Geometry.

If we make this change, it will also mean the code will be more 'readable':

pn = OpenPNM.Geometry.Cubic().generate()

As opposed to the redundant:

pn = OpenPNM.Generators.Cubic().generate()

(Note that use used to have the word generate 3 times, and now its only 2)

The other Geometry functions would be accessed with:

pn = OpenPNM.Geometry.Cubic.translate(displacement=[-10,4,5])

or

pn = OpenPNM.Geometry.Cubic.stitch(pn1,pn2)