The code produces the same op file as the input file

"""
Region, Country,State
Americas, United States AA
Americas, United States AE
Americas, United States AK
Americas, United States AL
Americas, Canada, 00
Americas, Canada, AB
Americas, Canada, BC
Americas, Canada, MB
Americas, Canada, NB
Australia, Australia, NSW
Australia, Australia, VIC
Australia, Australia, SA
Europe, France, A1
Europe, France, A2
Europe, France, A3
"""

with open('test_file.csv', 'w') as f:
f.writelines("""Region , Country , State ,
Americas , United States , AA ,
, , AE ,
, , AK ,
, Canada , AB ,
, , BC ,
Europe , France , A1 ,
, , A2 ,
, , A3 ,
""")

problem is with cls_file_mapping.py referring to sub data folder not included in package - may need to move \data underneath \aikif so that it is installed - check for sizes first

import aikif.programs
Traceback (most recent call last):
File "", line 1, in
File "C:\Python34\lib\site-packages\aikif\programs.py", line 11, in
import aikif.cls_file_mapping as mod_filemap
File "C:\Python34\lib\site-packages\aikif\cls_file_mapping.py", line 21, in
with open(file_type_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Python34\lib\site
-packages\data\ref\ONTOLOGY_FILE_TYPE.txt'

the current cls_goal is very basic and uses the plans as lists. The cls_plan will manage the list of plans used by the goal classes. (see AI planning course notes)

The goals should be fairly textual though classified and the purpose of the plan class is to allow mapping of these goals to actual implementable classes (ie toolbox, dataTools).

Needs to be a new tab on the web interface that manages the toolbox.

This needs to include descriptive information on the tool , where it lives, params passed and expected pass/fail return value ranges.

Would also be nice to link it to an ontology to allow it to be categorised, though this may be tricky in a web interface - perhaps multiple tags is a reasonable start

Using it from a command line would be something like

tools = aikif.toolbox.Toolbox()
tools.add('email collection', method=agent.gather.agent_email, interval='Daily')
tools.add('file download', method=toolbox.download, interval='On call')
tools.add('file copy', cls_file.copy)

tools.add(‘script1’, PC=’linux_main’, method=’/usr/bin/script1’, interval=’Daily’, log=’script1.log’)
tools.add(‘script2’, PC=’ent_win8’, method=’C:\\script\run2.BAT’’, interval=’Daily’, log=’script2.log’)
tools.add(‘script3’, PC=’linux_NAS’, method=’/user/home/backup’, interval=’Daily’, log=’script3.log’)

So the information to be kept is
Tool Schedule

• a schedule to run a set of tools (time, date, [tools] )

Tools

• tool name
• location
• description
• params []
• return values []
• tags
• capabilities (link to standard AIKIF tool processes)
• type (move, copy, delete, assign, search, save, validate)

the STRING2 datatype in the random_table function needs to be implemented so that it takes a number with strings being generated of that length

The core of AIKIF is to be able to easily see what processes have run and what results occurred.

This can be Implemented via a core log/report class (not sure what name yet) which all modules call for all logging and recording of progress.

The purpose of the module is to record the details, but primarily to aggregate at various levels to enable simple, english readable summaries to be seen.

• complete class to include logging (see orig requirements notes)
• add example using data flow (dl > CSV > data_table)
• add example using maths computation (hard code source > algorithm > log result

use project in a real data example

The point of the web interface is to see how things are running, and to make sense of the logs, but the current (prototype) display is not doing this.

Projects

• implement local storage of projects list
• test with external program (LifePIM or VAIS)
• change web interface to show projects with status, last update time, key tasks (human and AI)

Agents

• use the existing local store of agents to display in web interface
• change to allow starting and stopping of agents via web

need to implement a test to make sure the run function correctly works in all cases - eg python program with no function, program with multiple functions, different arguments.

Make sure it fails if incorrect args passed or return type expected

*update setup.py *
include new packages
aikif.agents.learn
aikif.examples

increment version number from 0.0.0 to 0.0.4

README.md

• version number
• progress (date updated)
• Quick start to actually show a quick start (actual code to run, not just notes)
• delete data structures section, replace with diagram
• delete standard library programs section

Rest of document to show code and results of running things, eg a single notebook running a few examples

Delete either manifest or manifest.in (one is redundant)

CHANGES.txt
v0.0.4, 15/1/2015 -- examples, additional agents, mappers

• add examples folder:
  [list each example and one line what it does]
• add mapper
• agents : learn has demo for pyaixi, new gather agent for Outlook and gmail

So use projects.add (project)
Instead of separate objects

Goal - single example showing management of several actual projects. This will show how it can be used and what benefits are.

Projects to manage

AIKIF
rawdata
virtual-AI-simulator
Linux backup
Windows backup
file metadata collection

Data collections to manage

File metadata
Image metadata
ebooks
project keywords and links
Ontology downloads
All CSV / YAML reference files used in AIKIF production

line 93 reads = self.receiveserver.login(username,password)
should be = self.receiveserver.login(self.username,self.password)

shouldn't be a root script, or if it is then move it to scripts.

Better to make it a callable tool that can be used as needed rather than a one off thing that indexes everything

EDIT: only appears to be an issue on Windows with Python 3.5.

Issue with the noise package.

Think now that the world generator should be moved to the separate package as originally intended.

filelist generation works ok, but there are issues with the unicode filenames.

Personally they are a massive pain in the neck and would love to just rename the damn folder to make them clean, but it needs to be done properly (also to allow printing to screen)

https://www.quantifiedcode.com/app/project/9919962e4ba8448a8a74d9a6381102dd

Global statistics
Indicator Value
Lines of code 19513
Number of functions 1325
Number of classes 158
Total cyclomatic complexity 2584

Potential bug 22
Minor issue 202
Recommendation 54

Potential Bugs

Avoid accessing protected class members from outside

aikif/toolbox/image_tools.py : 21

Empty exception handler

tests/test_toolbox_network_tools.py : 25 [done]
aikif/programs.py : 116 [done]
aikif/lib/cls_file.py : 50, 83 [done]
aikif/index.py : 77 [done]
aikif/toolbox/image_tools.py : 215, [done]

aikif/ontology/createMindOntology.py : 83, 123 [moved to prototype]
aikif/lib/cls_filelist.py : 109
aikif/dataTools/cls_datatable.py : 264
tests/run_tests.py : 19

aikif/web_app/web_aikif.py : 162

aikif/toolbox/Toolbox.py : 47
aikif/lib/cls_context.py : 54

Following on from issue #20 Semantic Web, decided not to link to external ontology but use local one initially.

The original hard coded dictionary in cls_file_mapping.py needs to be in an external text file.

Should probably also rename this file, maybe ontology.py?? but keep existing methods

New methods need to include

• self test = make sure no duplicates exist in the file imported
• show_potential_collisions = find partial strings that may collide - e.g. small words in a larger word. You will need to implement this properly with ID's anyway though this test is needed as there will always be a requirement to 'guess' a link. (NO - simply return all matches in this case)

Resolved this after a lot of stuffing around. Basically all tests pass on Windows and travis-CI but on a fresh install of Ubuntu 14.04 LTS Pillow failed tests.

Read the docs says:
http://pillow.readthedocs.org/installation.html

sudo apt-get install python3-dev python3-setuptools

Prerequisites are installed on Ubuntu 14.04 LTS with:

$ sudo apt-get install libtiff5-dev libjpeg8-dev zlib1g-dev
libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk

Also tried (thanks to helpful folks at IRC python

apt-get build-dep

Finally worked in virtualenv only:

virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
pip install Pillow

This is the download web file function - build it as a class in toolbox with a proper test function

Also need to replace the existing hack code in various downloadweb calls in the project

uses old folder structure, and only does tests for CSV files.

Need to implement several tests for different database enviroments (MySQL, CSV, Json, Oracle, MSSqlServer) - not all now, just a couple to get the database functionality and sub classing tested properly, eg everything should come from the Data class

There are too many required data files to leave out of pip install, such as the bias ratings and ontology files.

git mv /data/. /aikif/data/.

Then modify the (many) root_folder definitions so that they point to the correct place

NOTE - leave very large files such as external datasets in the root folder (not installed via pip) but call it something else like /external

currently the filemapping module is a collection of functions which needs more structure.

Build into a class
modify the test case to include more tests
Incorporate ontology parent lookup (walks up tree) until matches found. not sure exactly how this will or should fit mixing created top ontology and opencyc, but opencyc would be preferable as long as the filename generation is not too complex and dirvese.

maybe try unique ids as filenames and have a filename lookup

This should allow for project based logfiles - ie separate set of logs for each application running (ie agent_Explore_command.log, dataTools_csv.log

Programming Language :: Python :: 3

Purpose:
determine is happening to allow for single line log entries. useful to testing aikif, generating timesheets, showing list of likely documents / links to be used next and possible caching

cls_context:
ususally aub classed this determines via estimated guess what the agent/user/pc is doing and what it is likely to do next.

cls_context_agent
likely to be simply the agents known environment returned

cls_pc_usage
what the current usage on the PC is focused on - games, dev, documenting, reddit

cls_context_physical
will only work if on smartphone with gps or with external sensors,but reads where you are and relates to prior movements to estimate destination

in generate_test_data there are hard coded windows paths

aikif doesnt install without sudo - fix it

Building using 4 processes
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.4/_imagingmorph.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-3.4/PIL/_imagingmorph.cpython-34m.so
--------------------------------------------------------------------
PIL SETUP SUMMARY
--------------------------------------------------------------------
version      Pillow 2.9.0
platform     linux 3.4.0 (default, Jun 19 2015, 14:20:21)
             [GCC 4.8.2]
--------------------------------------------------------------------
*** TKINTER support not available
*** JPEG support not available
*** OPENJPEG (JPEG2000) support not available
*** ZLIB (PNG/ZIP) support not available
*** LIBTIFF support not available
*** FREETYPE2 support not available
*** LITTLECMS2 support not available
*** WEBP support not available
*** WEBPMUX support not available
--------------------------------------------------------------------
To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.

To check the build, run the selftest.py script.

changing mode of build/scripts-3.4/pilfile.py from 644 to 755
changing mode of build/scripts-3.4/pilprint.py from 644 to 755
changing mode of build/scripts-3.4/pilfont.py from 644 to 755
changing mode of build/scripts-3.4/pildriver.py from 644 to 755
changing mode of build/scripts-3.4/pilconvert.py from 644 to 755
changing mode of /usr/local/bin/pilfile.py to 755
changing mode of /usr/local/bin/pilprint.py to 755
changing mode of /usr/local/bin/pilfont.py to 755
changing mode of /usr/local/bin/pildriver.py to 755
changing mode of /usr/local/bin/pilconvert.py to 755

Running setup.py install for xlrd
changing mode of build/scripts-3.4/runxlrd.py from 644 to 755

changing mode of /usr/local/bin/runxlrd.py to 755

Running setup.py install for pyyaml
checking if libyaml is compilable
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -c build/temp.linux-x86_64-3.4/check_libyaml.c -o build/temp.linux-x86_64-3.4/check_libyaml.o
build/temp.linux-x86_64-3.4/check_libyaml.c:2:18: fatal error: yaml.h: No such file or directory
#include <yaml.h>
^
compilation terminated.

libyaml is not found or a compiler error: forcing --without-libyaml
(if libyaml is installed correctly, you may need to
 specify the option --include-dirs or uncomment and
 modify the parameter include_dirs in setup.cfg)

Running setup.py install for psutil
building 'psutil._psutil_linux' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_VERSION=321 -I/usr/include/python3.4m -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-3.4/psutil/_psutil_linux.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.4/psutil/_psutil_linux.o -o build/lib.linux-x86_64-3.4/psutil/_psutil_linux.cpython-34m.so
building 'psutil._psutil_posix' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-3.4/psutil/_psutil_posix.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.4/psutil/_psutil_posix.o -o build/lib.linux-x86_64-3.4/psutil/_psutil_posix.cpython-34m.so

warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: manifest_maker: MANIFEST.in, line 18: 'recursive-include' expects <dir> <pattern1> <pattern2> ...

Running setup.py install for itsdangerous

warning: no previously-included files matching '*' found under directory 'docs/_build'

Running setup.py install for aniso8601

Successfully installed AIKIF nose flask flask-httpauth flask-restful requests beautifulsoup4 Pillow xlrd pyyaml psutil Werkzeug Jinja2 itsdangerous pytz aniso8601
Cleaning up...
duncan@TREEBEARD-LINUX:~$

Doesnt really belong here - has already been copied to VAIS, but might belong in WorldBuild

Move .z_prototype/crypt_utils.py to a toolbox method in aikif

needs a better way to clean the column headings

Also - check that folder names are copied properly and not missing first 2 characters of root folder name

Have moved most of the data collection routines to rawdata package, so need to redo the documentation to show decent use cases

aikif/toolbox/Toolbox.py

Line 74
1
Try to rename user-defined function. Overloading built-in functions is bad practice and should be avoided as much as possible.
self.lg.record_process(self._get_tool_str(tool))

def list(self):
    """
    Display the list of items 
    """

• found by quantifiedcode

aikif / toolbox / cls_grid.py needs to be based on an abstract class in environments which does logging such as
d/m/y , world started: grid
d/m/y , world ended: grid

Think about moving cls_grid to environment folder, though it may be better in \lib

Time to think about a bootstrap process, which even if it doesn't actually load all the data, we have
documented the way it loads

1. Self test environment and install of pacakges
2. Load all toolbox items and verifiy
3. Load all mappings and verifiy
4. Start API server
5. Load schedules (?? maybe use API instead of internal schedular)
6. Wait for schedule or triggers

missing self reference in the methods of class meaning variables (though there are none at the moment) will not be correctly referenced

Getting issues when testing on different environments, and cannot easily run pip install to get latest package, so update it. Below shows most of dataTools is missing (also rename to data_tools)

help(aikif.dataTools)
Help on package aikif.dataTools in aikif:

NAME
aikif.dataTools

PACKAGE CONTENTS
cls_data
cls_database
cls_dataset
cls_datatable
cls_mssqlserver
cls_oracle
cls_sql_code_generator
dataTools
form_example_simple
generateTestData

FILE
c:\python34\lib\site-packages\aikif\datatools__init__.py

Hi - I'd be interested in general and or specific feedback about this project so feel free to post here or email me [email protected]

Did you take a look at semantic web standards and tech? Most of your design goals are met by those standards and technologies.

make sample file from names_with_countries.csv (starting at line 48255) to get data to test with. Basically ALL read parts of this class needs to be bullet proof to allow any file type to be read.

Should also have a function identify which tells you what the encoding is, and how many issues etc if you read it in the wrong encoding.

In AIKIF/tests/test_api.py the local API should be started, sleep for a second then run the tests without any try catches (that was done as a fudge to get travis-ci passing.

See example of unit testing a webserver at https://github.com/web2py/web2py/blob/master/gluon/tests/test_web.py

Some examples (autobackup.py) are still using the incorrect log calls.

Ensure all examples are updated

Make a new test called run_examples.py which uses unittest to clear a log folder, run all examples, and test the resulting logs

document_AIKIF.py = include new programs
ex_index_mydocs.py = use config.py for folders
ex_project_management.py = uses project.py (in conjunction with text files from existing dicts)
finance_example.py = needs to be completed - loads but no mappers used.
game_of_life_console.py = needs new logging
puzzle_*.py = needs implementing (just headers at the moment)

When processing large files with dataTools/cls_datatable.py load_to_array
it crashes with an out of memory error.