ancir / grano Goto Github PK

Suppose I'm working with this schema:

- name: fellow
  label: An OpenNews fellow
  obj: entity
  hidden: no
  attributes:
    - name: twitter_handle
      label: Twitter handle

- name: news_organization
  label: A news organization
  obj: entity
  hidden: no
  attributes:
    - name: url
      label: URL

- name: fellowship
  label: A Fellowship
  label_in: Was hosted by
  label_out: Worked for
  obj: relation
  attributes:
    - name: start_date
      label: Start date
    - name: end_date
      label: End date

That defines a bipartite graph (two clases of vertices). How would I use grano-client to get all the "fellows" or the "news organizations"? In other words, is it possible to get the schema that a given properties was defined in?

Now at: https://github.com/pudo/grano-opencorporates

Currently, only "name" is exported - this should be configurable, both for entities and relations.

IMO, it feels a little bit too specific.

How embarrassing.

Something along these lines:

- name: organizacion
  label: Una organizacion
  obj: entity
  hidden: no
  attributes:
    - name: nombre
      label: Nombre
      required: true
    - name: date
      label: Date
      pattern: \d\d\d\d-\d\d-\d\d

Gremlin looks like a great resource to draw inspiration from.

More relevant links:

• Bulbs — A Python framework for graph databases (big overlap with grano)
• sql2gremlin — Gremlin tutorial
• SQL adapter for Blueprints

Idea: implement a subset of Gremlin. Doesn't look that hard.

You want to facet entities by:

• Properties
• Schemata
• Relations (#), schema
• Status

And relations by:

• Properties
• Schema
• Source
• Target

Query syntax

How would you query these different facets?

• Properties: ?facet=properties.name
• Schemata: ?facet=schema
• Status: ?facet=status

More complex (i.e. nested) facets:

• Relations: ?facet=inbound.count, ?facet=outbound.count or ?facet==relations.count
• Nested: ?facet=inbound.schema, ?facet=inbound.properties.started_at

To load in data from scratch without losing schema/permissions.

Devise a standard way of filtering the basic and advanced attributes of all index REST views.

1. The Alembic migration tries to create the relation table first, but it depends (FK) on schema and entity. Temporarily fixed moving the op.create_table('relation', ...) method call to the bottom of the file

2. Alembic (or SQLAlchemy) generates faulty SQL:

   CREATE TABLE account (
   created_at TIMESTAMP WITHOUT TIME ZONE,
   updated_at TIMESTAMP WITHOUT TIME ZONE,
   id INTEGER DEFAULT 'nextval('account_id_seq'::regclass)' NOT NULL,
   github_id INTEGER,
   login VARCHAR,
   email VARCHAR,
   api_key VARCHAR,
   CONSTRAINT account_pkey PRIMARY KEY (id)
   )

   'nextval('account_id_seq'::regclass)' as a string makes no sense —plus the obvious escaping issue—. Full traceback below.

   INFO  [alembic.migration] Context impl PostgresqlImpl.
   INFO  [alembic.migration] Will assume transactional DDL.
   INFO  [alembic.migration] Running upgrade None -> 4f21a77e91be, init
   Traceback (most recent call last):
   File "/Users/manuel/Work/pyenvs/grano/bin/alembic", line 11, in <module>
     sys.exit(main())
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/config.py", line 298, in main
     CommandLine(prog=prog).main(argv=argv)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/config.py", line 293, in main
     self.run_cmd(cfg, options)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/config.py", line 279, in run_cmd
     **dict((k, getattr(options, k)) for k in kwarg)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/command.py", line 124, in upgrade
     script.run_env()
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/script.py", line 199, in run_env
     util.load_python_file(self.dir, 'env.py')
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/util.py", line 205, in load_python_file
     module = load_module_py(module_id, path)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/compat.py", line 58, in load_module_py
     mod = imp.load_source(module_id, path, fp)
   File "alembic/env.py", line 75, in <module>
     run_migrations_online()
   File "alembic/env.py", line 68, in run_migrations_online
     context.run_migrations()
   File "<string>", line 7, in run_migrations
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/environment.py", line 681, in run_migrations
     self.get_context().run_migrations(**kw)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/migration.py", line 225, in run_migrations
     change(**kw)
   File "alembic/versions/4f21a77e91be_init.py", line 26, in upgrade
     sa.PrimaryKeyConstraint('id', name=u'account_pkey')
   File "<string>", line 7, in create_table
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/operations.py", line 647, in create_table
     self._table(name, *columns, **kw)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/ddl/impl.py", line 149, in create_table
     self._exec(schema.CreateTable(table))
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/ddl/impl.py", line 76, in _exec
     conn.execute(construct, *multiparams, **params)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 664, in execute
     return meth(self, multiparams, params)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 67, in _execute_on_connection
     return connection._execute_ddl(self, multiparams, params)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 718, in _execute_ddl
     compiled
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
     context)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1023, in _handle_dbapi_exception
     exc_info
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
     reraise(type(exception), exception, tb=exc_tb)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
     context)
   File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 388, in do_execute
     cursor.execute(statement, parameters)
   sqlalchemy.exc.ProgrammingError: (ProgrammingError) syntax error at or near "account_id_seq"
   LINE 5:  id INTEGER DEFAULT 'nextval('account_id_seq'::regclass)' NO...
   

So we can build an egg that contains the compiled assets.

If you have a numeric primary key in your original dataset, it's perfect for uniqueness but inconvenient in the grano admin for identifying entites.

Mainly: degree, some measure of betweenness centrality

Questions:

• Where does this get computed? Where does it get cached?

Can re-use sections of the nomenklatura API as they follow the same patterns.

Basic version is ready here: http://grano.pudo.org/rest_api.html

Given a core node, export a set of adjacent nodes (with a given recursive depth) via the API as JSON (or GEXF).

Wikidata seems to have three ranks of data. This seems useful to rate user-provided property values over scraper-generated input. In effect, the following data quality ranks should be considered:

• Known to be false / discredited (score 10)
• Information submitted by uncredited user (score 20)
• Information acquired by an automatic process (score 30)
• Information validated by an trusted user / editor (score 40)
• Information with court-level evidence (score 50)

The question now is: does this apply to properties only, or also to entities and relations directly? If it does apply to the main objects, would that mean a new column on the relevant tables?

Copy it over from nomenklatura :)

Wrote a simple loader using the Loader API. It wants to connect to an ampq queue.

For small-ish data sets, installing a queue and running a job broker, might not be worth the hassle.

Could be done based on #14, i.e. editing an entity or relation triggers a background daemon, which, in turn, generates a static fragment.

See this post by one of the creators of the JSON-LD standard.

Why do I think it makes sense to switch to JSON-LD?

• It's a perfect match: A @context is a schema, period.
• It's still JSON, no need to deal with crazy triples, turtles and sparqls
• It's a good compromise between pragmatism and ivory-tower RDF. If you want RDF you can transform it

Support for adding images/media as a property. Since we already have a File type, this should somehow be a reference rather than an actual upload.

• /api/1/entities/<id>/properties
• /api/1/entities/<id>/properties/<name>

I noticed that properties are stored as a db.Unicode value, regardless of their type.

Is this by design?

(just so you remember)

• create schema_property table
• remove name from property
• add a FK from property to schema_property

The loader API should be stable and available externally.

I'm currently integrating grano with another application that already uses a DB. The grano tables have pretty generic names, might make sense to prefix them with grano_.

When running a loader script:

WARNING:grano.logic.entities:Processing change in entity: ea38762f90cdaf448
DEBUG:stevedore.extension:found extension EntryPoint.parse('indexer = grano.service.indexer:AutoIndexer')
ERROR:stevedore.extension:Could not load 'indexer': No module named service.indexer
ERROR:stevedore.extension:No module named service.indexer
Traceback (most recent call last):
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/stevedore/extension.py", line 162, in _load_plugins
    verify_requirements,
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/stevedore/enabled.py", line 66, in _load_one_plugin
    verify_requirements,
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/stevedore/extension.py", line 177, in _load_one_plugin
    plugin = ep.load(require=verify_requirements)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/pkg_resources.py", line 2021, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named service.indexer
WARNING:grano.logic.entities:Processing change in entity: 1b16485bddec93d23
Traceback (most recent call last):
  File "loader.py", line 29, in <module>
    fellowship = fellowship.set('end_date', record[3])
AttributeError: 'NoneType' object has no attribute 'set'

Probably also add an option to the loader API to raise warnings on load instead of just logging.

It would look like this:

    - name: person_id_type
      label: ID Type
      datatype: ['foo', 'bar', 'quux']::tuple

Not sure about the possible values syntax. What do you think?

Also have a proper display name field.

Import basic matching logic from nomenklatura.

The graph export currently generates a multigraph, there needs to be an option to aggregate multiople relations into one weighted link.

It'd be nice to be able to do something like this, instead of having to create a separate file for each entity or relation type.

- name: person
  label: A person
  obj: entity
  hidden: no
  attributes:
    - name: first_name
      label: First name
    - name: last_name
      label: Last name
    - name: title
      label: Title
      hidden: yes

- name: gift
  label: A present
  label_in: Gifts received
  label_out: Gifts given
  obj: relation
  attributes:
    - name: description
      label: Description
    - name: value
      label: Value (in EUR)

Hopefully based on pynomenklatura.

Have a simple mapping format, e.g.:

source_url: http://...
mapping:
  - column: person_a
    element: source
    schema: base
    attribute: name
  - column: person_b
    element: target
    schema: base
    attribute: name
  - column: relation_description
    element: relation
    schema: my_schema
    attribute: description

Sub-tasks:

• pipeline model for data process logging
• implement an importer
• make a command-line tool
• document the mapping format.

This is more precise, should be used in the UI as well as in the API.

Fragment is another type of relation which can link to any number of entities through a piece of marked-up text. The reason for implementing this is to combine narrative elements into the database, so that you can capture non-structured parts of a story in a way that is easy to understand for readers.

The metaphor for this is the Facebook timeline, where each update would be shown on the feeds for all entities mentioned within. Subscriptions would help users make their own news feeds with the updates of all entities to which they are subscribed.

A domain model could look like this:

• fragment: has text, start_date, end_date, account_id, source_url.
• fragment_entity: has fragment_id, entity_id, text_start, text_end.
• entity_subscription: has account_id, entity_id, since_date (used to subscribe users to an entity's news feed - but do we want all subscribers to have to be users?)

There's a mockup of what the finished function could look like here: http://opendatalabs.org/misc/demo/grano/_mockup/

Import schema.org.

• Have a Permission table
• Add authorization to project and schema APIs
• Filter entity and relation APIs.

grano will need some sort of front-end. What functions does it have to support?

• Emergency editing of entities and relations
• Verifying proposed changes (needs proposed changes first)
• Importing and mapping CSVs to entities and relations
• WYSIWYG schema editor

Optional/Nice-to-have:

• Merging entities / Deduplication
• Default rendering for entities and relations (tables?)
• OpenCorporates recon integrator

Followed the instructions, got an exception when running alembic upgrade head:

INFO  [alembic.migration] Context impl SQLiteImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running upgrade None -> 4f21a77e91be, init
Traceback (most recent call last):
  File "/Users/manuel/Work/pyenvs/grano/bin/alembic", line 11, in <module>
    sys.exit(main())
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/config.py", line 298, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/config.py", line 293, in main
    self.run_cmd(cfg, options)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/config.py", line 279, in run_cmd
    **dict((k, getattr(options, k)) for k in kwarg)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/command.py", line 124, in upgrade
    script.run_env()
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/script.py", line 199, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/util.py", line 205, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "alembic/env.py", line 75, in <module>
    run_migrations_online()
  File "alembic/env.py", line 68, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/environment.py", line 681, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/migration.py", line 225, in run_migrations
    change(**kw)
  File "alembic/versions/4f21a77e91be_init.py", line 38, in upgrade
    sa.PrimaryKeyConstraint('id', name=u'account_pkey')
  File "<string>", line 7, in create_table
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/operations.py", line 647, in create_table
    self._table(name, *columns, **kw)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/ddl/impl.py", line 149, in create_table
    self._exec(schema.CreateTable(table))
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/alembic/ddl/impl.py", line 76, in _exec
    conn.execute(construct, *multiparams, **params)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 664, in execute
    return meth(self, multiparams, params)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 67, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 718, in _execute_ddl
    compiled
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
    context)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1023, in _handle_dbapi_exception
    exc_info
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
    context)
  File "/Users/manuel/Work/pyenvs/grano/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 388, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (OperationalError) near "account_id_seq": syntax error u"\nCREATE TABLE account (\n\tcreated_at TIMESTAMP, \n\tupdated_at TIMESTAMP, \n\tid INTEGER DEFAULT 'nextval('account_id_seq'::regclass)' NOT NULL, \n\tgithub_id INTEGER, \n\tlogin VARCHAR, \n\temail VARCHAR, \n\tapi_key VARCHAR, \n\tCONSTRAINT account_pkey PRIMARY KEY (id)\n)\n\n" ()

Here's my settings.py:

import os

DEBUG = True
ASSETS_DEBUG = False

SECRET_KEY = 'test'
SQLALCHEMY_DATABASE_URI = 'sqlite:////tmp/grano.db'
APP_NAME = CELERY_APP_NAME = ES_INDEX = 'grano'
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672//'

GITHUB_CLIENT_ID = 'da79a6b5868e690ab984'
GITHUB_CLIENT_SECRET = '1701d3bd20bbb29012592fd3a9c64b827e0682d6'

# Generate a public URI for an entity, based on its ID.
ENTITY_VIEW_PATTERN = 'http://grano.io/e/%s'