basilapi / basil Goto Github PK

View Code? Open in Web Editor NEW

24.0 9.0 5.0 51.16 MB

Building Apis SImpLy from sparql endpoints

Home Page: http://basil.kmi.open.ac.uk

Java 18.49% HTML 0.04% CSS 3.35% JavaScript 78.05% Shell 0.07%

basil's Introduction

BASIL

BASIL is designed as middleware system that mediates between SPARQL endpoints and applications.

With BASIL you can build Web APIs on top of SPARQL endpoints.

BASIL stores SPARQL queries and builds APIs with standard and customizable formats.

Run

Latest runnable jar can be downloaded from the releases section.

TDB2 backend (default)

An example configuration file can be found in the project root folder (basil-tdb2.ini). An example log4j configuration file is at server/src/test/resources/log4j.xml.

Execute:

$ java -jar -Dbasil.configurationFile=./basil-tdb2.ini -Dlog4j.configurationFile=./log4j2.xml basil-server-VERSION.jar -p 8080
#1: welcome to the world's helthiest food
#2: basil is starting on port 8080
#3: done
#4: enjoy

MySQL backend (legacy)

You need to:

Have a MySQL server.
Prepare a database running the db.sql queries (at the root of the codebase).
Prepare the configuration file (the connection parameters), see this file as an example.
Prepare a log4j2 configuration file (if you want logging). See this file as an example.

When ready, execute:

$ java -jar -Dbasil.configurationFile=../basil.ini -Dlog4j.configurationFile=src/test/resources/log4j2.xml basil-server-0.3.0.jar -p 8080
#1: welcome to the world's helthiest food
#2: basil is starting on port 8080
#3: done
#4: enjoy

Usage

BASIL is a system based on a Web API for creating and managing a stack of Web APIs on top SPARQL endpoints. On its own, BASIL does not come with a graphic user interface. Instead, it can be fully controlled via its API. See the cURL tutorial for details.

Build

The basil project is managed and built with Maven.

mvn clean install

Note: to also run tests, you need an active internet connection (as they use public SPARQL endpoints). If you want to skip tests, you can:

mvn install -DskipTests

Releasing

The following command will pack a release, sign the artefacts, and push them to maven central.

mvn deploy -DperformRelease=true

basil's People

Contributors

Stargazers

Watchers

Forkers

kmitd gitter-badger enridaga justin2004 gro-intelligence

basil's Issues

Modularization of the project

Plan:

Parent pom - DONE
Server (single runnable jar) - DONE
Reactor (to build all together) - DONE
Rendering to host code around serialization - DONE.
Core java api (operations+store api+users mng)
Sparql (only the query rewriting)
Search API
REST interfaces (+war building)
JDBC/MySQL implementations of store/search/usermanagement
File store implementation (dummy)
Graphic UI (html interface for CRUD operations) - Outside: see project Pesto [DONE]

Add a redirect method to exploit basil as a sparql url shortener

This could be done by adding a resource listening to calls like:

http://basil.kmi.open.ac.uk/basil/.../direct?param=value

Registering new user returns 500 error

But: it seems everything went fine, and I can login without problems

Implement DELETE on /api-id

As the summary says.

DELETE method does not work with MySQL implementation

Throws an SQL error but deletes part of the data

<id>/api-docs should accept / and return JSON by default

To be flexible, for example when using curl we don't want necessarily to add the accept header.

Integration tests module for basil

We need integration tests to avoid features degradation/bugs while developing new functionalities

Release 0.1.1

To publish the fix of #22

NPE when listing latest queries

From the logs:

2015-10-21 10:37:33.771 [qtp278934944-11] WARN  org.eclipse.jetty.server.HttpChannel - //basil.kmi.open.ac.uk/basil
javax.servlet.ServletException: java.lang.NullPointerException
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:397) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:816) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1684) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) ~[basil-server-0.3.1.jar:?]
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667) ~[basil-server-0.3.1.jar:?]
    at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:205) ~[basil-server-0.3.1.jar:?]
    at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:266) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1121) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1055) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:118) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.Server.handle(Server.java:515) ~[basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:291) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:238) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:57) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:191) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:126) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) [basil-server-0.3.1.jar:?]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) [basil-server-0.3.1.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
Caused by: java.lang.NullPointerException
    at uk.ac.open.kmi.basil.rest.core.SpecificationResource.list(SpecificationResource.java:115) ~[basil-server-0.3.1.jar:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_51]
    at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_51]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:172) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) ~[basil-server-0.3.1.jar:?]
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:377) ~[basil-server-0.3.1.jar:?]
    ... 37 more

RDF output for SELECT/ASK queries

The reason is to have a uniform support for all APIs, independently from the nature of the query.

Remove empty model from swagger description

In the meantime, then we should re-implement it properly.

Fix root context to be an index file

And do not show folder structure

Implementation of the Swagger documentation for the core API

As the summary says

Remove reference to path /basil in swagger annotations

In my understanding, that's not portable.

Automatic swagger documentation for created APIs

This refers to the documentation of generated basil APIs

Strange behavior of /view when requsting html

It looks like /api is performed instead...:

$ curl -v http://basil.kmi.open.ac.uk/basil/1jyd93olk8c3b/view -H "Accept: text/html"
...
< HTTP/1.1 400 Bad Request
...
Missing mandatory query parameters: qid

Re-establish accept header list in swagger documentation

It disappeared after the latest changes.

Add swagger annotation for sub-resource /explain

As the summary says

A way to pre-process parameters and post-process results

This is more of a new feature for the wishlist.

Say for example, if I have a simple query such as

select distinct ?city where {
?city a <http://dbpedia.org/ontology/City> ;
<http://dbpedia.org/ontology/country> ?_name_iri .
}

I would like to pass just the local name in the API and concatenate it with a prefix on the fly.

e.g. Italy -> <http://dbpedia.org/resource/Italy>

Similarly it would be nice if some post processing results is possible, for example

select distinct ?country where {
?country a <http://dbpedia.org/ontology/City> .
}

So in this case, to convert <http://dbpedia.org/resource/Italy> to "Italy".

I was guessing may this what the scripts are for but I couldn't find the information in the documentation.

And one last minor request, will it be possible to preserve the variable names when your generate the Swagger specification?

Implement the Search service

As the summary says.

Authentication failures return Server error

We should return a proper status and message instead

Use YASQE as query editor

I noticed you're still using text areas to render the queries. You might want to use the YASQE (http://yasqe.yasgui.org) instead.

To get the YASQE query editor, include the css/js, and run something like YASQE.fromTextArea($('textarea')) (see attachment).
To get a read-only YASQE rendering, add {'readOnly': true} as configuration

If you'd like to integrate this and bump into issues not described in the docs (http://yasqe.yasgui.org/doc), feel free to ping me

Move shiro-specific configuration inside the packed server

This is not meant to be changed by users, and should be moved out of the user configuration and hard-coded in the BASIL+Auth+MySQL server.

Implement a /meta subresource with RDF description of the API itself

This could use:

VOID: to describe the graph
PROV-O: to give provenance information
DCAT: to refer to the BASIL data catalogue?

Fix swagger description of /view

Remove the 'ext' parameter

is group by allowed by basil?

i've got an error when adding a query at http://basil.kmi.open.ac.uk/ap attached query and screenshot.

Line 54 is

      SELECT DISTINCT ?contrato count(?tender) AS ?numLicitadores

PREFIX pproc: <http://contsem.unizar.es/def/sector-publico/pproc#>
PREFIX s: <http://schema.org/>
PREFIX xsd: >http://www.w3.org/2001/XMLSchema#>
PREFIX pc: <http://purl.org/procurement/public-contracts#>
PREFIX gr: <http://purl.org/goodrelations/v1#>
PREFIX dcterms: <http://purl.org/dc/terms/>

SELECT DISTINCT ?contrato ?nombreContrato ?type ?adjudicatario ?impLicitacionSinIVA ?impLicitacionConIVA ?impCanon 
?impAdjudicacionSinIVA ?impAdjudicacionConIVA ?procedimiento ?entidad ?servicio ?numLicitadores 
WHERE {
  ?contrato dcterms:title ?nombreContrato;
  pproc:contractObject ?contractObject;
  pproc:managingDepartment ?managing.
  ?tender pc:supplier ?supplier.
  ?managing dcterms:title ?servicio.
  ?supplier s:name ?adjudicatario.
  ?contractObject pproc:contractEconomicConditions ?contractEconomicConditions.
  OPTIONAL{
    ?contractEconomicConditions pproc:budgetPrice ?budgetPriceNOVAT;
    pproc:budgetPrice ?budgetPriceVAT.
    ?budgetPriceNOVAT gr:hasCurrencyValue ?impLicitacionSinIVA;
    gr:valueAddedTaxIncluded "false"^^xsd:boolean.
    ?budgetPriceVAT gr:hasCurrencyValue ?impLicitacionConIVA;
    gr:valueAddedTaxIncluded "true"^^xsd:boolean.
  }
  OPTIONAL {
    ?contractEconomicConditions pproc:feePrice ?feePrice.
    ?feePrice gr:hasCurrencyValue ?impCanon;
    gr:valueAddedTaxIncluded "false"^^xsd:boolean.
  }
  ?tender pc:offeredPrice ?offeredPriceNOVAT;
  pc:offeredPrice ?offeredPriceVAT.
  ?offeredPriceNOVAT gr:hasCurrencyValue ?impAdjudicacionSinIVA;
  gr:valueAddedTaxIncluded "false"^^xsd:boolean.
  ?offeredPriceVAT gr:hasCurrencyValue ?impAdjudicacionConIVA;
  gr:valueAddedTaxIncluded "true"^^xsd:boolean.
    {
      SELECT DISTINCT ?contrato ?procedimiento ?type ?tender ?entidad ?servicio 
        WHERE {
          ?contrato a pproc:Contract;
          a ?type;
          pproc:contractProcedureSpecifications/pproc:procedureType ?procedimiento;
          pc:contractingAuthority/dcterms:title ?entidad;
          pc:tender ?tender;
          pproc:managingDepartment ?managing.
          ?tender a pproc:FormalizedTender;
          pproc:formalizedDate ?formalizedDate.
          ?managing dcterms:title ?servicio.
          FILTER  (regex(?formalizedDate, "2014")) 
        } 
    } 
    {
      SELECT DISTINCT ?contrato count(?tender) AS ?numLicitadores 
        WHERE {
          ?contrato a pproc:Contract;
          a ?type;pc:tender ?tender.
        } 
      GROUP BY ?contrato
    }
}

Swagger html UI base URL for scripts: default port is no port

As the summary says.

xml format for results does not include an items element wrapper

This is inconsistent with the way XML is rendered for triple lists (and with the default return model in general, for example json).

It is:

<data>
 <vars>...</vars>
 <item>...</item>
 <item>...</item>
 <item>...</item>
  ...
</items>
</data>

It should be:

<data>
<vars>...</vars>
<items>
  <item>...</item>
  ...
</items>
</data>

Introduction/readme file for APIs documentation

A CRUD subresource to document APIs.
PUT a file
GET as text/plain
To be put as introduction in the Swagger UI

Remove embedded swagger ui

It does not work at the moment, and we will deploy a separate static app, as we don't want to have it within the basil source code.

Implementation of paging of API results by setting LIMIT and OFFSET

The LIMIT and OFFSET from SPARQL specification should be used to implement paging of results returned by an API.

Allow mapping of request parameters into SPARQL template slots

It would be cool to allow API requests to come with GET/POST parameters, and allow those parameters to be mapped into certain dynamic areas of the stored SPARQL queries (this would need some SPARQL templating).

.sparql-xml and .sparql-json extensions not working

The following request [1] get 404:

 http://localhost:8080/basil/qxib7zacli5u/api.sparql-xml?geoid=2328926

[1] https://raw.githubusercontent.com/the-open-university/basil/master/src/test/resources/sparql/select_1.txt

Allow for SPARQL INSERT

It would be cool to cover RESTful PUT HTTP requests through allowing INSERT in the Linked Data APIs (given some authentication method to write to the triplestore)

/api: the {ext} path param should be optional

This problem was introduced when removing regular expressions from JAX-RS path params

Document API parameters

Implement a method to document the parameters of each API.

Implementation of Cloning of APIs

An API can be cloned with his documentation and views

Set version to 0.1.0

We are far from 1.0.0 IMHO

Implement a Search service

We add search capabilities to basil...

Add parameter -p (port) to runnable jar

As the summary says

Method not allowed when requesting /basil/<API>

It should redirect to the API specification at /basil/%API%/spec

Extend the Basil Discovery API

Implement the back-end logic and support queries like:

http://basil.kmi.open.ac.uk/basil/discovery/search?endpoint=http://dbpedia.org/sparql
http://basil.kmi.open.ac.uk/basil/discovery/search?resource=http://dbpedia.org/resource/Athens
http://basil.kmi.open.ac.uk/basil/discovery/search?ns=http://www.w3.org/2002/07/owl%23

do not update the search results when an empty string is given on the search input

As I see in the UI, when you search with an empty string, it still updates the results! it gets worse when you search for some terms and then want to go back to the main results without clicking the back button in the browser.
My suggestion would be to have a live search without need to click on the search button.

Location header wrong on view creation

$ curl -v -X PUT http://localhost:8080/basil/qxib7zacli5u/view/html-list -T src/test/resources/mustache/select_1.tmpl 
* Hostname was NOT found in DNS cache 
*   Trying ::1...
* Connected to localhost (::1) port 8080 (#0)
> PUT /basil/qxib7zacli5u/view/html-list HTTP/1.1 
> User-Agent: curl/7.37.1
> Host: localhost:8080
> Accept: */*
> Content-Length: 61
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Date: Fri, 20 Mar 2015 18:02:08 GMT
< Location: http://localhost:8080/basil/html-list
< Content-Length: 0
* Server Jetty(9.3.0.M1) is not blacklisted
< Server: Jetty(9.3.0.M1)
<  
* Connection #0 to host localhost left intact

{
    "vars": ["identifier", "type", "label", "link"],
    "items": [{
        "link": {
            "value": "https://audioboo.fm/boos/1695040",
            "type": "literal"
        },
        "label": {
            "value": "Holism: the whole truth (2\/10)@en",
            "type": "literal"
        },
        "type": {
            "value": "AudiobooPost",
            "type": "literal"
        },
        "identifier": {
            "value": "http://data.open.ac.uk/audioboo/post/1695040",
            "type": "uri"
        }
    }, {
        "link": {
            "value": "https://audioboo.fm/boos/1695018",
            "type": "literal"
       },
        ...
    }
    ...
}

we want:

{
 "vars": ["identifier", "type", "label", "link"], "items": [{
 "link": "https://audioboo.fm/boos/1695040",
 "label": "Holism:␣the␣whole␣truth␣(2\/10)@en",
 "type" : "AudiobooPost" ,
 "identifier": "http://data.open.ac.uk/audioboo/post/1695040"
 },{
 "link": "https://audioboo.fm/boos/1695018",
...
}

Build as a runnable stand-alone jar

We need to pack a stand-alone runnable application with embedded jetty.

basilapi / basil Goto Github PK

basil's Introduction

BASIL

Run

TDB2 backend (default)

MySQL backend (legacy)

Usage

Build

Releasing

basil's People

Contributors

Stargazers

Watchers

Forkers

basil's Issues

Recommend Projects

Recommend Topics

Recommend Org