fcrepo4-labs / fcrepo-api-x Goto Github PK
View Code? Open in Web Editor NEWFedora API Extension Framework
License: Apache License 2.0
Fedora API Extension Framework
License: Apache License 2.0
Islandora CLAW PHP/Camel microservices share some similar patterns to API-X. Take a look at CLAW microservices & middleware to see where similarities and differences in API-X lie. Be the point person for communication between API-X or Islandora team members. A face-to-face may be appropriate
This is a public-facing doc, authored by the stakeholders, intended to give an overview of API-X for those interested in assessing its suitability for meeting business needs.
This is a document that describes the features, installed extensions, and evaluation steps:
Guide for stakeholders, providing instructions for evaluation tasks, see milestone doc
When the API-X features start up, they want to initialize state in the Fedora repository.
When building docker images it would be useful to be able to install the API-X features at image build time.
The problem is that a Fedora repository isn't available at Docker image build time, so installing the features fails.
It would be useful to be able to suppress the initialization of the Fedora repository in this case, allowing the features to be installed in a Docker image, bypassing the initialization of the Fedora repository.
This is an implementation of #6 (client-side service discovery specification). For a given object, the set of services on this list is determined by applying the selection criteria defined in the extension registry model
Depends on:
Simplifying Assumptions:
Testing:
Use camel toolbox to create a simple service that listens for messages, indexes service documents of new and updated objects
Compare & contrast the different approaches discussed so far, write up a proposal for an invocation model to be used API-X. This could simply involve choosing an approach from below, a hybrid, or something else entirely. Seek out consensus on the chosen appproach
In order to be in line with other fcrepo-exts
projects, the API-X code repository will need at least two people listed in a "Maintainers" section in the README document.
Create simple, quick-to-produce walkthrough videos that illustrate demo evaluation tasks
See 2015-06-23 meeting minutes.
Should API-X be given the task of caching or otherwise providing durability for ontologies referenced in extensions?
The concern is that since ontologies are used for purposes of extension binding (e.g. reasoning, see extension definition & binding, and ontologies can be imported via owl:imports
, ontology resolution failures could impact service binding. It is likely the case that any sane implementation will perform some sort of caching - but should API-X be making any guarantees?
As the API-X codebase matures, it becomes more difficult to manage the proof-of-concept related material from the core code. Not only can it be more difficult to manage, but it also doesn't make sense to keep mature code together with code that is intended to demonstrate nascent concepts. Down the road, it may make sense to promote the API-X code from the "labs" repository, but leave the proofs-of-concept behind.
Ideally, the proof-of-concept (POC) related items can be separated into a distinct repository, for the reasons discussed above. The POC repository should maintain its history from this repository.
Create simple mechanism (service?) that loads extension into API-X based on URI of an Amherst service. An extension definition document is present in the body of OPTIONS
requests
OPTIONS
to the Amherst service, registers extension and service in API-XAn interesting and uncomplicated example for API-X might be offering Trusty URIs over a repo.
Implement mechanism for routing URIs exposed on objects (e.g. /path/to/object/ext:extension/), and proxied by API-X.
Simplifying assumptions:
Depends on:
Testing
Contact Andrew Woods for setting up Jenkins to deploy snaps into sonatype.
Artifacts intended for release:
groupid: org.fcrepo.apix
proof-of-concept artifacts
groupid: org.fcrepo.apix.poc
As noted in #42, builds are consistently successful when run locally on developer machines, but fail for various (not necessarily consistent) causes when executed on Travis
This should be one of the last steps of creating the milestone, as bugs may be discovered and fixed in the process of assembling/testing the demo
Release as 0.1.0?
For the milestone release, the docker-compose.yaml should refer to concrete versions of an image, not :latest
.
Some consideration should be made regarding the docker image tags used for this milestone, depending on how long we think the milestone should continue to be viable.
Upon retrieving a ~15 mb binary through API-X (works fine through Fedora):
$ curl -i http://apix/fcrepo/rest/images/porsche.jpg
HTTP/1.1 500 Server Error
Content-Type: text/plain; charset=ISO-8859-1
Transfer-Encoding: chunked
Server: Jetty(9.2.17.v20160517)
org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-5dabce1f8a67-39534-1478205115914-2-29]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
at org.apache.camel.component.jetty9.JettyContentExchange9.doTaskCompleted(JettyContentExchange9.java:155)
at org.apache.camel.component.jetty9.JettyContentExchange9$2.onComplete(JettyContentExchange9.java:221)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:453)
at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:434)
at org.eclipse.jetty.client.HttpReceiver.responseHeaders(HttpReceiver.java:278)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.headerComplete(HttpReceiverOverHTTP.java:228)
at org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1036)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1257)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:156)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:117)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:123)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
at org.eclipse.jetty.client.util.BufferingResponseListener.onHeaders(BufferingResponseListener.java:75)
at org.eclipse.jetty.client.ResponseNotifier.notifyHeaders(ResponseNotifier.java:103)
at org.eclipse.jetty.client.ResponseNotifier.notifyHeaders(ResponseNotifier.java:95)
at org.eclipse.jetty.client.HttpReceiver.responseHeaders(HttpReceiver.java:257)
... 12 more
Determine shared state, configuration, and communication needs between instances of API-X core. Determine key patterns and use cases (e.g. load balance, failover)
(see patterns doc)
Simplifying assumptions:
Depends on:
Testing:
ServiceDocumentGenerator will throw an NPE instead of <insert a more acceptable behavior> when requesting a service doc in a non-rdf type (e.g. text/html)
See comments on design doc draft
Regarding Service Discovery & Binding:
Such a management interface should provide an API for monitoring the running components and probably gather some metrics about things like request service time and failure conditions. For example via JMX or an HTTP API.
This issue is to determine if API-X needs to provide an interface for monitoring components and retrieving metrics, and what form that may take. Can any existing technologies and standards be leveraged for this?
We already have a Docker image for the wrk utility, which measures throughput and latency (manually)
May wish to engage Fedora’s performance/scaling community to see if any of their techniques (jmeter?) are appropriate.
Refactor the design page into a proper, informative landing page for API-X design with links to the appropriate documentation and goings on
... or one of the POCs (amherst-apix or go-proxy) if api-x is not available yet. Compare results to test suite run against bare fedora, create github issues for any failures.
(e.g. consensus-based distributed registry, DNS-based discovery).
Select appropriate Maven plugin for launching api-x core, services, and Fedora for integration testing. Presumably, this will involve running Docker images akin to those defined in docker-compose.yml
To reproduce:
Similar behavior using either Fedora 4.6.0 or 4.7.0 in the 'fcrepo' container.
HTTP/1.1 500 Server Error
Content-Type: text/turtle
Accept: /
Accept-Patch: application/sparql-update
Accept-Post: text/turtle,text/rdf+n3,text/n3,application/rdf+xml,application/n-triples,multipart/form-data,application/sparql-update
Allow: MOVE,COPY,DELETE,POST,HEAD,GET,PUT,PATCH,OPTIONS
breadcrumbId: ID-9039a37c0da5-41693-1478206589033-2-200
ETag: W/"85a2b321e51df70cfa3b5706b4b8b1446998cf00"
Last-Modified: Thu, 03 Nov 2016 21:08:28 GMT
Link: http://www.w3.org/ns/ldp#Container;rel="type"
Link: http://www.w3.org/ns/ldp#IndirectContainer;rel="type"
Link: http://www.w3.org/ns/ldp#Resource;rel="type"
Link: http://192.168.99.100/discovery/apix/indirect4; rel="service"
Preference-Applied: return=representation
Server: Jetty(9.2.3.v20140905)
Vary: Prefer
Vary: Accept, Range, Accept-Encoding, Accept-Language
Transfer-Encoding: chunked
java.lang.NullPointerException
at com.google.common.collect.ImmutableBiMap.copyOf(ImmutableBiMap.java:201)
at org.fcrepo.kernel.modeshape.rdf.converters.PropertyConverter.getPropertyNameFromPredicate(PropertyConverter.java:148)
at org.fcrepo.kernel.modeshape.rdf.converters.PropertyConverter.getPropertyNameFromPredicate(PropertyConverter.java:102)
at org.fcrepo.kernel.modeshape.rdf.impl.LdpContainerRdfContext.lambda$memberRelations$108(LdpContainerRdfContext.java:131)
at org.fcrepo.kernel.modeshape.utils.UncheckedFunction.apply(UncheckedFunction.java:37)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
at java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:419)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:731)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:731)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:728)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:731)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:731)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:728)
at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:735)
at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169)
at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at org.openrdf.rio.Rio.write(Rio.java:564)
at org.fcrepo.http.commons.responses.RdfStreamStreamingOutput.write(RdfStreamStreamingOutput.java:150)
at org.fcrepo.http.commons.responses.RdfStreamStreamingOutput.write(RdfStreamStreamingOutput.java:118)
at org.fcrepo.http.commons.responses.RdfStreamProvider.writeTo(RdfStreamProvider.java:102)
at org.fcrepo.http.commons.responses.RdfStreamProvider.writeTo(RdfStreamProvider.java:58)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:86)
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:745)
esm:~/workspaces/apix/repository-extension-services/acrepo-exts-image/src/main/resources (master)*$
The API-X demo will need to have demonstration objects pre-loaded, just needs to be implemented. Whether this happens at image build time or at container runtime is TBD.
Create corresponding github issues.
See if there’s potential for consolidation of effort regarding Base Docker images (e.g fcrepo image, base karaf, etc). Seek out others in the community that are using Docker in a similar way (Islandora? Hydra? Va Tech?)
Investigate possible Fedora memory leak and seemingly related latency anomalies that complicate isolating latency effects due to API-X
Initial content from emetsger/apix-docker-test)
Shall contain at least docker-compose.yml launching all necessary image(s) for running the demo
Have stakeholders look at PR containing the licensing and CLA info. Standard Duraspace, Apache 2.0 text
Decide its canonical location (wiki, github?). Make sure definitions and descriptions reflect consensus of API-X stakeholders. Current thinking on this is the best home for this would be in the github repo, in an .md file
Part of an extension definition in the registry will describe objects that can be bound to a given extension.
Simplifying assumptions:
Depends on:
Testing:
Use camel toolbox to create simple updater service that listens for messages, notifies API-X when objects are added/deleted/modified
There is already an org.fcrepo.apix.model.components.Updateable
interface. This listener would listen for messages from the repository, and trigger all Updateable
services when appropriate
As API-X provides a proxy endpoint for Fedora, repository resources exposed through API-X have different URIs than their fedora counterparts. When relative URIs are used in requests and response, there is no issue. Absolute URIs may need to be rewritten in request and response bodies. This is akin to mod_proxy_html, except for rdf.
Testing:
Camel 2.7.1. contains fixes for CAMEL-9462 and CAMEL-9200
This pattern is necessary when multiple extensions are bound to a single resource
If it’s not part of the registry model already, find out how to deterministically arrive at the order in which the request/response are routed through extensions
Depends on:
Simplifying assumptions:
Testing:
Add a docker container running Karaf + API-X to the repository created in #44
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.