iay / md-validator Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 343 KB

Metadata Validator Web Service

License: Apache License 2.0

Dockerfile 3.94% Shell 1.11% Java 62.78% XSLT 32.17%

md-validator's Introduction

If you want to know more about me, the about page on my web site is the best place to go.

md-validator's People

Contributors

Watchers

md-validator's Issues

Support a template bean definition for the IdentifiableBeanPostProcessor

The IdentifiableBeanPostProcessor bean is directly defined in validator configuration files by its fully-qualified class name (FQCN). The FQCN has changed between MDA 0.9 and 0.10 due to backend library changes (from Shib java-extensions to shib-shared). Rather than requiring different validator configurations for each MDA version just to change the FQCN of this bean, changes to the FQCN (or even the class itself) can be abstracted from the configuration by using a template bean definition with a suitable identifier defined in a parent context that belongs to the specific MDA being used (unlike the FQCN, the identifier should remain stable).

For MDA 0.10.0, this template bean will exist in the beans.xml file of the MDA itself (see MDA-293). Once added to the upstream MDA, on the main branch of md-validator, we can replace any existing IdentifiableBeanPostProcessor definitions with a bean definition that uses the parent template bean.

For MDA 0.9, there is no central beans.xml file and so the work in MDA-293 will need to be directly applied to the mda-beans.xml file of the mda-validator's 0.9 branch. As with 0.10.0, once the parent bean has been added, we will also need to replace any existing IdentifiableBeanPostProcessor definitions with a bean definition that uses the parent template bean.

Add mda-keylists-rsa dependency

MDA 0.10 splits out the RSA keylists into a different module and hence different dependency (which is included in the aggregator-pipeline module already imported for the 0.9 branch). This should be included as a runtime dependency.

<dependency>
         <groupId>net.shibboleth.metadata</groupId>
        <artifactId>mda-keylists-rsa</artifactId>
        <version>${mda.version}</version>
        <scope>runtime</scope>
</dependency>

Migrate from springfox to springdoc-openapi

The Swagger UI included in this project uses a version of springfox from 2018, which is known to have at least one issue:

https://security.snyk.io/vuln/SNYK-JAVA-IOSPRINGFOX-1075064

Unfortunately, subsequent releases change the API significantly, and I haven't managed to get either 2.10.x or 3.0.0 to drop in.

Even more unfortunately, springfox appears to unmaintained now, so putting a lot of effort into moving to an unaffected version seems like more work than it's worth. In either case, we need to migrate to something else.

The flavour of the month appears to be springdoc-openapi:

https://springdoc.org

As of 2021-11-24 this seems to be actively developed:

https://github.com/springdoc/springdoc-openapi

For example, support for Spring Boot 6 has been added recently (springfox has outstanding issues in the same area, another reason to walk away from it over time).

The biggest issue with springdoc-openapi is that it does not support Swagger 1 or 2 APIs, only OpenAPI a.k.a. Swagger 3. Moving to it will therefore require converting the Swagger spec for the validator API to OpenAPI (probably straightforward) and then using the Swagger 3 code generator (perhaps not trivial, as we have made changes to the generated 2.x code and may need to rework that).

Handle TerminationException as premature end of pipeline, not a service-level error

A common pattern in validation pipelines is to perform a few critical checks (e.g., signature or schema validation) that head off possibly large numbers of consequent errors later, then throw a TerminationException. This is a subclass of StageProcessingException and of PipelineProcessingException.

At the moment, if a PipelineProcessingException is thrown, it's re-thrown like this:

throw new ApiException(HttpStatus.INTERNAL_SERVER_ERROR,
                    "Pipeline failed: " + ex.getMessage(), ex.getCause());

It would be better to catch TerminationException specifically and treat it the same as falling off the end of the pipeline, thus allowing any status metadata it possesses to be sent back to the client. There must be at least one such in all sensible use cases, or the TerminationException would not have been thrown.

The image fails to build

When I try to build the docker image, I'm getting the following error:

tep 7/17 : RUN mvn --batch-mode     -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn     package
 ---> Running in 345fe109d37c
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------< uk.org.iay.md.validator:md-validator >----------------
[INFO] Building Metadata Validation Micro-Service 0.1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (maven-version) @ md-validator ---
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:copy-resources (copy-resources) @ md-validator ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- swagger-codegen-maven-plugin:2.3.1:generate (default) @ md-validator ---
[INFO] reading from swagger/swagger.yaml
[INFO] Set base package to invoker package (uk.org.iay.md.validator)
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/models/InlineResponse404.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/models/Status.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/models/Validator.java
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ValidatorsApiController.java due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ValidatorsApi.java due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/pom.xml due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/README.md due to rule in .swagger-codegen-ignore
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/io/swagger/configuration/HomeController.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/Swagger2SpringBoot.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/resources/application.properties due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ApiException.java due to rule in .swagger-codegen-ignore
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ApiResponseMessage.java
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/NotFoundException.java due to rule in .swagger-codegen-ignore
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ApiOriginFilter.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/io/swagger/configuration/SwaggerDocumentationConfig.java
[INFO] writing file /application/target/generated-sources/swagger/.swagger-codegen/VERSION
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ md-validator ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 35 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ md-validator ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 18 source files to /application/target/classes
[ERROR] error reading /root/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.67/bcprov-jdk15on-1.67.jar; invalid manifest format
[INFO] /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java: /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java uses or overrides a deprecated API.
[INFO] /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ md-validator ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ md-validator ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /application/target/test-classes
[ERROR] error reading /root/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.67/bcprov-jdk15on-1.67.jar; invalid manifest format
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ md-validator ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running uk.org.iay.md.validator.api.ValidatorsApiControllerTest
[ERROR] Tests run: 13, Failures: 9, Errors: 0, Skipped: 4, Time elapsed: 5.795 s <<< FAILURE! - in uk.org.iay.md.validator.api.ValidatorsApiControllerTest
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextPrepareTestInstance  Time elapsed: 5.556 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testGitHub6() throws java.lang.Exception](0)  Time elapsed: 5.566 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testGitHub6() throws java.lang.Exception](0)  Time elapsed: 0.01 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testSchemaFailure() throws java.lang.Exception](0)  Time elapsed: 0.014 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testSchemaFailure() throws java.lang.Exception](0)  Time elapsed: 0.002 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testTestValidation() throws java.lang.Exception](0)  Time elapsed: 0.005 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testTestValidation() throws java.lang.Exception](0)  Time elapsed: 0.003 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testValidators() throws java.lang.Exception](0)  Time elapsed: 0.006 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testValidators() throws java.lang.Exception](0)  Time elapsed: 0.007 s  <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR]   ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » NoSuchMethod
[INFO] 
[ERROR] Tests run: 13, Failures: 9, Errors: 0, Skipped: 4
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:48 min
[INFO] Finished at: 2021-02-10T17:28:41Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project md-validator: There are test failures.
[ERROR] 
[ERROR] Please refer to /application/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
The command '/bin/sh -c mvn --batch-mode     -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn     package' returned a non-zero code: 1

Don't assume the item passed through the pipeline emerges from it

The current implementation assumes that it should serialise the item statuses attached to the item which was put into the pipeline.

It is, however, possible to write stages which replace the Item being processed with another. In this case, the service will serialise the statuses on the wrong item. XSLTransformationStage is one of these, and is fairly heavily used.

The implementation should be changed to serialise the statuses attached to the item emerging from the pipeline instead.

Suppress stack traceback on invalid XML

The java-support library's default configuration of BasicParserPool gives a stack trace if the source can't be parsed. This is supposed to be fixed in java-support v7.4.0, but for now it means the logs get unnecessarily cluttered.

We could try bumping the dependency and see if the issue went away.

Code doesn't compile under Java 11

The code doesn't compile under Java 11 (probably, Java >= 9) because it relies on JAXB being part of the JDK.

This dependency should be made explicit.

Add a container health check

The container should have a HEALTHCHECK so that container orchestration systems have some awareness of the status. At present in the ukf-testbed application, the md-validator-based containers start instantly but are not immediately ready for use, which can cause failures.

Use layered Spring Boot container building

At present, the project builds an artifact which is an executable bundle, then optionally packs that into a Docker container. It would be better (once we move to a more recent version of Spring Boot) to build a container directly so that it can be more simply used as the basis for a deployment with configuration data layered in.

Migrate to latest Spring Boot, Spring Framework, MDA snapshot

This is currently targeted at Spring Boot 1.5.15; it really needs to be retargeted at a currently supported release such as 2.3.x.

This will probably require an update to Spring Framework 5, MDA 0.10.0-SNAPSHOT (for now) and Java 11.

It may make sense to get a lot of that by swapping from the Spring Boot Starter parent POM to the Shibboleth Java 11 platform parent POM.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.