If you want to know more about me, the about page on my web site is the best place to go.
iay / md-validator Goto Github PK
View Code? Open in Web Editor NEWMetadata Validator Web Service
License: Apache License 2.0
Metadata Validator Web Service
License: Apache License 2.0
If you want to know more about me, the about page on my web site is the best place to go.
The IdentifiableBeanPostProcessor bean is directly defined in validator configuration files by its fully-qualified class name (FQCN). The FQCN has changed between MDA 0.9 and 0.10 due to backend library changes (from Shib java-extensions to shib-shared). Rather than requiring different validator configurations for each MDA version just to change the FQCN of this bean, changes to the FQCN (or even the class itself) can be abstracted from the configuration by using a template bean definition with a suitable identifier defined in a parent context that belongs to the specific MDA being used (unlike the FQCN, the identifier should remain stable).
For MDA 0.10.0, this template bean will exist in the beans.xml
file of the MDA itself (see MDA-293). Once added to the upstream MDA, on the main
branch of md-validator
, we can replace any existing IdentifiableBeanPostProcessor definitions with a bean definition that uses the parent template bean.
For MDA 0.9, there is no central beans.xml
file and so the work in MDA-293 will need to be directly applied to the mda-beans.xml file of the mda-validator
's 0.9 branch. As with 0.10.0, once the parent bean has been added, we will also need to replace any existing IdentifiableBeanPostProcessor definitions with a bean definition that uses the parent template bean.
MDA 0.10 splits out the RSA keylists into a different module and hence different dependency (which is included in the aggregator-pipeline
module already imported for the 0.9 branch). This should be included as a runtime dependency.
<dependency>
<groupId>net.shibboleth.metadata</groupId>
<artifactId>mda-keylists-rsa</artifactId>
<version>${mda.version}</version>
<scope>runtime</scope>
</dependency>
The Swagger UI included in this project uses a version of springfox
from 2018, which is known to have at least one issue:
https://security.snyk.io/vuln/SNYK-JAVA-IOSPRINGFOX-1075064
Unfortunately, subsequent releases change the API significantly, and I haven't managed to get either 2.10.x or 3.0.0 to drop in.
Even more unfortunately, springfox
appears to unmaintained now, so putting a lot of effort into moving to an unaffected version seems like more work than it's worth. In either case, we need to migrate to something else.
The flavour of the month appears to be springdoc-openapi
:
As of 2021-11-24 this seems to be actively developed:
https://github.com/springdoc/springdoc-openapi
For example, support for Spring Boot 6 has been added recently (springfox
has outstanding issues in the same area, another reason to walk away from it over time).
The biggest issue with springdoc-openapi
is that it does not support Swagger 1 or 2 APIs, only OpenAPI a.k.a. Swagger 3. Moving to it will therefore require converting the Swagger spec for the validator API to OpenAPI (probably straightforward) and then using the Swagger 3 code generator (perhaps not trivial, as we have made changes to the generated 2.x code and may need to rework that).
A common pattern in validation pipelines is to perform a few critical checks (e.g., signature or schema validation) that head off possibly large numbers of consequent errors later, then throw a TerminationException
. This is a subclass of StageProcessingException
and of PipelineProcessingException
.
At the moment, if a PipelineProcessingException
is thrown, it's re-thrown like this:
throw new ApiException(HttpStatus.INTERNAL_SERVER_ERROR,
"Pipeline failed: " + ex.getMessage(), ex.getCause());
It would be better to catch TerminationException
specifically and treat it the same as falling off the end of the pipeline, thus allowing any status metadata it possesses to be sent back to the client. There must be at least one such in all sensible use cases, or the TerminationException
would not have been thrown.
When I try to build the docker image, I'm getting the following error:
tep 7/17 : RUN mvn --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn package
---> Running in 345fe109d37c
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------< uk.org.iay.md.validator:md-validator >----------------
[INFO] Building Metadata Validation Micro-Service 0.1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (maven-version) @ md-validator ---
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:copy-resources (copy-resources) @ md-validator ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- swagger-codegen-maven-plugin:2.3.1:generate (default) @ md-validator ---
[INFO] reading from swagger/swagger.yaml
[INFO] Set base package to invoker package (uk.org.iay.md.validator)
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/models/InlineResponse404.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/models/Status.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/models/Validator.java
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ValidatorsApiController.java due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ValidatorsApi.java due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/pom.xml due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/README.md due to rule in .swagger-codegen-ignore
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/io/swagger/configuration/HomeController.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/Swagger2SpringBoot.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/resources/application.properties due to rule in .swagger-codegen-ignore
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ApiException.java due to rule in .swagger-codegen-ignore
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ApiResponseMessage.java
[INFO] Skipped generation of /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/NotFoundException.java due to rule in .swagger-codegen-ignore
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/api/ApiOriginFilter.java
[INFO] writing file /application/target/generated-sources/swagger/src/main/java/io/swagger/configuration/SwaggerDocumentationConfig.java
[INFO] writing file /application/target/generated-sources/swagger/.swagger-codegen/VERSION
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ md-validator ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 35 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ md-validator ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 18 source files to /application/target/classes
[ERROR] error reading /root/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.67/bcprov-jdk15on-1.67.jar; invalid manifest format
[INFO] /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java: /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java uses or overrides a deprecated API.
[INFO] /application/target/generated-sources/swagger/src/main/java/uk/org/iay/md/validator/RFC3339DateFormat.java: Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ md-validator ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ md-validator ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /application/target/test-classes
[ERROR] error reading /root/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.67/bcprov-jdk15on-1.67.jar; invalid manifest format
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ md-validator ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running uk.org.iay.md.validator.api.ValidatorsApiControllerTest
[ERROR] Tests run: 13, Failures: 9, Errors: 0, Skipped: 4, Time elapsed: 5.795 s <<< FAILURE! - in uk.org.iay.md.validator.api.ValidatorsApiControllerTest
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextPrepareTestInstance Time elapsed: 5.556 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testGitHub6() throws java.lang.Exception](0) Time elapsed: 5.566 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testGitHub6() throws java.lang.Exception](0) Time elapsed: 0.01 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testSchemaFailure() throws java.lang.Exception](0) Time elapsed: 0.014 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testSchemaFailure() throws java.lang.Exception](0) Time elapsed: 0.002 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testTestValidation() throws java.lang.Exception](0) Time elapsed: 0.005 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testTestValidation() throws java.lang.Exception](0) Time elapsed: 0.003 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextBeforeTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testValidators() throws java.lang.Exception](0) Time elapsed: 0.006 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[ERROR] uk.org.iay.md.validator.api.ValidatorsApiControllerTest.springTestContextAfterTestMethod[public void uk.org.iay.md.validator.api.ValidatorsApiControllerTest.testValidators() throws java.lang.Exception](0) Time elapsed: 0.007 s <<< FAILURE!
java.lang.NoSuchMethodError: org.springframework.test.context.TestContext.computeAttribute(Ljava/lang/String;Ljava/util/function/Function;)Ljava/lang/Object;
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextAfterTestMethod:214 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextBeforeTestMethod:161 » NoSuchMethod
[ERROR] ValidatorsApiControllerTest>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » NoSuchMethod
[INFO]
[ERROR] Tests run: 13, Failures: 9, Errors: 0, Skipped: 4
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:48 min
[INFO] Finished at: 2021-02-10T17:28:41Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project md-validator: There are test failures.
[ERROR]
[ERROR] Please refer to /application/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
The command '/bin/sh -c mvn --batch-mode -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn package' returned a non-zero code: 1
The current implementation assumes that it should serialise the item statuses attached to the item which was put into the pipeline.
It is, however, possible to write stages which replace the Item
being processed with another. In this case, the service will serialise the statuses on the wrong item. XSLTransformationStage
is one of these, and is fairly heavily used.
The implementation should be changed to serialise the statuses attached to the item emerging from the pipeline instead.
The java-support
library's default configuration of BasicParserPool
gives a stack trace if the source can't be parsed. This is supposed to be fixed in java-support
v7.4.0, but for now it means the logs get unnecessarily cluttered.
We could try bumping the dependency and see if the issue went away.
The code doesn't compile under Java 11 (probably, Java >= 9) because it relies on JAXB being part of the JDK.
This dependency should be made explicit.
The container should have a HEALTHCHECK
so that container orchestration systems have some awareness of the status. At present in the ukf-testbed
application, the md-validator
-based containers start instantly but are not immediately ready for use, which can cause failures.
At present, the project builds an artifact which is an executable bundle, then optionally packs that into a Docker container. It would be better (once we move to a more recent version of Spring Boot) to build a container directly so that it can be more simply used as the basis for a deployment with configuration data layered in.
This is currently targeted at Spring Boot 1.5.15; it really needs to be retargeted at a currently supported release such as 2.3.x.
This will probably require an update to Spring Framework 5, MDA 0.10.0-SNAPSHOT (for now) and Java 11.
It may make sense to get a lot of that by swapping from the Spring Boot Starter parent POM to the Shibboleth Java 11 platform parent POM.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.