terminalstatic / go-xsd-validate Goto Github PK
View Code? Open in Web Editor NEWXsd validation for go based on libxml2
License: MIT License
Xsd validation for go based on libxml2
License: MIT License
# github.com/terminalstatic/go-xsd-validate
./libxml2.go: In function ‘cleanup’:
./libxml2.go:116:5: warning: ‘xmlSchemaCleanupTypes’ is deprecated [-Wdeprecated-declarations]
116 | xmlSchemaCleanupTypes();
| ^~~~~~~~~~~~~~~~~~~~~
In file included from ./libxml2.go:9:
/usr/include/libxml2/libxml/xmlschemastypes.h:37:17: note: declared here
37 | xmlSchemaCleanupTypes (void);
| ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go: In function ‘parseSchema’:
./libxml2.go:186:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
186 | xmlLineNumbersDefault(1);
| ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/libxml2/libxml/tree.h:17,
from /usr/include/libxml2/libxml/schemasInternals.h:24,
from /usr/include/libxml2/libxml/xmlschemastypes.h:19:
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
967 | xmlLineNumbersDefault (int val);
| ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go: In function ‘cParseDoc’:
./libxml2.go:247:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
247 | xmlLineNumbersDefault(1);
| ^~~~~~~~~~~~~~~~~~~~~
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
967 | xmlLineNumbersDefault (int val);
| ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go: In function ‘cValidate’:
./libxml2.go:300:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
300 | xmlLineNumbersDefault(1);
| ^~~~~~~~~~~~~~~~~~~~~
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
967 | xmlLineNumbersDefault (int val);
| ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go:328:59: warning: passing argument 2 of ‘xmlSchemaSetValidStructuredErrors’ from incompatible pointer type [-Wincompatible-pointer-types]
328 | xmlSchemaSetValidStructuredErrors(schemaCtxt, simpleStructErrorCallback,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
| |
| void (*)(void *, xmlError *) {aka void (*)(void *, struct _xmlError *)}
In file included from /usr/include/libxml2/libxml/xmlschemastypes.h:20:
/usr/include/libxml2/libxml/xmlschemas.h:185:65: note: expected ‘xmlStructuredErrorFunc’ {aka ‘void (*)(void *, const struct _xmlError *)’} but argument is of type ‘void (*)(void *, xmlError *)’ {aka ‘void (*)(void *, struct _xmlError *)’}
185 | xmlStructuredErrorFunc serror,
| ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
./libxml2.go: In function ‘cValidateBuf’:
./libxml2.go:353:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
353 | xmlLineNumbersDefault(1);
| ^~~~~~~~~~~~~~~~~~~~~
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
967 | xmlLineNumbersDefault (int val);
| ^~~~~~~~~~~~~~~~~~~~~
These changes seem to appear in 2.12.0.
I'll submit a PR shortly.
Please replace your dependency malloc.h
by stdlib.h
. Then it starts working on MacOS, too.
Here he you can see that malloc.h
seems to be deprecated: https://stackoverflow.com/a/56463133
Under Linux I want to cross compile my project with xsd-validate module for Windows platform. For Linux platform the build runs and the executable works. But If I try to build for Windows I go following message and the build fails.
Under Linux I installed version 2.9.12 of libxml2
../../internal/datamodel/datamodel.go:57:14: undefined: xsdvalidate.Init
../../internal/datamodel/datamodel.go:58:20: undefined: xsdvalidate.Cleanup
../../internal/datamodel/datamodel.go:59:33: undefined: xsdvalidate.NewXsdHandlerMem
../../internal/datamodel/datamodel.go:59:105: undefined: xsdvalidate.ParsErrDefault
../../internal/datamodel/datamodel.go:65:33: undefined: xsdvalidate.NewXmlHandlerMem
../../internal/datamodel/datamodel.go:65:74: undefined: xsdvalidate.ParsErrDefault
../../internal/datamodel/datamodel.go:72:52: undefined: xsdvalidate.ValidErrDefault
Can I use xsd-validate for Windows?
What I have to do to compile my project for Windows under Linux?
Thank you very much
Andreas
I am currently using the go-xsd-validate library and noticed a deprecation warning for the function xmlSchemaCleanupTypes when compiling my project:
warning: 'xmlSchemaCleanupTypes' is deprecated [-Wdeprecated-declarations]
Attempting to use xsdvalidate.NewXsdHandlerUrl with the url http://cyclonedx.org/schema/bom/1.3 fails without anydetails. Investigation shows that this is because it is a redirect to a https endpoint.
This issue in xmllint
https://gitlab.gnome.org/GNOME/libxml2/-/issues/160
indicates that libxml2 will never support https and may even stop supporting its own http client.
So this is just a heads up for anyone else who may encounter this.
With the general move from http to https NewXsdHandlerUrl stands less and less chance of actually working.
Hi,
I currently develop a CSW API (OGC norm). I found 0 libraries for xsd validation in golang, so for now my program validates xsd using a process call to libxml2, it works but this is a temporary solution.
Example:
xmllint --noout --nocompact --nsclean --schema http://schemas.opengis.net/cat/csw/3.0/cswAll.xsd
When xmllint encounters an unknown schema, it fetches it on the internet. I bypassed those connexions by setting up a local server on port 80 and adding schemas.opengis.net on /etc/hosts.
In brief, it works but it sucks.
So, I was glad to find your library today, but it doesn't seem to work for my use case, and I get the following error: Xsd parser error
which is not very developer-friendly.
Here is a non-working sample code:
import (
xsd "github.com/terminalstatic/go-xsd-validate"
)
func myFunction() {
xsd.Init()
defer xsd.Cleanup()
xsdhandler, err := xsd.NewXsdHandlerUrl("/path/to/schemas.opengis.net/cat/csw/3.0/cswAll.xsd", xsd.ParsErrDefault)
if err != nil {
panic("handleUrl " + err.Error())
}
xmlhandler, err := xsd.NewXmlHandlerMem(input, xsd.ParsErrDefault)
if err != nil {
panic("handleMem " + err.Error())
}
defer xmlhandler.Free()
err = xsdhandler.Validate(xmlhandler, xsd.ValidErrDefault)
if err != nil {
panic("Validate " + err.Error())
}
}
The code panics at xsd.NewXsdHandlerUrl(...)
Anyway, what do you recommand for this case ? Do I need to register every schemas ? How do I get better errors message ?
Thanks
Does the validation work with split xsd files? For example you have one main xsd file that references other xsd files using
xs:import
This is commonly used in Germany for xsd validation of e-invoices.
I attached 4 xsd files and one example xml file to be validated.
I've got strange issue when trying to validate large number (more than 100) of large xml files (20-130mb).
It looks like this
unexpected fault address 0xc0068c3000
fatal error: fault
[signal SIGBUS: bus error code=0x4 addr=0xc0068c3000 pc=0x55dffd]
or
SIGSEGV: segmentation violation
PC=0x7f19813d22222e3 m=43 sigcode=1
signal arrive during cgo execution
Stacktrace always the same
runtime.cgocall(0x868de0, 0xc000739698)
/usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc000739670 sp=0xc000739638 pc=0x40a68b
github.com/terminalstatic/go-xsd-validate._C2func_cParseDoc(0x7fe986193010, 0x5e6c74f, 0x1)
_cgo_gotypes.go:254 +0x57 fp=0xc000739698 sp=0xc000739670 pc=0x538d57
github.com/terminalstatic/go-xsd-validate.parseXmlMem.func3(0x7fe986193010?, {0xc020aa0000?, 0x5e6c74f, 0x757e000?}, 0x1?)
/builds/app/.go/pkg/mod/github.com/terminalstatic/[email protected]/libxml2.go:433 +0x5a fp=0xc0007396e8 sp=0xc000739698 pc=0x5397fa
github.com/terminalstatic/go-xsd-validate.parseXmlMem({0xc020aa0000, 0x5e6c74f, 0x757e000}, 0xfe?)
/builds/app/.go/pkg/mod/github.com/terminalstatic/[email protected]/libxml2.go:433 +0xa5 fp=0xc000739780 sp=0xc0007396e8 pc=0x5395e5
github.com/terminalstatic/go-xsd-validate.NewXmlHandlerMem({0xc020aa0000?, 0xc00f542010?, 0x0?}, 0x1d?)
/builds/app/.go/pkg/mod/github.com/terminalstatic/[email protected]/validate_xsd.go:94 +0x29 fp=0xc0007397d8 sp=0xc000739780 pc=0x53a3e9
Problem is that I can't reproduce it on my machine with same files and I don't have access to server, where error occurs.
Error can happen at any time, on any file, can't reproduce it on exact one file or set of files. Only on bunch of xml's. Error can happen at second file or at 29th, no pattern.
How can I debug or reproduce error? Maybe there is a bug C
code?
I am facing an issue with huge memory usage while validating large XML files (about 90 MB).
I might be doing something wrong but there seems to be a memory leakage because the used memory grows after every file and never clears up until I restart the server.
My setup is an HTTP server with a handler that validates an uploaded XML file against a provided XSD.
My main func looks something like this:
func main() {
// init xsdvalidate
// xsdvalidate.InitWithGc(2 * time.Minute) // didn't help
err = xsdvalidate.Init()
defer xsdvalidate.Cleanup()
if err != nil {
log.WithError(err).Fatalf("could not init xsdvalidate")
}
...
}
and inside my handler I am doing something like this:
func (it *service) handler(file io.ReadCloser, filesize int64) error {
defer file.Close()
xsdHandler, err := xsdvalidate.NewXsdHandlerMem(it.meteringObjectsXSD, xsdvalidate.ParsErrDefault) // it.meteringObjectsXSD is a small XSD file (3KB) read into a byte slice
defer xsdHandler.Free()
if err != nil {
return err
}
fileContent := make([]byte, filesize)
_, err = io.ReadFull(file, fileContent)
if err != nil {
return err
}
err = xsdHandler.ValidateMem(fileContent, xsdvalidate.ParsErrDefault)
if err != nil {
return err
}
...
}
(I also tried creating the XSD handler only once inside main and injecting it into my service but that didn't help either)
When uploading a 90MB XML file and validating it with the XSD the memory usage looks like this:
After uploading it again the memory usage grows almost twice the size:
Unfortunately, the memory never frees up.
Now when using the same service again but commenting out the validation part: err = xsdHandler.ValidateMem(fileContent, xsdvalidate.ParsErrDefault)
the memory usage looks like this
After the first time uploading the 90MB XML file:
After the second time uploading the file:
The file was still read into memory but after the GC kicks in it will be all freed up.
Any idea why the memory usage is so high and why it's only growing and never being freed up?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.