terminalstatic / go-xsd-validate Goto Github PK

View Code? Open in Web Editor NEW

65.0 65.0 18.0 107 KB

Xsd validation for go based on libxml2

License: MIT License

Go 100.00%

go golang libxml2 validation xml xsd

go-xsd-validate's People

Contributors

Stargazers

Watchers

Forkers

arthurgustin c0sco bmalavan buddhamagnet matiasinsaurralde agileinsider jeremy-hanna profbiss giacomocortesi slax0rr thanson16 gkoutsoumpakis-form3 eliziorezende deemount form3tech-oss dukeluke94 wgh-

go-xsd-validate's Issues

Deprecation and incompatible pointer type warning on new libxml2 versions

# github.com/terminalstatic/go-xsd-validate
./libxml2.go: In function ‘cleanup’:
./libxml2.go:116:5: warning: ‘xmlSchemaCleanupTypes’ is deprecated [-Wdeprecated-declarations]
  116 |     xmlSchemaCleanupTypes();
      |     ^~~~~~~~~~~~~~~~~~~~~
In file included from ./libxml2.go:9:
/usr/include/libxml2/libxml/xmlschemastypes.h:37:17: note: declared here
   37 |                 xmlSchemaCleanupTypes           (void);
      |                 ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go: In function ‘parseSchema’:
./libxml2.go:186:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
  186 |     xmlLineNumbersDefault(1);
      |     ^~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/libxml2/libxml/tree.h:17,
                 from /usr/include/libxml2/libxml/schemasInternals.h:24,
                 from /usr/include/libxml2/libxml/xmlschemastypes.h:19:
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
  967 |                 xmlLineNumbersDefault   (int val);
      |                 ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go: In function ‘cParseDoc’:
./libxml2.go:247:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
  247 |     xmlLineNumbersDefault(1);
      |     ^~~~~~~~~~~~~~~~~~~~~
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
  967 |                 xmlLineNumbersDefault   (int val);
      |                 ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go: In function ‘cValidate’:
./libxml2.go:300:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
  300 |     xmlLineNumbersDefault(1);
      |     ^~~~~~~~~~~~~~~~~~~~~
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
  967 |                 xmlLineNumbersDefault   (int val);
      |                 ^~~~~~~~~~~~~~~~~~~~~
./libxml2.go:328:59: warning: passing argument 2 of ‘xmlSchemaSetValidStructuredErrors’ from incompatible pointer type [-Wincompatible-pointer-types]
  328 |             xmlSchemaSetValidStructuredErrors(schemaCtxt, simpleStructErrorCallback,
      |                                                           ^~~~~~~~~~~~~~~~~~~~~~~~~
      |                                                           |
      |                                                           void (*)(void *, xmlError *) {aka void (*)(void *, struct _xmlError *)}
In file included from /usr/include/libxml2/libxml/xmlschemastypes.h:20:
/usr/include/libxml2/libxml/xmlschemas.h:185:65: note: expected ‘xmlStructuredErrorFunc’ {aka ‘void (*)(void *, const struct _xmlError *)’} but argument is of type ‘void (*)(void *, xmlError *)’ {aka ‘void (*)(void *, struct _xmlError *)’}
  185 |                                          xmlStructuredErrorFunc serror,
      |                                          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
./libxml2.go: In function ‘cValidateBuf’:
./libxml2.go:353:5: warning: ‘xmlLineNumbersDefault’ is deprecated [-Wdeprecated-declarations]
  353 |     xmlLineNumbersDefault(1);
      |     ^~~~~~~~~~~~~~~~~~~~~
/usr/include/libxml2/libxml/parser.h:967:17: note: declared here
  967 |                 xmlLineNumbersDefault   (int val);
      |                 ^~~~~~~~~~~~~~~~~~~~~

These changes seem to appear in 2.12.0.

I'll submit a PR shortly.

element omitempty not work

element, omitempty - it does not work
When I want to omit an element that is empty so that it is not shown in the XML, this library continues to validate the element and indicates "his element is not expected."

malloc.h is not available on MacOS

Please replace your dependency malloc.h by stdlib.h. Then it starts working on MacOS, too.

Here he you can see that malloc.h seems to be deprecated: https://stackoverflow.com/a/56463133

Getting issue in installation

Hello,
I'm tring to compare xsd with xml but getting error while installation. Can you please suggest the solution.

Before

After go get

In debug

Cross compiling for Windows under Linux

Under Linux I want to cross compile my project with xsd-validate module for Windows platform. For Linux platform the build runs and the executable works. But If I try to build for Windows I go following message and the build fails.

Under Linux I installed version 2.9.12 of libxml2

../../internal/datamodel/datamodel.go:57:14: undefined: xsdvalidate.Init
../../internal/datamodel/datamodel.go:58:20: undefined: xsdvalidate.Cleanup
../../internal/datamodel/datamodel.go:59:33: undefined: xsdvalidate.NewXsdHandlerMem
../../internal/datamodel/datamodel.go:59:105: undefined: xsdvalidate.ParsErrDefault
../../internal/datamodel/datamodel.go:65:33: undefined: xsdvalidate.NewXmlHandlerMem
../../internal/datamodel/datamodel.go:65:74: undefined: xsdvalidate.ParsErrDefault
../../internal/datamodel/datamodel.go:72:52: undefined: xsdvalidate.ValidErrDefault

Can I use xsd-validate for Windows?
What I have to do to compile my project for Windows under Linux?

Thank you very much
Andreas

Deprecation Warning for xmlSchemaCleanupTypes in libxml2

I am currently using the go-xsd-validate library and noticed a deprecation warning for the function xmlSchemaCleanupTypes when compiling my project:

warning: 'xmlSchemaCleanupTypes' is deprecated [-Wdeprecated-declarations]

http vs https for downloading schema

Attempting to use xsdvalidate.NewXsdHandlerUrl with the url http://cyclonedx.org/schema/bom/1.3 fails without anydetails. Investigation shows that this is because it is a redirect to a https endpoint.

This issue in xmllint

https://gitlab.gnome.org/GNOME/libxml2/-/issues/160

indicates that libxml2 will never support https and may even stop supporting its own http client.

So this is just a heads up for anyone else who may encounter this.

With the general move from http to https NewXsdHandlerUrl stands less and less chance of actually working.

How to validate CSW schema

Hi,
I currently develop a CSW API (OGC norm). I found 0 libraries for xsd validation in golang, so for now my program validates xsd using a process call to libxml2, it works but this is a temporary solution.

Example:

xmllint --noout --nocompact --nsclean --schema http://schemas.opengis.net/cat/csw/3.0/cswAll.xsd

When xmllint encounters an unknown schema, it fetches it on the internet. I bypassed those connexions by setting up a local server on port 80 and adding schemas.opengis.net on /etc/hosts.

In brief, it works but it sucks.

So, I was glad to find your library today, but it doesn't seem to work for my use case, and I get the following error: Xsd parser error which is not very developer-friendly.

Here is a non-working sample code:

import (
    xsd "github.com/terminalstatic/go-xsd-validate"
)

func myFunction() {
        xsd.Init()
	defer xsd.Cleanup()
	xsdhandler, err := xsd.NewXsdHandlerUrl("/path/to/schemas.opengis.net/cat/csw/3.0/cswAll.xsd", xsd.ParsErrDefault)
	if err != nil {
		panic("handleUrl " + err.Error())
	}

	xmlhandler, err := xsd.NewXmlHandlerMem(input, xsd.ParsErrDefault)
	if err != nil {
		panic("handleMem " + err.Error())
	}
	defer xmlhandler.Free()

	err = xsdhandler.Validate(xmlhandler, xsd.ValidErrDefault)
	if err != nil {
		panic("Validate " + err.Error())
	}
}

The code panics at xsd.NewXsdHandlerUrl(...)

Anyway, what do you recommand for this case ? Do I need to register every schemas ? How do I get better errors message ?

Thanks

Question: Split xsd - functionality <xs:import>

Does the validation work with split xsd files? For example you have one main xsd file that references other xsd files using
xs:import

This is commonly used in Germany for xsd validation of e-invoices.
I attached 4 xsd files and one example xml file to be validated.

XSD-Test.zip

Error: malloc_trim

Error: github.com\terminalstatic\go-xsd-validate\libxml2.go line 480
unknow C.malloc_trim(0)
i replace by C.malloc(0) and build

Windows 8.1 x64
MinGW-W64-builds-4.3.5 - install preferences:
mingw-w64
64bits
posix
sjlj

SIGSEGV or SIGBUS when validating

I've got strange issue when trying to validate large number (more than 100) of large xml files (20-130mb).
It looks like this

unexpected fault address 0xc0068c3000
fatal error: fault
[signal SIGBUS: bus error code=0x4 addr=0xc0068c3000 pc=0x55dffd]

SIGSEGV: segmentation violation
PC=0x7f19813d22222e3 m=43 sigcode=1
signal arrive during cgo execution

Stacktrace always the same

runtime.cgocall(0x868de0, 0xc000739698)
  /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc000739670 sp=0xc000739638 pc=0x40a68b
github.com/terminalstatic/go-xsd-validate._C2func_cParseDoc(0x7fe986193010, 0x5e6c74f, 0x1)
  _cgo_gotypes.go:254 +0x57 fp=0xc000739698 sp=0xc000739670 pc=0x538d57
github.com/terminalstatic/go-xsd-validate.parseXmlMem.func3(0x7fe986193010?, {0xc020aa0000?, 0x5e6c74f, 0x757e000?}, 0x1?)
  /builds/app/.go/pkg/mod/github.com/terminalstatic/[email protected]/libxml2.go:433 +0x5a fp=0xc0007396e8 sp=0xc000739698 pc=0x5397fa
github.com/terminalstatic/go-xsd-validate.parseXmlMem({0xc020aa0000, 0x5e6c74f, 0x757e000}, 0xfe?)
  /builds/app/.go/pkg/mod/github.com/terminalstatic/[email protected]/libxml2.go:433 +0xa5 fp=0xc000739780 sp=0xc0007396e8 pc=0x5395e5
github.com/terminalstatic/go-xsd-validate.NewXmlHandlerMem({0xc020aa0000?, 0xc00f542010?, 0x0?}, 0x1d?)
  /builds/app/.go/pkg/mod/github.com/terminalstatic/[email protected]/validate_xsd.go:94 +0x29 fp=0xc0007397d8 sp=0xc000739780 pc=0x53a3e9

Problem is that I can't reproduce it on my machine with same files and I don't have access to server, where error occurs.

Error can happen at any time, on any file, can't reproduce it on exact one file or set of files. Only on bunch of xml's. Error can happen at second file or at 29th, no pattern.

How can I debug or reproduce error? Maybe there is a bug C code?

Memory usage while validating

I am facing an issue with huge memory usage while validating large XML files (about 90 MB).
I might be doing something wrong but there seems to be a memory leakage because the used memory grows after every file and never clears up until I restart the server.

My setup is an HTTP server with a handler that validates an uploaded XML file against a provided XSD.

My main func looks something like this:

func main() {
  // init xsdvalidate
  // xsdvalidate.InitWithGc(2 * time.Minute) // didn't help
  err = xsdvalidate.Init()
  defer xsdvalidate.Cleanup()
  if err != nil {
  log.WithError(err).Fatalf("could not init xsdvalidate")
  }
  ...
}

and inside my handler I am doing something like this:

func (it *service) handler(file io.ReadCloser, filesize int64) error {
  defer file.Close()
  
  xsdHandler, err := xsdvalidate.NewXsdHandlerMem(it.meteringObjectsXSD, xsdvalidate.ParsErrDefault) // it.meteringObjectsXSD is a small XSD file (3KB) read into a byte slice
  defer xsdHandler.Free()
  if err != nil {
    return err
  }

  fileContent := make([]byte, filesize)
  _, err = io.ReadFull(file, fileContent)
  if err != nil {
    return err
  }

  err = xsdHandler.ValidateMem(fileContent, xsdvalidate.ParsErrDefault)
  if err != nil {
    return err
  }
  ...
}

(I also tried creating the XSD handler only once inside main and injecting it into my service but that didn't help either)

When uploading a 90MB XML file and validating it with the XSD the memory usage looks like this:

After uploading it again the memory usage grows almost twice the size:

Unfortunately, the memory never frees up.

Now when using the same service again but commenting out the validation part: err = xsdHandler.ValidateMem(fileContent, xsdvalidate.ParsErrDefault) the memory usage looks like this

After the first time uploading the 90MB XML file:

After the second time uploading the file:

The file was still read into memory but after the GC kicks in it will be all freed up.

Any idea why the memory usage is so high and why it's only growing and never being freed up?