freeconf / restconf Goto Github PK

When doing a POST with content-type "application/yang-data+json", the restconf server responds with a correctly formated response that conforms to our yang model.

But if the request is made with application/yang-data+xml. then the response is send as application/yang-data+json, but with the top level attribute stripped out.

In server_test.go the test in line 54 seems to be wrong... the return type suggest this was meant to do a "GET" operation on the "car" data, but what the test does is to retrieve the schema of "car", exactly as the test beginning in line 51.

This can also be seen in the file testdata/gold/car.json, which doesn't contain JSON data but a YANG file.

I guess the test should be a line like:
req, _ := http.NewRequest("GET", fmt.Sprintf("%s/restconf/data/car", addr), nil)

I'm not 100% sure if this is RFC compliant but I think the logic would be valid:
Restconf could default to always return data with yang-data+json.
Unless the client sends "Accept: yang-data+xml" and nothing else. In this case, restconf would send the reply in xml.

bascically:

So the logic is only based on the Accept header and not on the Content-Type.

Hi!

I'm still tweaking the "Next step" and "RESTCONF client" example. This time I want to start the car through the RPC and then see that status reflected in the car:running leaf:

import (
	"fmt"
	"strings"
	"testing"

	"github.com/freeconf/examples/car"
	"github.com/freeconf/restconf"
	"github.com/freeconf/restconf/client"
	"github.com/freeconf/restconf/device"
	"github.com/freeconf/yang/fc"
	"github.com/freeconf/yang/node"
	"github.com/freeconf/yang/nodeutil"
	"github.com/freeconf/yang/source"
)

func connectClient() {
	// YANG: just need YANG file ietf-yang-library.yang, not the yang of remote system as that will
	// be downloaded as needed
	ypath := restconf.InternalYPath

	// Connect
	proto := client.ProtocolHandler(ypath)
	dev, err := proto("http://localhost:9998/restconf")
	if err != nil {
		panic(err)
	}

	// Get a browser to walk server's management API for car
	car, err := dev.Browser("car")
	if err != nil {
		panic(err)
	}
	root := car.Root()
	defer root.Release()

        // Subscribe to notifications
        sel, err = root.Find("update")
	if err != nil {
		panic(err)
	}
	defer sel.Release()
	unsub, err := sel.Notifications(func(n node.Notification) {
		msg, err := nodeutil.WriteJSON(n.Event)
		if err != nil {
			panic(err)
		}
		fmt.Println(msg)
	})
	if err != nil {
		panic(err)
	}
	defer unsub()

	// Wait a bit just to ensure we receive the car started notification
	time.Sleep(time.Second)

	// Start car
	sel, err = root.Find("start")
	if err != nil {
		panic(err)
	}
	if _, err = sel.Action(nil); err != nil {
		panic(err)
	}

	println("Waiting for some notifications...")
	for i := 0; i < 10; i++ {
		sel, err = root.Find("running")
		if err != nil {
			panic(err)
		}
		running, err := sel.Get()
		if err != nil {
			panic(err)
		}
		fmt.Printf("Is the car running? %v\n", running)
		time.Sleep(time.Second)
	}
}

func main() {
	connectClient()
}

Output:

Waiting for some notifications...
Is the car running? false
{"event":"carStarted"}
Is the car running? false
Is the car running? false
Is the car running? false
Is the car running? false
Is the car running? false
{"event":"flatTire"}
{"event":"carStopped"}
Is the car running? false
Is the car running? false
Is the car running? false
Is the car running? false

I understand that the configuration is retrieved from the server and cached locally. I just wonder how I can forcibly re-retrieve the configuration? I appreciate your time!

Hi!

I'm trying to set up a RESTCONF client through FREECONF. I'm working through the Car example and aim to present the user with configuration, update the configuration, and display the new configuration. Unfortunately, after a configuration update I fail to display the full configuration again.

My client code:

// Courtesy to https://freeconf.org/docs/examples/restconf-client/

package main

import (
	"github.com/freeconf/restconf"
	"github.com/freeconf/restconf/client"
	"github.com/freeconf/yang/nodeutil"
)

func connectClient() {

	// YANG: just need YANG file ietf-yang-library.yang, not the yang of remote system as that will
	// be downloaded as needed
	ypath := restconf.InternalYPath

	// Connect
	proto := client.ProtocolHandler(ypath)
	dev, err := proto("http://localhost:8080/restconf")
	if err != nil {
		panic(err)
	}

	// Get a browser to walk server's management API for car
	car, err := dev.Browser("car")
	if err != nil {
		panic(err)
	}
	root := car.Root()
	defer root.Release()

	actual, err := nodeutil.WritePrettyJSON(car.Root())
	if err != nil {
		panic(err)
	}
	println("========== initial config ==========")
	println(actual)

	n, err := nodeutil.ReadJSON(`{"speed":50}`)
	if err != nil {
		panic(err)
	}

	err = root.UpsertFrom(n)
	if err != nil {
		panic(err)
	}

	actual, err = nodeutil.WritePrettyJSON(car.Root())
	if err != nil {
		panic(err)
	}
	println("========== config after setting speed ==========")
	println(actual)

	n, err = nodeutil.ReadJSON(`{"pollInterval":50}`)
	if err != nil {
		panic(err)
	}
	err = root.UpsertFrom(n)
	if err != nil {
		panic(err)
	}
	actual, err = nodeutil.WritePrettyJSON(car.Root())
	if err != nil {
		panic(err)
	}
	println("========== config after setting pollInterval ==========")
	println(actual)
}

func main() {
	connectClient()
}

The output:

========== initial config ==========
{
"speed":50,
"pollInterval":50,
"running":false,
"miles":0,
"lastRotation":0,
"tire":[
  {
    "pos":0,
    "size":"H15",
    "worn":false,
    "wear":100,
    "flat":false},
  {
    "pos":1,
    "size":"H15",
    "worn":false,
    "wear":100,
    "flat":false},
  {
    "pos":2,
    "size":"H15",
    "worn":false,
    "wear":100,
    "flat":false},
  {
    "pos":3,
    "size":"H15",
    "worn":false,
    "wear":100,
    "flat":false}]}
========== config after setting speed ==========
{
"speed":50,
"tire":[
  {
    "size":"H15"},
  {
    "size":"H15"},
  {
    "size":"H15"},
  {
    "size":"H15"}]}
========== config after setting pollInterval ==========
{
"pollInterval":50,
"tire":[
  {
    "size":"H15"},
  {
    "size":"H15"},
  {
    "size":"H15"},
  {
    "size":"H15"}]}

I had expected that the nodeutil.WritePrettyJSON(car.Root()) calls would return the full config, not just the updated part + all configuration further dow the hierarchy. I'd be happy to learn how to obtain the full config (and why the above output is sensible).

Could you help me out? Looking forward to your (usually impressively swift) reply!

Comply with media types outlines in RESTCONF RFC
https://datatracker.ietf.org/doc/html/rfc8040#section-11.3
Currently freeconf mostly ignores media types except in a few rare instances

Hello, I try to use the following yang file to test config-false:

module hello {
	leaf message {
		type string;
	}

	leaf config-false-message {
		type string;
		config false;
	}
}

And then I POST to http://localhost:8080/restconf/data/hello: with the body:

{"config-false-message":"test"}

I observed that the POST method successfully edited the "config-false-message".

However, based my understanding from dicussion and RFC 6020-YANG, the leaf marked with config false cannot be manipulated by POST or other method.

Could you please confirm if this behavior is expected?

Thank you very much.

Content-Type set in util::handleErr() is overwritten by function Error() in net/http/server.go

Here is the code in server.go caussing the problem:

func Error(w ResponseWriter, error string, code int) {
	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
	w.Header().Set("X-Content-Type-Options", "nosniff")
	w.WriteHeader(code)
	fmt.Fprintln(w, error)
}

What I was doing in util::handleErr() was to not call http.Error(). I was doing this:

w.Header().Set("Content-Type", string(mime))
w.WriteHeader(code)
w.Header().Set("X-Content-Type-Options", "nosniff")
fmt.Fprintln(w, msg)

I am using go version go1.17.6 windows/amd64

When I try to go get freeconf/restconf I get the following error:

go get  github.com/freeconf/restconf
# github.com/freeconf/yang/val
..\..\..\..\go\pkg\mod\github.com\freeconf\[email protected]\val\conv.go:596:36: rv.CanInt undefined (type reflect.Value has no field or method CanInt)
..\..\..\..\go\pkg\mod\github.com\freeconf\[email protected]\val\conv.go:689:36: rv.CanUint undefined (type reflect.Value has no field or method CanUint)

The current content-type in a POST response is set to application/json.

According to RFC8040 it must be application/yang-data+xml or application/yang-data+json. In our case it is application/yang-data+json.

See https://www.rfc-editor.org/rfc/rfc8040.html#section-3.2

notification should be wrapped with

{
     "ietf-restconf:notification" : {
         "eventTime": "2008-07-08T00:01:00Z",
          "event": {  
                 payload here
           }
      }
}

see https://datatracker.ietf.org/doc/html/rfc7950#section-7.16.3

The RFC shows that the data being posted as inputs to a RPC call should be embeded in an object called "input" as shown below

{
        "example-ops:input" : {
          "delay" : 600,
          "message" : "Going down for system maintenance",
          "language" : "en-US"
        }
      }

It seems that freeconf only supports posting the payload directly such as

{
          "delay" : 600,
          "message" : "Going down for system maintenance",
          "language" : "en-US"
 }

Response content type not correctly set after XML pull request merge.

The current code on master is calling browser_handler::setContentType(). See

It must calls util::setResponseContentType() .

At the moment its not possible to use freeconf/restconf without it opening a HTTP server in the background, which makes it difficult/impossible to integrate into existing services.

#36 should fix this

Hi!

If I interpret the supported RFC 8525 correctly, the container modules-state from restconf/data/ietf-yang-library is deprecated in favor of a container yang-library. However, when I call the endpoint on the Car example, the reply contains modules-state but not yang-library. Is this a bug?

If I understand correctly, by adding CAs in fc-restconf.web.tls.ca, a client will have its cert validated against any of the CAs according to

Does this mean that if a new CA needs to be added, then the server needs to be restarted?
it would be nice if we could set this to RequireAnyClientCert or RequestClientCert and then be able to manually validate the cert using a Filter. This way, the application using restconf would be able to handle requests and validate the certs using a dynamic list that is handled outside of restconf

Or any other suggestions to achieve the same goal?

When returning an error with
// Option #2 - enhanced but still an 401 error return fmt.Errorf("Bad ACL %w", fc.UnauthorizedError)
It would be useful if restconf could create a payload according to the model of ietf-restconf:errors

{
  "ietf-restconf:errors": {
    "error": [
      {
        "error-type": "protocol",
        "error-tag": "lock-denied",
        "error-message": "Lock failed; lock already held"
      }
    ]
  }
}

The server is returning a 200 OK even if set an error. The error handling in https://github.dev/freeconf/restconf/util.go calls http.Error(w, msg, code) and I clearly see while debugging that the code is set to 401:

func handleErr(compliance ComplianceOptions, err error, r *http.Request, w http.ResponseWriter) bool {
	if err == nil {
		return false
	}
	fc.Debug.Printf("web request error [%s] %s %s", r.Method, r.URL, err.Error())
	msg := err.Error()
	code := fc.HttpStatusCode(err)
	if !compliance.SimpleErrorResponse {
		errResp := errResponse{
			Type:    "protocol",
			Tag:     decodeErrorTag(code, err),
			Path:    decodeErrorPath(r.RequestURI),
			Message: msg,
		}
		var buff bytes.Buffer
		fmt.Fprintf(&buff, `{"ietf-restconf:errors":{"error":[`)
		json.NewEncoder(&buff).Encode(&errResp)
		fmt.Fprintf(&buff, `]}}`)
		msg = buff.String()
	}
	http.Error(w, msg, code)
	return true
}

http.Error(w, msg, code) generates the error message http: superfluous response.WriteHeader call from github.com/freeconf/restconf.handleErr (util.go:91)

This is the http.Error() function:

func Error(w ResponseWriter, error string, code int) {
	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
	w.Header().Set("X-Content-Type-Options", "nosniff")
	w.WriteHeader(code)
	fmt.Fprintln(w, error)
}

The error occurs when calling w.WriteHeader(code). The variable wroteHeader in the response is already set and cannot be changed.

func (w *response) WriteHeader(code int) {
	if w.conn.hijacked() {
		caller := relevantCaller()
		w.conn.server.logf("http: response.WriteHeader on hijacked connection from %s (%s:%d)", caller.Function, path.Base(caller.File), caller.Line)
		return
	}
	if w.wroteHeader {
		caller := relevantCaller()
		w.conn.server.logf("http: superfluous response.WriteHeader call from %s (%s:%d)", caller.Function, path.Base(caller.File), caller.Line)
		return
	}
.
.
.

This is caused by the sendOutput done in freeconf/restconf/browser_handler.go at line 171 before the handleErr and line 172.

sendOutput calls fmt.Fprintf at line 198. This is calling WriteHeader eventually.

when adding a leaf, in the car example, with "type empty", I get the error: typedef empty not found

In the case where content-type is set to yang-data+xml in a request, we would need restconf to be able to parse the XML data into the yang model.
This is irrelevant from the Accept header.

I don't think we need to force the response to XML. The response can still be sent with yang-data+json, as long as the Accept header contains it. I will address this in a different issue.

Hey!

I've been playing around with FreeCONF for some days. In general I like it, and it really helps me to get a better understanding of YANG and RESTCONF. Today I tried to patch/put/post data to a binary leaf, without success. Perhaps you could point me to a solution?

Approach: I follow the "Getting started" tutorial and change the type of message into type binary;. If I start the server, I get the following error:

Starting server...
Traceback (most recent call last):
  File ".../freeconf/server.py", line 62, in <module>
    browser = main()
  File ".../freeconf/server.py", line 55, in main
    dev.apply_startup_config_file("./startup.json")
  File ".../freeconf/.venv/lib/python3.10/site-packages/freeconf/device.py", line 26, in apply_startup_config_file
    self.__apply_startup_config_stream(self.driver.fs.new_rdr_file(configFile))
  File ".../freeconf/.venv/lib/python3.10/site-packages/freeconf/device.py", line 36, in __apply_startup_config_stream
    self.driver.g_device.ApplyStartupConfig(req)
  File ".../freeconf/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1176, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File ".../freeconf/.venv/lib/python3.10/site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "cannot coerse 'string' to binary value"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-04-09T14:41:51.964747905+02:00", grpc_status:2, grpc_message:"cannot coerse \'string\' to binary value"}"
>

So that didn't go very well. For now, I can remove the "message" key from ./startup.json (so "hello: {}") and start the server again without issues. Then, I try to POST a base64-encoded string as described in RFC7950 as follows:

# base64 encoding of "Hello, world!"
curl -X POST -d '{"message": "SGVsbG8sIHdvcmxkIQ=="}' http://localhost:8080/restconf/data/hello

which returns

cannot coerse 'string' to binary value

Is base64 decoding of JSON strings for binary types not implemented? If it is, what am I doing wrong?

Looking forward to your help.

Restconf go.mod does not refer to the latest freeconf/yang with commit:

freeconf/yang@991b775

With this change freecong/nodeutil/json_rdr.go is returning two values and the current code in browser_handler.go is also expecting two values but restconf/go.mod is pointing to an older version of freeconf/yang.

Here is an example of an errors body in xml:

<errResponseWrapper>
   <errors>
      <error>
         <error-type>protocol</error-type>
         <error-tag>access-denied</error-tag>
         <error-path>ietf-sztp-bootstrap-server:report-progress</error-path>
         <error-message>Bad ACL not authorized</error-message>
      </error>
   </errors>
</errResponseWrapper>

errResponseWrapper should not be there and xmlns must be set to "urn:ietf:params:xml:ns:yang:ietf-restconf"

ApplyStartupConfigFile from local.go file currently panics when something goes wrong opening the config file (for instance when we provide an incorrect startup file path). This prevents applications from properly handling errors. It is also not in line with the function signature, which suggests it should return errors when something goes wrong

Current implementation:
https://github.com/freeconf/restconf/blob/master/device/local.go#L112

func (self *Local) ApplyStartupConfigFile(fname string) error {
	cfgRdr, err := os.Open(fname)
	defer cfgRdr.Close()
	if err != nil {
		panic(err)
	}
	return self.ApplyStartupConfig(cfgRdr)
}

For example,

POST /restconf/operations/example-ops:reboot HTTP/1.1
      Host: example.com
      Content-Type: application/yang-data+json
      {
        "example-ops:input" : {
          "delay" : 600,
          "message" : "Going down for system maintenance",
          "language" : "en-US"
        }
      }

does not work. We get an error saying "Expected format: http://server/restconf[=device]/operation/module:path"
But the RFC does not mention anything about "=device". Also, looking at server.go#L146, it seems that "operations" is simply not implemented and the error message is misleading

The ServeHTTP() function in server.go does create its own context based on context.Background(). Is there a special reason not to use r.Context() ? Using the http.Request context (which is normally context.Background()) should make it easier to handle situations with http based context interruptions.

Hi, I was trying to figure out how to add a list to my restconf interface and I eventually found your more comprehensive car example. I think adding a couple of examples for lists and maps in the README would improve adoption. I would also clarify that reflection doesn't work in that case and highlight the parts where it does. I would also show how to update and query list/maps using curl. I would provide you with my example but you already have a better one in your car.yaml.

Let me know if you would like me to contribute with an example.

I am no expert on reflection in go but I was wondering if you had any plan to support reflection for base types such as the example that follows. I was surprised to have to use a custom parser (albeit a simple one) to parse a list of string and integers.

    list bar {
        key foo;
        leaf foo {
            type int32 {
                range "0..120";
            }
        }
        leaf baz {
            type string;
        }
    }

the ApplyOptions() function in stock/web.go creates a new http server every times new parameters are set via restconf.

I think it should use Shutdown()/Close() on the old server before opening the new one.

Maybe with something like the following change (this one has a hardcoded 10 second wait for graceful shutdown):

diff --git a/stock/web.go b/stock/web.go
index 6cbdf4b..192f38d 100644
--- a/stock/web.go
+++ b/stock/web.go
@@ -42,6 +42,14 @@ func (service *HttpServer) ApplyOptions(options HttpServerOptions) {
        if options == service.options {
                return
        }
+       if service.Server != nil {
+               ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+               go func() {
+                       _ = service.Server.Shutdown(ctx)
+                       cancel()
+               }()
+               _ = service.Server.Close()
+       }
        service.options = options
        service.Server = &http.Server{
                Addr:           options.Port,

I get this error when I have
Device (uses choose) <- Client Gateway <- REST GET
I assume the client doesn't properly support Choose

json object lack the "prefix" from the modules they are imported from to avoid name collisions.

note: beyond the technical challenge, this breaks compatibility fairly significantly. might consider a mode where this be be opt-out

I noticed that when restconf panics due to the use of a wrong type for reflection it does not print the name of the field that caused the problem but only the type. I think it could simplify debugging to have access to that information

It is meant to be directed to web app, which have have any endpoint

You reference should be updated to latest version to pull in the new union fixes.

Restconf does not set the body in the right format (XML) for errors with a body when the accept in the request is XML. The response is always in JSON.

The merge request for XML were including the required code in util.go but it has not been taken.

Support verifying that a leaf of type empty is present or not in ActionRequest.Input.

Currently, calling req.Input.GetValue(...) will always return nil whatever the leaf is there or not.

Having something like req.Input.HasLeaf(name) would be nice.