JIT model multiple inputs about gotch HOT 9 CLOSED

@fteufel

A fix #20 done on master and v0.3.8. Now you can cast output to []ts.Tensor as below. If that what you expect, please close this issue.

package main

import (
	"fmt"
	"log"

	"github.com/sugarme/gotch"
	ts "github.com/sugarme/gotch/tensor"
)

func main() {
	input_ids, _ := ts.Ones([]int64{2, 73}, gotch.Int64, gotch.CPU)
	input_mask, _ := ts.Ones([]int64{2, 73}, gotch.Int64, gotch.CPU)
	input_ids_ival := ts.NewIValue(*input_ids)
	input_mask_ival := ts.NewIValue(*input_mask)

	inputs := []ts.IValue{*input_ids_ival, *input_mask_ival}

	model, err := ts.ModuleLoad("distilled_scripted.pt")
	if err != nil {
		log.Fatal(err)
	}

	prediction, err := model.ForwardIs(inputs)
	if err != nil {
		log.Fatal(err)
	}

	xs := prediction.Value().([]ts.Tensor)
	for _, x := range xs {
		fmt.Printf("%i", &x)
	}
}

// Output:

TENSOR INFO:
        Shape:          [2 6]
        DType:          float32
        Device:         {CPU 1}
        Defined:        true

TENSOR INFO:
        Shape:          [2 70 37]
        DType:          float32
        Device:         {CPU 1}
        Defined:        true

TENSOR INFO:
        Shape:          [2 70]
        DType:          int64
        Device:         {CPU 1}
        Defined:        true

from gotch.

Just managed to make the multi-inputs work.

input_ids, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_mask, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
inputs := []ts.Tensor{*input_ids,*input_mask}

model, _ := ts.ModuleLoad("distilled_scripted.pt")

predictTmp,err := model.ForwardTs(inputs) //ForwardTs(Sclice)
fmt.Println(err)
fmt.Printf("%i", predictTmp)

However I'm still clueless for the multi-outputs. I coudn't find any function that allows the output to be a slice.
Libtorch API Error: forward did not return a tensor

from gotch.

@fteufel

Although JIT API has not been complete yet, gotch provides 2 APIs to forward pass CModule:

• func (cm *CModule) ForwardTs(tensors []Tensor) (*Tensor, error)
• func (cm *CModule) ForwardIs(ivalues []IValue) (*IValue, error)

Have a look at JIT unit test for how to use.

Also, there 2 separate examples of using JIT in example folder: inference and train

Not sure if that what you're after?

from gotch.

Just managed to make the multi-inputs work.

input_ids, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_mask, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
inputs := []ts.Tensor{*input_ids,*input_mask}

model, _ := ts.ModuleLoad("distilled_scripted.pt")

predictTmp,err := model.ForwardTs(inputs) //ForwardTs(Sclice)
fmt.Println(err)
fmt.Printf("%i", predictTmp)

However I'm still clueless for the multi-outputs. I coudn't find any function that allows the output to be a slice.
Libtorch API Error: forward did not return a tensor

Have you handled error in model, _ := ts.ModuleLoad("distilled_scripted.pt")? This error coming from C and just make sure the JIT module is loaded properly.

from gotch.

Hi, Thanks for the hints!

the error was coming from ForwardTs, because the python signature of the model looks like this:
forward(input_ids, input_mask) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor] which ForwardTs can't handle.

Converting the input tensors to IValue and using ForwardIs fixed it:

input_ids, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_mask, _ := ts.Ones([]int64{2,73}, gotch.Int64, gotch.CPU)
input_ids_ival := ts.NewIValue(*input_ids)
input_mask_ival := ts.NewIValue(*input_mask)

inputs := []ts.IValue{*input_ids_ival,*input_mask_ival}


model, _ := ts.ModuleLoad("distilled_scripted.pt")
prediction,err := model.ForwardIs(inputs) 

Now I would just figure out how to get the three Tensors back out of the prediction IValue.

from gotch.

@fteufel

prediction is IValue type. You may try to use func (iv *IValue) Value() interface{} and cast return value to your expected type.

Something like

var output []ts.Tensor
output = prediction.Value().([]ts.Tensor)

from gotch.

prediction.Kind() gives []tensor.IValue, that should be ok I suppose.
The expected type casting however is not doing it.
panic: interface conversion: interface {} is []interface {}, not []tensor.Tensor

I'm really new to Go, as you can tell - and the concept of interface especially. I actually just have a codebase that uses tfgo and I need to swap out the core model for pytorch.
Thanks a lot for your help!

from gotch.

@fteufel

I think the issue is related to type casting inside gotch JIT itself and it may fall into one of the unsupported cases.

As you mentioned output is Tuple[torch.Tensor, torch.Tensor, torch.Tensor], gotch may cast output to 'GenericList' and end up with []inferface{} instead of 'TensorList' which is []Tensor.

Can you try to cast in []interface{} type. I would suspect some error: IValueFromC method call - GenericList case: Unsupported item type....

var output []interface{}
output = prediction.Value().([]interface{})

// Or just
output := prediction.Value()

If you can share your JIT module .pt or any simplified version, I can trace up and may add more support to the library.

from gotch.

Hi,

gave it a try:
Casting works, yielding output type: []interface {} as expected.

Here's the jit model:
https://mega.nz/file/uZ0DVQ7b#oDDerv_8mJmj1ypC_CX8OaWBxfYe4dXDvHGVgOiwCks
It's rather large (essentially Bert+custom prediction head), but it's probably better to check the real thing. As stated, it takes two Int64 tensors of size (batch_size, 73) as inputs.

Update:
I changed the signature in Python to List[torch.Tensor, torch.Tensor, torch.Tensor]. Now

output := prediction.Value().([]ts.Tensor)

works. Easy fix for the problem, as there is no advantage of using Tuple over List really.

from gotch.