ferranbt / fastssz Goto Github PK

It seems like sszgen is not respecting the slice type when it is an alias of an integer. I think this has to do with one of the previous fixes that took advantage of the implicit conversion between the two.

package main

import (
        "encoding/json"
        "fmt"
        "strconv"
)

type U64Str uint64

func (i U64Str) MarshalJSON() ([]byte, error) {
        return json.Marshal(strconv.FormatUint(uint64(i), 10))
}

func (i *U64Str) UnmarshalJSON(b []byte) error {
        var s string
        if err := json.Unmarshal(b, &s); err != nil {
                return err
        }
        value, err := strconv.ParseInt(s, 10, 64)
        if err != nil {
                return err
        }
        *i = U64Str(value)
        return nil
}

type Foo struct {
        Bar []U64Str `json:"bar" ssz-max:"1024"`
}

func main() {
        input := `{"bar": ["1", "2", "3"]}`

        var val Foo
        if err := json.Unmarshal([]byte(input), &val); err != nil {
                panic(err)
        }
        fmt.Println("success")
}

$ sszgen --path main.go
$ go run .
./main_encoding.go:62:11: cannot use ssz.ExtendUint64(f.Bar, num) (value of type []uint64) as type []U64Str in assignment
./main_encoding.go:62:28: cannot use f.Bar (variable of type []U64Str) as type []uint64 in argument to ssz.ExtendUint64
./main_encoding.go:97:20: cannot use i (variable of type U64Str) as type uint64 in argument to hh.AppendUint64

This is an issue to track the Capella fork.

The following example fails due to the array length not being a *ast.BasicLit.

package badone

const ArrayLength = 1

type S struct {
	Arr [ArrayLength]byte `ssz-size:"10"`
}

It would be handy to be able to support such cases (this is coming up when sszgen-annotating geth's Header struct).

I am using this lib to get multiproofs for block_roots in the BeaconState. I download the BeaconState from Prysm, unmarshal the ssz bytes into a BeaconStateBellatrix object, then call GetTree on the object in preparation to get the proof. I keep running out of memory when calling GetTree.

I have made a demo repo to illustrate the issue: https://github.com/claravanstaden/beacon-state-scratchpad The beacon state as download from Prysm is included as a file.

Please let me know if I am doing something wrong?

type MasterAccumulator struct {
	HistoricalEpochs [][]byte `ssz-max:"1897,32"`
}

Must I create a struct for ssz vector? If Yes, what's the difference about a ssz container only have a ssz vector field vs a raw ssz vector? @ferranbt

In the current generated code, a malformed message may include a variable length value offset pointing to a location before the end of the encoded fixed-size values. In this instance UnmarshalSSZ will unmarshal the encoded bytes without complaint, resulting in a corrupt runtime value. An example encoded Attestation value can be downloaded from:
https://github.com/prysmaticlabs/ethereumapis/blob/check-variable-offset/eth/v1/testdata/invalid-offset.attestation.ssz

Phase 1 has a few new objects (e.g. ShardTransition.shard_data_roots and CompactCommittee.pubkeys) that requires [][]byte and ssz-max. To be specific ShardTransition.shard_data_roots is ssz-max:"12,32" and CompactCommittee.pubkeys is ssz-max:"2048,48"

I dont think [][]byte and ssz-max are supported as of today. When I try to generate for that combination, it fails

go version go1.19 linux/amd64

Hi, it seems like there is a regression against master. I am able to run sszgen@bc5fefe without error, but sszgen@fadd032 throws an error:

$ go install github.com/ferranbt/fastssz/sszgen@fadd032
$ rm -f types/builder_encoding.go types/signing_encoding.go
$ sszgen --path types --include ../go-ethereum/common/hexutil --exclude-objs ExtraData --objs Eth1Data,BeaconBlockHeader,SignedBeaconBlockHeader,ProposerSlashing,Checkpoint,AttestationData,IndexedAttestation,AttesterSlashing,Attestation,Deposit,VoluntaryExit,SyncAggregate,ExecutionPayloadHeader,VersionedExecutionPayloadHeader,BlindedBeaconBlockBody,BlindedBeaconBlock,RegisterValidatorRequestMessage,BuilderBid,SignedBuilderBid,SigningData,forkData,transactions
[ERR]: failed to encode SignedBuilderBid: type Signature not found
$ go install github.com/ferranbt/fastssz/sszgen@bc5fefe
$ sszgen --path types --include ../go-ethereum/common/hexutil --exclude-objs ExtraData --objs Eth1Data,BeaconBlockHeader,SignedBeaconBlockHeader,ProposerSlashing,Checkpoint,AttestationData,IndexedAttestation,AttesterSlashing,Attestation,Deposit,VoluntaryExit,SyncAggregate,ExecutionPayloadHeader,VersionedExecutionPayloadHeader,BlindedBeaconBlockBody,BlindedBeaconBlock,RegisterValidatorRequestMessage,BuilderBid,SignedBuilderBid,SigningData,forkData,transactions

This is running against mergemock@c7e64af.

I get the following error during code generation.

panic: ast type '*ast.InterfaceType' not expected

goroutine 1 [running]:
main.(*env).parseASTFieldType(0xc000129e28, 0xc000016570, 0xf, 0x0, 0x0, 0x5e4a08, 0xc00000c2a0, 0x0, 0xc000129b40, 0x412fbb)
        /home/matematik/ethereum/fastssz/sszgen/main.go:1043 +0x14cc
main.(*env).encodeItem(0xc000129e28, 0xc000016570, 0xf, 0x0, 0x0, 0xc000110c00, 0x4, 0x8)
        /home/matematik/ethereum/fastssz/sszgen/main.go:825 +0x34f
main.(*env).generateIR(0xc000129e28, 0xc000010d50, 0xc000129dc8)
        /home/matematik/ethereum/fastssz/sszgen/main.go:794 +0x835
main.encode(0x7ffefa5f62c0, 0x7, 0x6eb740, 0x0, 0x0, 0x0, 0x0, 0x6eb740, 0x0, 0x0, ...)
        /home/matematik/ethereum/fastssz/sszgen/main.go:95 +0x205
main.main()
        /home/matematik/ethereum/fastssz/sszgen/main.go:43 +0x3cb
exit status 2

command: go run ~/ethereum/fastssz/sszgen/*.go --path ./types
project: https://github.com/MariusVanDerWijden/go-eth2lc

This is probably due to some nested types right? Do all types need to be available in the project or can you import types from other projects?

Hi there. Noticed something that I believe is a bug in sszgen.

Given a structure with a uint64 array field, the field will unmarshal to a nil value if (1) the array is empty and (2) the variable that calls UnmarshalSSZ is not initialized. And if you swap uint64 for byte it works as expected.

See the following program:

package main

import "fmt"

type Foo struct {
  Bar []uint64 `json:"bar" ssz-max:"1024"`
}

func main() {
  val := Foo{Bar:[]uint64{}}
  var dec Foo // not initialized

  enc, err := val.MarshalSSZ()
  if err != nil {
    panic(err)
  }
  err = dec.UnmarshalSSZ(enc)
  if err != nil {
    panic(err)
  }

  fmt.Println("val.Bar nil?", val.Bar == nil)
  fmt.Println("dec.Bar nil?", dec.Bar == nil)
}

When executed, this will output:

$ sszgen --path main.go && go run .
val.Bar nil? false
dec.Bar nil? true

On the other hand, if dec.Bar is initialized with anything (even an empty array), it will behave as expected:

func main() {
  val := Foo{Bar:[]uint64{}}
  dec := Foo{Bar:[]uint64{}} // initialized

  enc, err := val.MarshalSSZ()
  if err != nil {
    panic(err)
  }
  err = dec.UnmarshalSSZ(enc)
  if err != nil {
    panic(err)
  }

  fmt.Println("val.Bar nil?", val.Bar == nil)
  fmt.Println("dec.Bar nil?", dec.Bar == nil)
}

When executed, this will output:

$ sszgen --path main.go && go run .
val.Bar nil? false
dec.Bar nil? false

Fixes: flashbots/go-boost-utils#23

Example

// a/stuff.go
package "a"

type Thing struct {}

type FooBar struct {}

// b/stuff.go
package "b"

import "a"

type Thing struct {
  Item *a.FooBar
}

type Sandwhich struct {
  Size uint64

  Item *a.FooBar
}

Running sszgen on b.Sandwhich causes conflicts between unrelated items in package a and package b.

[ERR]: two structs share the same name Thing

goos: linux
goarch: amd64
pkg: github.com/ferranbt/fastssz/spectests
BenchmarkMarshalGoSSZ
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x59135b]

goroutine 33 [running]:
github.com/ferranbt/fastssz/spectests.readValidGenericSSZ(0x0, 0x5faf18, 0x52, 0x5c3c40, 0xc000024040, 0xc00027d720, 0x4cae0f, 0xbfbeb46de790ec28)
        /tmp/fastssz/spectests/structs_test.go:406 +0x28b
github.com/ferranbt/fastssz/spectests.BenchmarkMarshalGoSSZ(0xc000120000)
        /tmp/fastssz/spectests/structs_test.go:285 +0x6e
testing.(*B).runN(0xc000120000, 0x1)
        /snap/go/6123/src/testing/benchmark.go:191 +0xe8
testing.(*B).run1.func1(0xc000120000)
        /snap/go/6123/src/testing/benchmark.go:231 +0x57
created by testing.(*B).run1
        /snap/go/6123/src/testing/benchmark.go:224 +0x7d
exit status 2
FAIL    github.com/ferranbt/fastssz/spectests   0.007s
FAIL

To reproduce

git clone [email protected]:ferranbt/fastssz.git --recursive
go test -v ./spectests/... -run=XXX -bench=.

A simple struct using only primitive types:

type Struct1 struct {
    Data [][]byte `ssz-size:"128,20"`
}

encodes fine, but if the type is abstracted:

type Slice []byte
type Struct2 struct {
    Data []Slice `ssz-size:"128,20"`
}

An attempt to create the encoding with sszgen returns an error:

[ERR]: failed to encode Slice: no ssz-size or ssz-max tags found for element. tag=

Similarly, attempting to do the same with a fixed-size array:

type FixedSizeArray [20]byte
type Struct3 struct {
    Data []FixedSizeArray `ssz-size:"128,20"`
}

returns

[ERR]: failed to encode FixedSizeArray: no ssz-size or ssz-max tags found for element. tag=

This is using current master.

In order to generate proofs for beacon states, we need uint and bitfield support. Currently, this panics with unimplemented when trying to generate fastssz code on the Prysm BeaconStateAltair struct due to uint and bitfields missing.

See: prysmaticlabs/prysm#9402

Prysm's implementation of state hash tree root is defined here: https://github.com/prysmaticlabs/prysm/blob/53f70308715bb9d2f0a97c327becef9304daf890/beacon-chain/state/v1/state_trie.go#L201

The result of the generated ssz method (*BeaconState).HashTreeRoot() returns a different result. The Prysm implementation is known to work and passes spectests.

I do not know what the difference is between the two methods.

What version of Go are you using (go version)?

go version go1.17.10 darwin/amd64

Does this issue reproduce with the latest release?

Yes. sszgen version 0.1.1

What did you expect to see?

Correctly generated code.

What did you see instead?

Broken code.

For example given the following code

// ForkData provides data about a fork.
type ForkData struct {
	// Current version is the current fork version.
	CurrentVersion Version `ssz-size:"4"`
	// GenesisValidatorsRoot is the hash tree root of the validators at genesis.
	GenesisValidatorsRoot Root `ssz-size:"32"`
}

And running sszgen --path . --objs ForkData will produce a file which will contain an invalid function:

// MarshalSSZTo ssz marshals the ForkData object to a target array
func (f *ForkData) MarshalSSZTo(buf []byte) (dst []byte, err error) {
	dst = buf
	offset := int(4)

	return
}

The offset variable above is never used and the compiler will produce an error.

As of now, sszgen only includes the hash of the input code in the header of the generated code. It helps to figure out when the input code has been updated.

Sszgen should also include the version of the generator in the header.

Cannot Marshal / SizeSSZ when a struct has a nil pointer.

For example, if BeaconBlock.Body is nil, this will panic.

// SizeSSZ returns the ssz encoded size in bytes for the BeaconBlock object
func (b *BeaconBlock) SizeSSZ() (size int) {
        size = 76

        // Field (3) 'Body'
        size += b.Body.SizeSSZ()

        return
}

Using commit f7a73f0

Hi,

Would it be conceivable to make ssz-size / ssz-max accept variable names that would get inserted in the generated code, instead of hardcoded values?

The idea behind this is that the Ethereum spec comes with configurable variables, and it's currently impossible (or at least quite complicated and repetitive) to write code that can be compatible with multiple presets at the same time.

For example, I'd like to be able to write something like this:

type ExecutionPayload struct {
	Withdrawals   []*Withdrawal           `ssz-max-var:"MAX_WITHDRAWALS_PER_PAYLOAD" ssz-var-pkg: config`
}

And it would then generate the file with MAX_WITHDRAWALS_PER_PAYLOAD in place of the hardcoded 16, as well as import the config package (although that might already be doable automatically?).

I'm guessing this could introduce some overhead in cases where calculations are required for buffer indexes etc, but for extremely high performance apps it would still be possible to use the current way.

Given the struct:

type Container struct {
	Proof  *BlockEligibilityProof
	Proof2 BlockEligibilityProof
}

I always get this check in encoding code:

       if c.Proof == nil {
		c.Proof = new(BlockEligibilityProof)
	}
	offset += c.Proof.SizeSSZ()

	// Offset (1) 'Proof2'
	dst = ssz.WriteOffset(dst, offset)
	if c.Proof2 == nil {
		c.Proof2 = new(BlockEligibilityProof)
	}

I made a workaround, but still would be nice to solve it properly:

iff --git a/sszgen/main.go b/sszgen/main.go
index b5d25af..efdc2b3 100644
--- a/sszgen/main.go
+++ b/sszgen/main.go
@@ -956,6 +956,7 @@ func (e *env) parseASTFieldType(name, tags string, expr ast.Expr) (*Value, error
                        if err != nil {
                                return nil, fmt.Errorf("type %s not found", obj.Name)
                        }
+                       vv.noPtr = true
                        return vv, nil
                }
                return v, nil

Example

func TestBeaconBlock(t *testing.T) {
	block := &ethpb.BeaconBlock{
		Slot: 55,
	}
	enc, err := block.MarshalSSZ()
	if err != nil {
		t.Fatal(err) // FAIL incorrect fixed bytes marshalling
	}
	dec := &ethpb.BeaconBlock{}
	if err := dec.UnmarshalSSZ(enc); err != nil {
		t.Fatal(err)
	}
}

We had this same problem in go-ssz too, which is why we have this regression test. See prysmaticlabs/go-ssz#69

To reproduce, add BeaconState to the list of "codecs" and remove the special case logic skipping it. Then run spectests to observe that the generated code does not currently produce the correct result.

Prysm is side-stepping this issue for now because we use a hand-written HTR function for BeaconState, but this bug poses a risk in the event that some code accidentally calls the fastssz-generated HTR code. We're working on an additional work-around to prevent that from happening, but we're hoping that you can identify the issue so that the generated code is correct. Thank you!

$ git diff
diff --git a/spectests/structs_test.go b/spectests/structs_test.go
index b9fcddd..ed06cbb 100644
--- a/spectests/structs_test.go
+++ b/spectests/structs_test.go
@@ -38,6 +38,7 @@ var codecs = map[string]testCallback{
        "AggregateAndProof": func(config string) codec { return new(AggregateAndProof) },
        "Attestation":       func(config string) codec { return new(Attestation) },
        "AttesterSlashing":  func(config string) codec { return new(AttesterSlashing) },
+       "BeaconState": func(config string) codec { return new(BeaconState) },
        "BeaconBlock": func(config string) codec {
                if config == "minimal" {
                        return new(BeaconBlockMinimal)
@@ -306,7 +307,7 @@ func TestSpecMainnet(t *testing.T) {
                spl := strings.Split(f, "/")
                name := spl[len(spl)-1]
 
-               if name == "BeaconState" || name == "HistoricalBatch" {
+               if name == "HistoricalBatch" {
                        continue
                }
                base, ok := codecs[name]


$ go test -v ./spectests/...
=== RUN   TestFuzzMarshalWithWrongSizes
    structs_test.go:121: Fuzz testing not enabled, skipping
--- SKIP: TestFuzzMarshalWithWrongSizes (0.00s)
=== RUN   TestErrorResponse
--- PASS: TestErrorResponse (0.02s)
=== RUN   TestFuzzEncoding
    structs_test.go:121: Fuzz testing not enabled, skipping
--- SKIP: TestFuzzEncoding (0.00s)
=== RUN   TestFuzzUnmarshalAppend
    structs_test.go:121: Fuzz testing not enabled, skipping
--- SKIP: TestFuzzUnmarshalAppend (0.00s)
=== RUN   TestFuzzUnmarshalShuffle
    structs_test.go:121: Fuzz testing not enabled, skipping
--- SKIP: TestFuzzUnmarshalShuffle (0.00s)
=== RUN   TestSpecMinimal
    structs_test.go:297: Process AggregateAndProof ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/AggregateAndProof
    structs_test.go:297: Process Attestation ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/Attestation
    structs_test.go:297: Process AttestationData ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/AttestationData
    structs_test.go:297: Process AttesterSlashing ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/AttesterSlashing
    structs_test.go:297: Process BeaconBlock ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/BeaconBlock
    structs_test.go:297: Process BeaconBlockBody ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/BeaconBlockBody
    structs_test.go:297: Process BeaconBlockHeader ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/BeaconBlockHeader
    structs_test.go:297: Process BeaconState ../eth2.0-spec-tests/tests/minimal/altair/ssz_static/BeaconState
    structs_test.go:339: vector does not have the correct length
--- FAIL: TestSpecMinimal (2.27s)
=== RUN   TestSpecMainnet
    structs_test.go:319: Process AggregateAndProof ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/AggregateAndProof
    structs_test.go:319: Process Attestation ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/Attestation
    structs_test.go:319: Process AttestationData ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/AttestationData
    structs_test.go:319: Process AttesterSlashing ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/AttesterSlashing
    structs_test.go:319: Process BeaconBlock ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/BeaconBlock
    structs_test.go:319: Process BeaconBlockBody ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/BeaconBlockBody
    structs_test.go:319: Process BeaconBlockHeader ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/BeaconBlockHeader
    structs_test.go:319: Process BeaconState ../eth2.0-spec-tests/tests/mainnet/altair/ssz_static/BeaconState
    structs_test.go:509: 1 error(s) decoding:
       
        * 'block_roots': expected source data to have length less or equal to 64, got 8192
--- FAIL: TestSpecMainnet (1.97s)
FAIL
FAIL github.com/ferranbt/fastssz/spectests	4.269s
?   github.com/ferranbt/fastssz/spectests/external	[no test files]
?   github.com/ferranbt/fastssz/spectests/external2	[no test files]
FAIL

Sometimes, when hash tree root encounters the following error:

error=bytes array does not have the correct length

It could be very useful to return the field name as part of the error message. This can be done via codegen

If we define alias to a byte array as:

type Alias [20]byte

type Example struct {
   Field []Alias
}

It will not be respected by generator, and we will get [][20]byte in the generated code.

For a work around i used:

diff --git a/sszgen/unmarshal.go b/sszgen/unmarshal.go
index 836fd3c..3222b6b 100644
--- a/sszgen/unmarshal.go
+++ b/sszgen/unmarshal.go
@@ -338,6 +338,9 @@ func (v *Value) createSlice() string {
                if v.c {
                        return ""
                }
+               if len(v.e.obj) > 0 {
+                       return fmt.Sprintf("::.%s = make([]%v, %s)", v.name, v.e.objRef(), size)
+               }
                if v.e.c {
                        return fmt.Sprintf("::.%s = make([][%d]byte, %s)", v.name, v.e.s, size)
                }

I've tried the latest, v0.12.3 and v0.11.3 and both fail on incorrect offset

Bytes []byte

type A struct {
        Foo Bytes `ssz-max:"2048"`
}

type B struct {
        Bar Bytes `ssz-max:"32"`
}

If you run sszgen on these structs, both A and B will use 2048 as the max size for the byte list. In the case of B, this causes an incorrect hash-tree-root to be calculated, because the mix in should be based on 32, not `2048.

We'll need to support nested List for the upcoming merge milestone. Referencing the spec below, transactions in BeaconBlockBody is a nested List which I don't think we have ever had it before. Can we please add support this?

OpaqueTransaction: ByteList[MAX_BYTES_PER_OPAQUE_TRANSACTION]
transactions: List[OpaqueTransaction, MAX_EXECUTION_TRANSACTIONS]

https://github.com/ethereum/eth2.0-specs/blob/dev/specs/merge/beacon-chain.md#executionpayload

cc @ferranbt

Consider this struct used in ethereum beacon chain client

type ExecutionPayload struct {
        // ...
	Transactions  [][]byte  `ssz-max:"1048576,1073741824" ssz-size:"?,?" json:"transactions"`
}

The max-size of 1073741824 (2^30) makes the generated GetTree() pretty much unusable as the length Transactions can also be in the hundreds. One call could use hundreds of GB of memory.

In real life, each transaction []byte is only about several KB. Is this reasonably easy to optimize if instead of constructing the entire tree by filling the rest of the leaves with 0s, we construct a tree in which empty subtrees are represented by zero hashes at those depths?

When fastssz is ran multiple times against the same package, even without changes to the original files, the generated file keeps changing. This is because the hash that is present on the header of the generated files is not deterministic. This is reproducible when fastssz is ran on any package with more than one source file.

This is most likely due to a bug in

It would be handy to have built-in support for big.Int (along the lines of the time.Time support added in #100).

(This is motivated by ssz-encoding geth block headers).

Would you be open to a PR doing this? If so i'll put one up.

This line in getDepth should change to return 0 when the argument is 1 https://github.com/ferranbt/fastssz/blob/master/hasher.go#L333

Hey!

I've an issue with sszgen.

My sszgen generate code, when executed, produces a file with an unused import.

The unused library is provided in the generate command by the --include option and it's necessary for the generator to find the SyncCommitteeContribution type.

Is it a problem with v0.1.3 or did I code the command wrong?

Hello,

I would love to request further documentation so I can utilize this library to decode SSZ objects. My goal is:

1. Import this library into my go project.
2. Query a Beacon Client's API, such as eth/v1/events?topics=head, get the block and state hash.
3. Utilize this libraries decode functionality, and decode the block and state hash into a Go native struct.
4. Do as I please with the Go native struct.

If you can provide me with any guidance on how to turn an SSZ encoded hash (such as: 0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf) into a go-object, it would be immensely useful.

Thank You :D

We're using sszgen for an updated eip4844 beacon block body:

class BeaconBlockBody(Container):
    randao_reveal: BLSSignature
    eth1_data: Eth1Data  # Eth1 data vote
    graffiti: Bytes32  # Arbitrary data
    # Operations
    proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS]
    attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS]
    attestations: List[Attestation, MAX_ATTESTATIONS]
    deposits: List[Deposit, MAX_DEPOSITS]
    voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS]
    sync_aggregate: SyncAggregate
    # Execution
    execution_payload: ExecutionPayload 
    blob_kzg_commitments: List[KZGCommitment, MAX_BLOBS_PER_BLOCK]  # [New in EIP-4844]

The BeaconBlockBody contains a new field, blob_kzg_commitments, where KZGCommitment is a Bytes48 ssz type and MAX_BLOBS_PER_BLOCK is 16. So we add the field to the new BeaconBlockBody struct in our prysm fork:

BlobKzgs [][]byte `ssz-size:"?,48" ssz-max:"16"`

However, the updated generated tree routine for the beacon block looks incorrect:

func (b *BeaconBlockBody) HashTreeRootWith(hh *ssz.Hasher) (err error) {
	// <omitted other fields for brevity>

	// Field (10) 'BlobKzgs'
	{
		if size := len(b.BlobKzgs); size > 16 {
			err = ssz.ErrListTooBigFn("--.BlobKzgs", size, 16)
			return
		}
		subIndx := hh.Index()
		for _, i := range b.BlobKzgs {
			if len(i) != 48 {
				err = ssz.ErrBytesLength
				return
			}
			hh.PutBytes(i)
		}

		numItems := uint64(len(b.BlobKzgs))
		hh.MerkleizeWithMixin(subIndx, numItems, ssz.CalculateLimit(16, numItems, 0))
	}

I'd expect that the size input for CalculateLimit to be 48 but here it's 0 which causes the new BeaconBlockBody root to be incorrect.

If I create a simple type:

type Obj1 struct {
        T1 [][]byte `ssz-max:"1024,256" ssz-size:"?,?"`
}

sszgen generates code that handles the two separate maximums fine:

func (o *Obj1) MarshalSSZTo(buf []byte) (dst []byte, err error) {
    ...
    // Field (0) 'T1'
    if size := len(o.T1); size > 1024 {
        err = ssz.ErrListTooBigFn("Obj1.T1", size, 1024)
        return
    }
    ...
    for ii := 0; ii < len(o.T1); ii++ {
        if size := len(o.T1[ii]); size > 256 {
            err = ssz.ErrBytesLengthFn("Obj1.T1[ii]", size, 256)
            return
        }
        dst = append(dst, o.T1[ii]...)
    }
    ...
}

It can be seen that the two sizes (1024 and 256) are used correctly. However, if I add an intermediate type in to the object:

type Data []byte

type Obj2 struct {
        T1 []Data `ssz-max:"1024,256" ssz-size:"?,?"`
}

sszgen generates this:

func (o *Obj2) MarshalSSZTo(buf []byte) (dst []byte, err error) {
    // Field (0) 'T1'
    if size := len(o.T1); size > 256 {
        err = ssz.ErrListTooBigFn("Obj2.T1", size, 256)
        return
    }
    ...
    for ii := 0; ii < len(o.T1); ii++ {
        if size := len(o.T1[ii]); size > 256 {
            err = ssz.ErrBytesLengthFn("Obj2.T1[ii]", size, 256)
            return
        }
        dst = append(dst, o.T1[ii]...)
    }
    ...
}

Here it can be seen that in the first piece of code 256 is being used where it should still be 1024.

I'm mainly pointing out the fields that I know that don't work well today. These are the great entry point but to be fully comply we should pass using the spec tests provided.

1.) Altair

class BeaconState(Container):
    #  bytes previous_epoch_participation = 7001 [(gogoproto.moretags) = "ssz-max:\"1099511627776\""]
    previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT]  # [Modified in Altair]

    current_sync_committee: SyncCommittee  # [New in Altair]

class SyncCommittee(Container):
    # repeated bytes pubkeys = 1  [(gogoproto.moretags) = "ssz-size:\"1024,48\""];
    pubkeys: Vector[BLSPubkey, SYNC_COMMITTEE_SIZE]
    # repeated bytes pubkey_aggregates = 2  [(gogoproto.moretags) = "ssz-size:\"16,48\""];
    aggregate_pubkey: BLSPubkey

Spec tests: https://github.com/ethereum/eth2.0-spec-tests/tree/master/tests/mainnet/altair/ssz_static

2.) Merge

# OpaqueTransaction  is ByteList[MAX_BYTES_PER_OPAQUE_TRANSACTION]
class ExecutionPayload(Container):
    #  repeated bytes transactions = 11  [(gogoproto.moretags) = "ssz-size:\"?,?\" ssz-max:\"16384,1048576\""]
    transactions: List[OpaqueTransaction, MAX_EXECUTION_TRANSACTIONS]

Spec tests: https://github.com/protolambda/eth2.0-spec-tests/tree/v1.1.0-alpha.4-pre2/tests/mainnet/merge/ssz_static

3.) Sharding

class BeaconState(merge.BeaconState):  # [extends The Merge state]
    # [Updated fields]
    grandparent_epoch_confirmed_commitments: Vector[Vector[DataCommitment, SLOTS_PER_EPOCH], MAX_SHARDS]

# Note, we maybe change this to Vector[List[DataCommitment so let's support that as well.

No spec test yet

=== RUN TestSpecMinimal
TestSpecMinimal: structs_test.go:263: ../eth2.0-spec-tests/tests/minimal/phase0/ssz_static/AggregateAndProof
TestSpecMinimal: structs_test.go:263: ../eth2.0-spec-tests/tests/minimal/phase0/ssz_static/Attestation
TestSpecMinimal: structs_test.go:263: ../eth2.0-spec-tests/tests/minimal/phase0/ssz_static/AttestationData
TestSpecMinimal: structs_test.go:263: ../eth2.0-spec-tests/tests/minimal/phase0/ssz_static/AttesterSlashing
TestSpecMinimal: structs_test.go:263: ../eth2.0-spec-tests/tests/minimal/phase0/ssz_static/BeaconBlock
TestSpecMinimal: structs_test.go:299: list length is higher than max value
--- FAIL: TestSpecMinimal (0.35s)
=== RUN TestSpecMainnet
TestSpecMainnet: structs_test.go:284: ../eth2.0-spec-tests/tests/mainnet/phase0/ssz_static/AggregateAndProof
TestSpecMainnet: structs_test.go:284: ../eth2.0-spec-tests/tests/mainnet/phase0/ssz_static/Attestation
TestSpecMainnet: structs_test.go:284: ../eth2.0-spec-tests/tests/mainnet/phase0/ssz_static/AttestationData
TestSpecMainnet: structs_test.go:284: ../eth2.0-spec-tests/tests/mainnet/phase0/ssz_static/AttesterSlashing
TestSpecMainnet: structs_test.go:284: ../eth2.0-spec-tests/tests/mainnet/phase0/ssz_static/BeaconBlock
TestSpecMainnet: structs_test.go:302: bad marshalling
--- FAIL: TestSpecMainnet (0.02s)
FAIL
FAIL github.com/ferranbt/fastssz/spectests 0.640s
? github.com/ferranbt/fastssz/spectests/external [no test files]
? github.com/ferranbt/fastssz/spectests/external2 [no test files]
FAIL

package testcases

import "github.com/prysmaticlabs/go-bitfield"

type Foo struct {
	Bar bitfield.Bitlist
}

$ sszgen --path bitlist.go

Output

[ERR]: failed to encode Foo: bitlist bitfield does not have ssz-max tag

Expected Output

[ERR]: failed to encode Foo: bitlist Bar does not have ssz-max tag

With a message structure like this:

message ErrorResponse {
  bytes message = 1 [(gogoproto.moretags) = "ssz-max:\"256\""];
}

fastssz is generating a MarshalTo that only allows messages of length 0.

/ MarshalSSZTo ssz marshals the ErrorResponse object to a target array
func (e *ErrorResponse) MarshalSSZTo(buf []byte) (dst []byte, err error) {
	dst = buf
	offset := int(4)

	// Offset (0) 'Message'
	dst = ssz.WriteOffset(dst, offset)
	offset += len(e.Message)

	// Field (0) 'Message'
	if len(e.Message) != 0 { // BUG: Only length zero allowed?
		err = ssz.ErrBytesLength
		return
	}
	dst = append(dst, e.Message...)

	return
}