Giter VIP home page Giter VIP logo

go-clone's Introduction

go-clone: Clone any Go data structure deeply and thoroughly

Go Go Doc Go Report Coverage Status

Package clone provides functions to deep clone any Go data. It also provides a wrapper to protect a pointer from any unexpected mutation.

For users who use Go 1.18+, it's recommended to import github.com/huandu/go-clone/generic for generic APIs and arena support.

Clone/Slowly can clone unexported fields and "no-copy" structs as well. Use this feature wisely.

Install

Use go get to install this package.

go get github.com/huandu/go-clone

Usage

Clone and Slowly

If we want to clone any Go value, use Clone.

t := &T{...}
v := clone.Clone(t).(*T)
reflect.DeepEqual(t, v) // true

For the sake of performance, Clone doesn't deal with values containing pointer cycles. If we need to clone such values, use Slowly instead.

type ListNode struct {
    Data int
    Next *ListNode
}
node1 := &ListNode{
    Data: 1,
}
node2 := &ListNode{
    Data: 2,
}
node3 := &ListNode{
    Data: 3,
}
node1.Next = node2
node2.Next = node3
node3.Next = node1

// We must use `Slowly` to clone a circular linked list.
node := Slowly(node1).(*ListNode)

for i := 0; i < 10; i++ {
    fmt.Println(node.Data)
    node = node.Next
}

Generic APIs

Starting from go1.18, Go started to support generic. With generic syntax, Clone/Slowly and other APIs can be called much cleaner like following.

import "github.com/huandu/go-clone/generic"

type MyType struct {
    Foo string
}

original := &MyType{
    Foo: "bar",
}

// The type of cloned is *MyType instead of interface{}.
cloned := Clone(original)
println(cloned.Foo) // Output: bar

It's required to update minimal Go version to 1.18 to opt-in generic syntax. It may not be a wise choice to update this package's go.mod and drop so many old Go compilers for such syntax candy. Therefore, I decide to create a new standalone package github.com/huandu/go-clone/generic to provide APIs with generic syntax.

For new users who use Go 1.18+, the generic package is preferred and recommended.

Arena support

Starting from Go1.20, arena is introduced as a new way to allocate memory. It's quite useful to improve overall performance in special scenarios. In order to clone a value with memory allocated from an arena, there are new methods ArenaClone and ArenaCloneSlowly available in github.com/huandu/go-clone/generic.

// ArenaClone recursively deep clones v to a new value in arena a.
// It works in the same way as Clone, except it allocates all memory from arena.
func ArenaClone[T any](a *arena.Arena, v T) (nv T) 

// ArenaCloneSlowly recursively deep clones v to a new value in arena a.
// It works in the same way as Slowly, except it allocates all memory from arena.
func ArenaCloneSlowly[T any](a *arena.Arena, v T) (nv T)

Due to limitations in arena API, memory of the internal data structure of map and chan is always allocated in heap by Go runtime (see this issue).

Warning: Per discussion in the arena proposal, the arena package may be changed incompatibly or removed in future. All arena related APIs in this package will be changed accordingly.

Struct tags

There are some struct tags to control how to clone a struct field.

type T struct {
    Normal *int
    Foo    *int `clone:"skip"`       // Skip cloning this field so that Foo will be zero in cloned value.
    Bar    *int `clone:"-"`          // "-" is an alias of skip.
    Baz    *int `clone:"shadowcopy"` // Copy this field by shadow copy.
}

a := 1
t := &T{
    Normal: &a,
    Foo:    &a,
    Bar:    &a,
    Baz:    &a,
}
v := clone.Clone(t).(*T)

fmt.Println(v.Normal == t.Normal) // false
fmt.Println(v.Foo == nil)         // true
fmt.Println(v.Bar == nil)         // true
fmt.Println(v.Baz == t.Baz)       // true

Memory allocations and the Allocator

The Allocator is designed to allocate memory when cloning. It's also used to hold all customizations, e.g. custom clone functions, scalar types and opaque pointers, etc. There is a default allocator which allocates memory from heap. Almost all public APIs in this package use this default allocator to do their job.

We can control how to allocate memory by creating a new Allocator by NewAllocator. It enables us to take full control over memory allocation when cloning. See Allocator sample code to understand how to customize an allocator.

Let's take a closer look at the NewAllocator function.

func NewAllocator(pool unsafe.Pointer, methods *AllocatorMethods) *Allocator
  • The first parameter pool is a pointer to a memory pool. It's used to allocate memory for cloning. It can be nil if we don't need a memory pool.
  • The second parameter methods is a pointer to a struct which contains all methods to allocate memory. It can be nil if we don't need to customize memory allocation.
  • The Allocator struct is allocated from the methods.New or the methods.Parent allocator or from heap.

The Parent in AllocatorMethods is used to indicate the parent of the new allocator. With this feature, we can orgnize allocators into a tree structure. All customizations, including custom clone functions, scalar types and opaque pointers, etc, are inherited from parent allocators.

There are some APIs designed for convenience.

  • We can create dedicated allocators for heap or arena by calling FromHeap() or FromArena(a *arena.Arena).
  • We can call MakeCloner(allocator) to create a helper struct with Clone and CloneSlowly methods in which the type of in and out parameters is interface{}.

Mark struct type as scalar

Some struct types can be considered as scalar.

A well-known case is time.Time. Although there is a pointer loc *time.Location inside time.Time, we always use time.Time by value in all methods. When cloning time.Time, it should be OK to return a shadow copy.

Currently, following types are marked as scalar by default.

  • time.Time
  • reflect.Value

If there is any type defined in built-in package should be considered as scalar, please open new issue to let me know. I will update the default.

If there is any custom type should be considered as scalar, call MarkAsScalar to mark it manually. See MarkAsScalar sample code for more details.

Mark pointer type as opaque

Some pointer values are used as enumerable const values.

A well-known case is elliptic.Curve. In package crypto/tls, curve type of a certificate is checked by comparing values to pre-defined curve values, e.g. elliptic.P521(). In this case, the curve values, which are pointers or structs, cannot be cloned deeply.

Currently, following types are marked as scalar by default.

  • elliptic.Curve, which is *elliptic.CurveParam or elliptic.p256Curve.
  • reflect.Type, which is *reflect.rtype defined in runtime.

If there is any pointer type defined in built-in package should be considered as opaque, please open new issue to let me know. I will update the default.

If there is any custom pointer type should be considered as opaque, call MarkAsOpaquePointer to mark it manually. See MarkAsOpaquePointer sample code for more details.

Clone "no-copy" types defined in sync and sync/atomic

There are some "no-copy" types like sync.Mutex, atomic.Value, etc. They cannot be cloned by copying all fields one by one, but we can alloc a new zero value and call methods to do proper initialization.

Currently, all "no-copy" types defined in sync and sync/atomic can be cloned properly using following strategies.

  • sync.Mutex: Cloned value is a newly allocated zero mutex.
  • sync.RWMutex: Cloned value is a newly allocated zero mutex.
  • sync.WaitGroup: Cloned value is a newly allocated zero wait group.
  • sync.Cond: Cloned value is a cond with a newly allocated zero lock.
  • sync.Pool: Cloned value is an empty pool with the same New function.
  • sync.Map: Cloned value is a sync map with cloned key/value pairs.
  • sync.Once: Cloned value is a once type with the same done flag.
  • atomic.Value/atomic.Bool/atomic.Int32/atomic.Int64/atomic.Uint32/atomic.Uint64/atomic.Uintptr: Cloned value is a new atomic value with the same value.

If there is any type defined in built-in package should be considered as "no-copy" types, please open new issue to let me know. I will update the default.

Set custom clone functions

If default clone strategy doesn't work for a struct type, we can call SetCustomFunc to register a custom clone function.

SetCustomFunc(reflect.TypeOf(MyType{}), func(allocator *Allocator, old, new reflect.Value) {
    // Customized logic to copy the old to the new.
    // The old's type is MyType.
    // The new is a zero value of MyType and new.CanAddr() always returns true.
})

We can use allocator to clone any value or allocate new memory. It's allowed to call allocator.Clone or allocator.CloneSlowly on old to clone its struct fields in depth without worrying about dead loop.

See SetCustomFunc sample code for more details.

Clone atomic.Pointer[T]

As there is no way to predefine a custom clone function for generic type atomic.Pointer[T], cloning such atomic type is not supported by default. If we want to support it, we need to register a custom clone function manually.

Suppose we instantiate atomic.Pointer[T] with type MyType1 and MyType2 in a project, and then we can register custom clone functions like following.

import "github.com/huandu/go-clone/generic"

func init() {
    // Register all instantiated atomic.Pointer[T] types in this project.
    clone.RegisterAtomicPointer[MyType1]()
    clone.RegisterAtomicPointer[MyType2]()
}

Wrap, Unwrap and Undo

Package clone provides Wrap/Unwrap functions to protect a pointer value from any unexpected mutation. It's useful when we want to protect a variable which should be immutable by design, e.g. global config, the value stored in context, the value sent to a chan, etc.

// Suppose we have a type T defined as following.
//     type T struct {
//         Foo int
//     }
v := &T{
    Foo: 123,
}
w := Wrap(v).(*T) // Wrap value to protect it.

// Use w freely. The type of w is the same as that of v.

// It's OK to modify w. The change will not affect v.
w.Foo = 456
fmt.Println(w.Foo) // 456
fmt.Println(v.Foo) // 123

// Once we need the original value stored in w, call `Unwrap`.
orig := Unwrap(w).(*T)
fmt.Println(orig == v) // true
fmt.Println(orig.Foo)  // 123

// Or, we can simply undo any change made in w.
// Note that `Undo` is significantly slower than `Unwrap`, thus
// the latter is always preferred.
Undo(w)
fmt.Println(w.Foo) // 123

Performance

Here is the performance data running on my dev machine.

go 1.20.1
goos: darwin
goarch: amd64
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkSimpleClone-12       7164530        156.7 ns/op       24 B/op        1 allocs/op
BenchmarkComplexClone-12       628056         1871 ns/op     1488 B/op       21 allocs/op
BenchmarkUnwrap-12           15498139        78.02 ns/op        0 B/op        0 allocs/op
BenchmarkSimpleWrap-12        3882360        309.7 ns/op       72 B/op        2 allocs/op
BenchmarkComplexWrap-12        949654         1245 ns/op      736 B/op       15 allocs/op

License

This package is licensed under MIT license. See LICENSE for details.

go-clone's People

Contributors

huandu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-clone's Issues

调研:使用 go-clone 的业务场景

大家好,我是这个库的作者,我很想调研一下大家是如何在项目中使用这个库的,请告诉我用法、使用的原因、相关业务场景。

我想通过这样的调研了解这个库的适用范围,从而帮助我更好的发展这个库。我的目标是,在 Go runtime 允许范围内提供一些尽可能高效的内存拷贝 API,满足各种性能敏感场景中的内存管理需求。经过这么多年的研究和积累,特别是通过这个库实践出来的很多有趣的 trick,一定程度上证明了 Go runtime 中内存管理相关机制的可塑性。如果我能更好的理解大家的用法和需求,我认为我可以将内存拷贝和管理这件事情做的更好,让这个库产生更大的价值。

欢迎大家在这个 issue 下面留言,我都会看到并跟大家讨论。感谢大家!

go1.16无法构建?

似乎是无法识别atomic_go119.go里的
//go:build
将go:build改成+build后构建成功

How to process nested pointer for arena.New

How to use arena to allocte memory for Outer,should we use arena to allocate memory for the nested field Inner?
Is there some method to identify the memory-allocated way.

type Outer struct {
   *Inner
}
type Inner Struct {
}

a arena object can not be used concurrently.Is there some example for arena pool

the same question :orderedmap used List

hello, when i use ordermap ,i find the same question

// import orderedmap "github.com/wk8/go-ordered-map"

om := orderedmap.New()

om.Set("foo", "bar")

// iterating pairs from oldest to newest:
for pair := om.Oldest(); pair != nil; pair = pair.Next() {
fmt.Printf("%s => %s\n", pair.Key, pair.Value)
}

newOM := clone.Slowly(om).(*orderedmap.OrderedMap)
for pair := newOM.Oldest(); pair != nil; pair = pair.Next() {
fmt.Printf("%s => %s\n", pair.Key, pair.Value)
}

CustomFunc should allow Ptr types as well

CustomFunc are helpful in creating custom clone functions. However, it doesn't work well with Ptrs. Because when adding *Struct, it gets resolved to Struct and similarly when looking up from the map, only the Struct is used.

There are some cases, when custom pointers would be helpful.

For example, in my case, under certain circumstances, I want the ptrs to reuse the same reference and sometimes not.
I am working on a model checker, where the spec might say, multiple alternate paths a process can take, and we would want to explore both the paths. So, from the parent, I would create separate clone of entire variable spaces for each fork, but within each fork I would want to reuse the variable.

Marking it opaque pointer doesn't help, as I would want to actually clone in some cases. So, the ideal solution would be to make allocator.SetCustomFunc(...) to not dereference the Ptr type, and when checking ptrs, and use the custom func when present, before dereferencing the ptr

Build failed on Linux MIPS 32bit

I want to deep copy a *tls.Certificate, but build failed on Linux MIPS 32bit(softfloat and hardfloat):

../../go/pkg/mod/github.com/huandu/[email protected]/clone.go:280:10: type [2147483647]byte larger than address space
../../go/pkg/mod/github.com/huandu/[email protected]/clone.go:390:11: type [2147483647]byte larger than address space
../../go/pkg/mod/github.com/huandu/[email protected]/clone.go:436:11: type [2147483647]byte larger than address space

Ignore serialization

hello, i create a struct like:
type struct{
a string
b string
c string
}

when call clone.Clone(). Is there a method to Ignore serialization
member b?

String 是否始终不应作为标量进行 Clone

因为常常会使用类似这样的 BytesToString 方法

func BytesToString(b []byte) string {
	if len(b) == 0 {
		return ""
	}
	return unsafe.String(&b[0], len(b))
}

因为 go-clone!arenaIsEnabled 的时候会将字符串作为标量处理不进行 copy,这样的话,如果对原Bytes进行修改了,也会修改掉 Clone 后的结果,就很郁闷。(至于为什么会修改原Bytes,诸多原因吧,例如我有使用bytebufferpool)

func TestClone(t *testing.T) {
	b := []byte("hello")
	s := BytesToString(b)
	s2 := gclone.Clone(s)
	b[1] = 'a'
	log.Println(s) // print `hallo`
	log.Println(s2) // pritn `hallo`
}

因为BytesToString是比较trick的方式,所以go-clone默认将字符串当作标量处理也算合理。
但是怎么才能在使用时候根据自身需要修改这个行为呢?

Help Wanted: Survey of use cases using this package

Hello everyone,

I am the author of this package. I would like to survey how you use this package in your project. Please tell me how you use it, the reason for using it, and the related business scenario.

I want to understand the scope of this package through this survey, so that I can better develop it. My goal is to provide some as efficient as possible memory copy APIs within the scope of Go runtime to meet the memory management needs of various performance-sensitive scenarios. After several years of research and accumulation, especially through the practice of this package, many interesting tricks have proved the plasticity of the memory management related mechanisms in the Go runtime. If I can better understand your usage and requirements, I think I can do a better job of memory copy and management, and make this package more valuable.

Welcome everyone to leave a message below this issue. I will see it and discuss with you.

Thank you!

(P.S. For users who speaks Chinese, please leave your message in #15. Thanks)

List clone incorrect

hello, I find a bug below:
l := list.New()
l.PushBack("123")

for i := l.Front(); i != nil; i = i.Next() {
	fmt.Println(i.Value)
}

newl := clone.Slowly(l).(*list.List)
for i := newl.Front(); i != nil; i = i.Next() {
	fmt.Println(i.Value)
}

output like this:
123
123

why cloned new object contain a ?
Is it the wrong way to use it?

Question: no clone

Hi,

Just a question about your package: If I want a field not to be copied, do I use SetCustomFunc or is there a tag system (like jinzhu copier package copier:"-")?

Thanks

can not copy *zap.Logger

import (
clone "github.com/huandu/go-clone"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)

func CloneLogger(logger *zap.Logger) *zap.Logger {
return clone.Clone(logger)
}

it will pacnic with message below:
panic: reflect.Value.Convert: value of type int8 cannot be converted to type zapcore.LevelEnabler

any help is appreciated.

"go vet" fails in Go 1.18

I just tried your generic package and all was good until I ran "go vet". I'm using Go 1.18.10 and wanted to use the generic version of go-clone. "go mod tidy" picks up v1.6.0, and then "go vet" produces:

$ go vet ./...
# github.com/huandu/go-clone/generic
../vendor/github.com/huandu/go-clone/generic/register.go:16:38: undefined: atomic.Pointer
../vendor/github.com/huandu/go-clone/generic/register.go:22:47: undefined: atomic.Pointer
../vendor/github.com/huandu/go-clone/generic/register.go:23:47: undefined: atomic.Pointer

This is (likely) because atomic.Pointer is a Go 1.19 feature. So... I don't think the generic version works in Go 1.18.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.