falcosecurity / plugin-sdk-go Goto Github PK
View Code? Open in Web Editor NEWFalco plugins SDK for Go
License: Apache License 2.0
Falco plugins SDK for Go
License: Apache License 2.0
Motivation
Although it's not official, a de-facto project layout standard exists and is widely used.
Moreover, this remarkable project ๐ https://github.com/golang-standards/project-layout includes all details, and we have already used it as a reference for the various go-lang projects within the falcosecurity org.
Feature
Refactor the current directories organization by following the layout proposed by https://github.com/golang-standards/project-layout. In particular:
/pkg
/test
for additional test (eg. integration test) if any/internal
when appropriate/examples
(and /docs
eventually); it would be beneficial to have some easily buildable examples directly in the SDK so users can experiment with no need of starting a new plugin projectAlternatives
Although this proposal is not strictly necessary to make the SDK work (so the do-nothing alternative is still possible ๐ธ ), I strongly believe that using a clear and widely used project layout benefits users.
Additional context
Current packages are organized in a way that facilitates the automatic generation of the documentation. We should preserve this organization.
For point (B3) of falcosecurity/falco#2074, we will need the Go SDK to be aware and resistant to concurrent access to some symbols of the Go SDK. This issue tracks and documents the thought process and the developments to achieve this.
The assumptions of falcosecurity/falco#2074 imply that a given application could run multiple sinsp
inspectors in parallel, each in its own thread. In this model, a given plugin registered and initialized in an inspector can't be shared across multiple inspectors. However, the same plugin dynamic library is shared as a singleton across all the inspectors of the application. This leads to the conclusion that the Plugin SDK Go must be able to support multiple consumers that:
ss_plugin_t
state, such as plugin_get_name()
) from multiple concurrent threads with no restrictionss_plugin_t
In the Go SDK, this maps to the following critical points:
plugin_init
, plugin_destroy
, plugin_open
, and plugin_close
, all rely on our optimized implementation of cgo.Handle
, which is very performant for our use case but not thread-safecgo.Handle
implementation caps the max number of handles to be used at the same time to 32plugin_extract_fields
currently relies on a single-consumer-single-worker execution model when the async extraction optimization is enabled (e.g. extract.SetAsync(true)
), which makes it thread-safe even if different threads use different ss_plugin_t
valuesplugin_next_batch
and plugin_extract_fields
workflows. For now, this requires no intervention on the Go SDK.What to document
plugin-go-sdk is now released. So we have to update the readme and possibly add some nice badges :)
Motivation
In the Info() *plugins.Info
method, the Go SDK sets automatically the RequiredAPIVersion
field to the latest supported plugin API version. Developers can choose to set their own value too, if desired. However, the SDK does not check that the developer-provided version is semver-compatible with the internal default one. This can cause silent or ambiguous failures due to the fact that the SDK is inherently developed to comply to a certain version of the plugin API.
Feature
If developer supply a custom RequiredAPIVersion
info value, then the SDK should perform a semver check with its internally-supported API version, and output an expressive error if the two versions are incompatible in order to prevent unpredicable failures.
Alternatives
Additional context
Motivation
Currently, the SDK does not have an embedded logging system. This could be useful for debugging plugins during the development phase and in many other use cases. Currently this is attained manually in each plugin (see an example).
Feature
Add some standard Go logging system in the SDK, such as https://pkg.go.dev/log. Plugins must be able to enable/disable and configure the logger programmatically.
The logging system could be bundled in the base level sdk
package, but we can also consider creating an single-reposibility log
package, just like we have ptr
and cgo
. Then, each internal part of the SDK should rely on that to log information with different severity level. For instance, each prebuilt C symbol should log meaningful information, and constructs such as EventWriters
can potentially log something too.
Alternatives
Leave things as they are, meaning that user each plugin should implement their logging measures. I don't like this option, especially because it makes not possible for the SDK to expose some of its internal process, which can be meaningful for debugging purposes.
What to document
Once falcosecurity/falco-website#493 gets merged, we have to update the links in the README.md
Motivation
Following a discussion with @leogr, @FedeDP, and @ldegio, we recognize that the sdk could be furtherly optimized to minimize the number of memory allocations and copies involved in the next/next_batch and extract flows. More specifically:
WrapExtractFuncs()
and RegisterAsyncExtractors()
we use both C.GoBytes()
and C.GoString()
to access C-allocated memory from Go. However, this causes a malloc and a copy under the hood, and can happens multiple times in a loop during batch extractions.next
/next_batch
and in the Events
wrapper we allocate multiple buffers at each iteration. First, we allocate a PluginEvent
struct for each event. Second, we allocate a slice ofPluginEvent
to host the array of fetched events. Last, we allocate a C memory buffer to be used inside C.fill_event
. This last buffer is then freed by scap
, outside the control of the plugin.As such, we see the following problems:
byte
slice (in extractor functions, for instance), which allows sdk users to do uncontrolled write operations in the data buffer, thus potentially breaking the sdk code/logic.Proposal
Next
), so that we can enforce read-only or write-only modes. This would make accessing C-allocated buffers safer, as we can have higher control on how it is accessed in the Go plugin.Alternatives
Leave things as they are.
Additional Context
Points 1 and 2 of our proposal can be attained by an abstraction like this one:
// BytesReadWriter is an opaque wrapper for fixed-size memory buffers, that can safely be
// used in the plugin framework in a Go-friendly way. The purpose is to provide means
// for safe memory access through the read/write interface primitives, regardless of how
// how the buffer is physically allocated under the hood. For instance, this can be used
// to wrap a C-allocated buffers, to hide both the type conversion magic and avoid illegal
// memory operations. The io.ReadWriteSeeker interface is leveraged to implement the safe
// random memory access semantic. Note, read-only or write-only modes to the memory buffer
// can easily be accomplished by casting this to either a io.Reader or io.Writer.
type BytesReadWriter interface {
io.ReadWriteSeeker
//
// Returns an unsafe.Pointer that points to the underlying memory buffer.
Buffer() unsafe.Pointer
//
// Size returns the physical size of the underlying memory buffer.
Size() int64
//
// Offset returns the current cursor position relatively to the underlying buffer.
// The cursor position represents the index of the next byte in the buffer that will
// be available for read\write operations. This value is altered through the usage of
// Seek, Read, and Write. By definition, we have that 0 <= Offset() <= Size().
Offset() int64
}
func NewBytesReadWriter(buf unsafe.Pointer, size int64) (BytesReadWriter, error) {
// Inspired by: https://stackoverflow.com/a/66218124
var bytes []byte
(*reflect.SliceHeader)(unsafe.Pointer(&bytes)).Data = uintptr(unsafe.Pointer(buf))
(*reflect.SliceHeader)(unsafe.Pointer(&bytes)).Len = int(size)
(*reflect.SliceHeader)(unsafe.Pointer(&bytes)).Cap = int(size)
return &bytesReadWriter{
buffer: buf,
bytesAlias: bytes,
offset: 0,
size: size,
}, nil
}
type bytesReadWriter struct {
offset int64
size int64
buffer unsafe.Pointer
bytesAlias []byte
}
... // (Intuitive from here) implement Read, Write, Seek, Size, Offset, Buffer
Motivation
Currently, we allow developers to implement the Progress()
plugin method optionally. If so, manually importing the github.com/falcosecurity/plugin-sdk-go/pkg/sdk/symbols/progress
is required. This is the only exception of this kind in the whole SDK. See here ๐๐ผ
plugin-sdk-go/examples/full/full.go
Line 189 in f1111df
This design choice was motivated by the fact that, by importing the package, a do-nothing C symbol would have been exported once plugins get compiled, which we thought could create problems on to the framework. However, this seems to be exactly what the C++ SDK does ๐๐ผ
https://github.com/falcosecurity/plugins/blob/019437ed9d427ba513555dc9ad8a2d5a3415bec0/sdk/cpp/falcosecurity_plugin.h#L127
Feature
We might want to add an import for the github.com/falcosecurity/plugin-sdk-go/pkg/sdk/symbols/progress
package in the sdk/plugins/source
module. In this way, developers would still be able to implement Progress
optionally, but without having to import the package manually. I personally think that this is clearer, as it is more consistent with the rest of the SDK.
Alternatives
Leave things as they are, with the package needing to be imported manually.
Additional context
This would not introduce any breaking change. Plugins already importing the github.com/falcosecurity/plugin-sdk-go/pkg/sdk/symbols/progress
would still compile successfully.
Motivation
So far, plugin developers are only able to test their plugins by compiling them into shared libraries and loading them in Falco. This is unhandy and limiting both for the development and testing phases.
Feature
Provide a builti-in framework in the Go SDK to run and test plugins with little effort. Since Go plugins are simple Go types compliant to few interfaces, they can be easily used in other Go programs and in the Go testing framework. We could leverage this to provide a better developer experience and to improve the overall quality assurance strategy of new plugins.
Alternatives
Additional context
Feature
We would like to:
Motivation
Some plugins, for example the k8saudit-xxx plugins, require an additional step to ingest events. The event from the source is not in the format as expected by the field extraction. E.g. in case of the k8saudit-gke
plugin, the raw source event is a Google specific logging event. To be able to use the k8saudit
plugin field extraction and rules a conversion/transformation is required.
To guarantee some sort of method signature and documentation for an optional conversion/transformation method, this method should preferably be part of the SDK. This allows building on top of the plugin code and reuse the conversion/transformation logic and extraction method.
Feature
Introduce a convert/transform method as part of the SDK api, just like the Open
and Extract
methods. Ideally the framework takes care of wiring everything together. This convert/transform method should maybe be part of the event sourcing
capability:
Alternatives
Additional context
For reference, falcosecurity/plugins issue #490
Motivation
Async extraction allows speeding up the extraction flow. The idea is to have a worker goroutine busy-waiting and synchronizing with C code through a shared atomic value. This is more efficient than the cost of a C -> Go call, but can be quite expensive in terms of cpu time.
Extraction is one of the hottest path in the whole framework, so it's an high priority to optimize it and speed it up as much as possible. The current implementation can still be improved in few ways, and it's worth investigating potential optimizations.
Feature
As for a preliminary research, our current implementation can be improved in the following:
async_extractor_wait
function, that implements the busy-wait of the worker goroutine, is implemented in C. This means that a Go -> C calls is required. Although this does not add a big overhead, we may consider implementing the atomic synchronization in Go directly using the atomic
package. Please not that the memory ordering should be consistent between the C and the Go implementations to avoid data races.memory_order_seq_cst
, which is sub-optimal. We can attempt using weaker memory orders, as our workflow would work well with the acquire-release semantics.libsisnp
yet), the async_extractor_extract_field
function is called in a loop for each field. This triggers the synchronization mechanism for each of those fields. It would be way more efficient to support batch extraction in the async worker to reduce the overhead.Additional context
Optimizations 1 and 2 might be mutually exclusive. Go memory guarantees are not explicit yet, but they seem to rely on sequential consistency (see: https://groups.google.com/g/golang-dev/c/vVkH_9fl1D8/m/azJa10lkAwAJ). Further investigation is required on this. Also, it would be meaningful to benchmark both option 1 and 2 to better understand what the most meaningful optimization is.
This issue tracks any remaining set up steps that I couldn't handle myself.
Describe the bug
The SDK allows code like the following to run:
func init() {
p1 := &SomePlugin{}
p2 := &AnotherPlugin{}
source.Register(p1)
extractor.Register(p2)
}
However, this does not make sense in terms of the Falco plugin system. By design the SDK should allow only one plugins to be registered, because the passed-in instance will be the one used to implement the C symbols required by the plugin API.
Expected behaviour
The SDK should either:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.