v3io / frames Goto Github PK
View Code? Open in Web Editor NEWGolang based remote data frames access (over gRPC or HTTP stream)
License: Apache License 2.0
Golang based remote data frames access (over gRPC or HTTP stream)
License: Apache License 2.0
instead of having specific implementation per backend of iterating over full/sparse rows and creating columns and frames add a generic class/method to add rows to a frame e.g. Frame.AddRow(index interface{}, row map[string]interface)
We use Verbose
in configuration to set the log level and use that name in the code. IMO this is misleading and we should rename Verbose
to LogLevel
.
We'd like to be to work with various file formats - CSV, Parquet, JSOn ...
We'd also like to have a way to fetch the file from various locations: file system, S3, HDFS ...
We'd like the ability to copy data from one backend to another without passing through the client.
We don't like to give access to the whole file system. Have the CSV backend "chroot" via some settings.
Using the following code:
import pandas as pd
import numpy as np
import v3io_frames as v3f
backend, table = 'tsdb', 'miki_stocks'
df = pd.read_csv('_t/stocks.csv', parse_dates=['Date'], index_col='Date')
chunks = (c for _, c in df.groupby(np.arange(len(df))//100))
c = v3f.Client('http://localhost:8080')
c.write(backend, table, chunks)
Current we create a backend on every request. We'd like to create them when the server starts.
We need to think how clients can refrence a specific backend and how can we avoid connections being opened for the life of the server (pool?)
@yaronha said
u need to set the
Index
(column) name not just value,
@yaronha reported that the server show the error below but the client operation ended successfully with a nil
(None
?) DataFrame
18.10.27 15:51:19.101 ?[37m v3io-frames?[0m ?[34m(I)?[0m read request {"request": {"backend":"stream","schema":null,"data_format":"","row_layout":false,"multi_index":false,"query":"","table":"stockes_stream","columns":null,"filter":"","group_by":""
,"Join":null,"limit":0,"max_in_message":0,"marker":"","segments":null,"total_segment":0,"sharding_keys":null,"sort_key_range_start":"","sort_key_range_end":"","start":"now-2d","end":"","step":"","aggragators":"","seek":"time","shard":"0","sequence":140}}
18.10.27 15:51:19.265 ?[37m v3io-frames?[0m ?[31m(E)?[0m can't query {"error": "Error in Seek operation - Failed POST with status 400"}
18.10.27 15:51:19.265 ?[37m v3io-frames?[0m ?[31m(E)?[0m error reading {"error": "can't query: Error in Seek operation - Failed POST with status 400", "errorVerbose": "Error in Seek operation - Failed POST with status 400\ncan't query\ngithub.com/v
3io/frames/api.(*API).Read\n\tC:/Users/yaronh/go/src/github.com/v3io/frames/api/api.go:96\ngithub.com/v3io/frames/server.(*Server).handleRead.func1\n\tC:/Users/yaronh/go/src/github.com/v3io/frames/server/server.go:180\nruntime.goexit\n\tC:/Go/src/runtime/asm_am
d64.s:2361", "errorCauses": [{"error": "can't query: Error in Seek operation - Failed POST with status 400", "errorVerbose": "Error in Seek operation - Failed POST with status 400\ncan't query", "errorCauses": [{"error": "Error in Seek operation - Failed POST w
ith status 400"}]}]}
@tebeka we are missing a Boolean array type
@yaronha said:
RawIter, doesnt handle the case were new rows dont have some of the fields the old ones had, see below "
18.10.27 20:11:16.451 ?[37m v3io-frames?[0m ?[31m(E)?[0m error during iteration {"error": "Failed to create frame - \"cleaned\" column size mismatch (15 != 239)"}
18.10.27 20:11:16.453 ?[37m v3io-frames?[0m ?[31m(E)?[0m error reading {"error": "error during iteration: Failed to create frame - \"cleaned\" column size mismatch (15 != 239)", "errorVerbose": "Failed to create frame - \"cleaned\" column size mism
atch (15 != 239)\nerror during iteration\ngithub.com/v3io/frames/api.(*API).Read\n\tC:/Users/yaronh/go/src/github.com/v3io/frames/api/api.go:106\ngithub.com/v3io/frames/server.(*Server).handleRead.func1\n\tC:/Users/yaronh/go/src/github.com/v3io/frames/server/se
rver.go:180\nruntime.goexit\n\tC:/Go/src/runtime/asm_amd64.s:2361", "errorCauses": [{"error": "error during iteration: Failed to create frame - \"cleaned\" column size mismatch (15 != 239)", "errorVerbose": "Failed to create frame - \"cleaned\" column size mism
atch (15 != 239)\nerror during iteration", "errorCauses": [{"error": "Failed to create frame - \"cleaned\" column size mismatch (15 != 239)"}]}]}
in create operation allow user to select what happens if table already exist:
fail (default), overwrite (delete old and write), ignore (dont fail, keep working with the old)
Currently in slice column message data fields we use ints
for []int
floats
for []float
...
Instead, have the data field name match the dtype (e.g. '[]int`), this will simplify the logic both in server and clients.
It's confusing since Frame
has Labels
as well.
Go has standard way of access databases with database/sql. We should implement a frames
driver.
Currently, the server holds the connection information per backend. We'd like to change that and have the client pass connection information with every request (very much like regular database driver).
If the column type is not string, convert all values to strings.
Maybe make it look more like a cursor?
frames.api
should not encode/decode. It should work with pure Go data structures (possibly use channels for streaming)
@yaronha said:
i notice in kv/reader
Next()
we have index in both Indices & Columns (only delete it fromByName
(delete(byName, indexColKey))
can u fix it so the kv Index col (__name) only adds to the indexes and is not duplicated in Columns (see pic below w the issue):
Clients should get URL and api key from environment if it's empty.
@yaronha said:
df.index.levels
only return the unique values, wheredf.index.labels
returns the mapping, the len(col) will not be the frame size if index values repeat and it cause an error on servercheckEqualLen()
i assume u need to use
df.index.get_level_values()
and we can optimize that iflen(level)==1
we can make it a Label column (i.e. the same value prob repeat it self in all the rows since there is only one such val)
see: http://pandas.pydata.org/pandas-docs/stable/advanced.html
Assuming config.toml
is
verbose = "info"
[[backends]]
name = "weather"
type = "csv"
rootDir = "/tmp/weather"
The run
#!/bin/bash
root_dir=/tmp/weather
rm -rf ${root_dir}
mkdir -p ${root_dir}
cp _examples/client/py/weather.csv ${root_dir}
go run cmd/framesd/framesd.go -config _t/config.toml
And then
PYTHONPATH=clients/py python _examples/client/py/client.py
You'll see the generated file is truncated.
Currently the backend gets only it's own configuration section. It might require more configuration values (such as log level)
We'd like to have an HTTP interface for grafana.
We just need a skeleton, @yaronha will take over and fill it.
Currently the Python client checks if there's only one value in a column and then encodes it as a label column. However the check to nunique
is expensive (84.2µs for a column with 3 unique elements and size of 1000) - Remove it.
Currently it raises MessageError: column without data
. We need to return an empty pd.Series
with the right name and dtype.
Probably
AppendRow(row map[string]interface{}, index map[string]interface{})`
@yaronha said:
Sometimes the client hangs when reading from channel.
We need to decide what to do with empty column names.
Looking at pandas:
read_csv
will automatically assign names to columns if it's empty.A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.