Comments (15)
@retorquere sorry for the delay in my reply. It would be certainly possible to add a new datasource to Mirador. You should look into miralib, which is the package that contains all the low-level data handling and statistical calculations.
In particular, look at how the DataSet class in miralib uses MiraTable, which in turns extends Table from Processing core. Table supports csv, tsv, ods, and bin formats, but should not too difficult to extend this code to use other sources. I'm happy to talk more about it.
from mirador.
Does that mean I should expect to make changes to all three classes, or are DataSet and MiraTable just for reference, and would I only be changing Table? I've looked at table and in particular its csv parsing, as the DB would just deliver columnar data without having to sift through an XML format such as with ODS, but I haven't yet figured out what the parsers do in response to their input. Do they call callbacks (doesn't look like it), construct a data structure (couldn't find it), a string (sort of looks like it does, but I don't know how it is expected to look).
from mirador.
I would say that some refactoring in MiraTable would be needed in order to switch (internally) between Table to read csv/tsv/ods, and other classes to handle SQL databases, etc., depending on the input source. MiraTable should encapsulate all this functionality so there is no need to know about it at the level of the DataSet API, which is what Mirador relies on.
What I would do is to implement some concept of parser to handle the appropriate source, either at the level of MiraTable, or at the level of DataSet, so the public API of DataSet does not change.
from mirador.
The source in this case would be a databases (InfluxDB in my case), so there wouldn't be parsing involved as such; the source knows what columns are present and what type they are, and would just hand you data row-by-row.
from mirador.
ok, I did some refactoring in miralib to make it easier to add support for other data sources: https://github.com/mirador/miralib/issues/16
You would need to write your InfluxDB wrapper as an implementation of the new DataSource interface. Use MiraTable as a reference, and let me know if you have any questions.
from mirador.
Super, just a few questions:
- I see the interface has
getRowCount
, this means I'll have to load the (potentially quite big) dataset in memory. Is that OK? What kind of volumes is miralib equipped to handle? - I assume the enum that holds the column type identifiers lives in
processing.data.TableRow
, but I can't find its source MiraTable
inherits fromTable
, but I can't find its sourcegetRow
returnsTableRow
, but I can't find its source to see how I should set it up.
from mirador.
- You don't need to load the entire dataset in memory, as long as you can return any row i when it is requested with the getRow(int i) method then things should be ok. In order to generate pairwise plots and calculate correlations, miralib creates copies or "slices" of the data that are disposed as soon as the plot is out of view, so memory consumption should be reasonable even for "large" datasets. For large, I mean in the order of a few million rows. I have tried Mirador with datasets that big, and it is usable.
- processing.data.TableRow is an interface defined here. This is the implementation I'm using in MiraTable.
- Table is defined here.
Note that miralib is built on top of the data classes from Processing.
from mirador.
So if I don't know the rowcount beforehand, what should I return from getRowCount
?
from mirador.
Well, Mirador needs a fixed sample size (the row count) to generate all the plots (histograms and eikosograms), as well as to evaluate the mutual information and other statistics.
These plots and calculations are all dynamic though, means that if the row count is in itself variable, the next time they are generated the new count will be used. But I haven't tested such situation. It could be feasible to add an internal timer in miralib to update the dataset at regular intervals, in order to support dynamic sources.
In any case, you would need to provide a row count greater than zero at any given moment in order to generate anything with Mirador.
from mirador.
My java just isn't good enough I'm afraid. I don't know how to put together things so that I can compile and test them. I'm OK with closing this issue.
from mirador.
I wrote some brief notes on how to compile Mirador with ant from the command line in the wiki. I can add some more details if that helps.
from mirador.
So I start the whole build from the mirador clone, not build the individual projects first?
from mirador.
I think so, since the main Mirador ant script uses the .class files from the dependencies to build the final package. Give it a try, in the meantime I will do some tests on my own (since I have everything already setup in Eclipse sometimes is easy to overlook problem with fresh installs) and update the wiki accordingly.
from mirador.
I'm doing everything from the command line when I can. I'll set up Eclipse if it's necessary, but if command-line ant works I'll take that.
from mirador.
Closing as there are no updates on this issue.
from mirador.
Related Issues (20)
- Save all use histories for a dataset HOT 1
- Support HiDPI displays on Windows HOT 1
- Add list of recently opened files to welcome screen HOT 1
- UI operations slow down when doing key presses HOT 1
- Selector in column labels show briefly when scrolling quickly HOT 1
- Add plot edges for low-contrast HOT 1
- Need horizontal scrollbar for group/table/var HOT 1
- Mirador fails to open HOT 7
- Show missing percentage on hover HOT 1
- Set plot color HOT 1
- Set plot width/height in preferences HOT 1
- Export selected pairs in Jupyter notebook
- Not starting: java.lang.IndexOutOfBoundsException HOT 6
- Windows Defender finds a virus in the installer HOT 1
- Resize plots from the UI
- Tiny graphics on Linux/Gnome
- 1.5.1 / latest macOS release HOT 2
- $RECYCLE.BIN file is considered binary data file HOT 1
- Opening data file on desktop gives error
- Opening csv file gives RuntimeException
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mirador.