Comments (2)
Hi, sorry for the late reply.
q does load the data into memory for processing, but it does contain an automatic caching feature which might help for large files.
If you run q with the -C readwrite
parameter, then a cache for each file will be generated automatically (if the cache file doesn't already exist). That would cause the first execution to be slow, but all additional executions of q for that file will be extremely fast.
In order to create the cache file, you could run q -t -C readwrite "select count(*) from original-file.tsv
. This will create another file called original-file.tsv.qsql
.
After this preparation step (which will take a considerable time for a 10GB file), you will be able to do either of the following:
- Run additional q commands which use original-file.tsv (with either
-C readwrite
or-C read
as additional params). The cache file will be detected automatically and used, speeding up queries considerably. - Run additional q command directly against the .qsql file. For example
q -t "select ... from original-file.tsv.qsql where ..."
. These will use the cache file directly and will not even require the original tsv file. Since you'll be using the .qsql file directly, you can copy it to another machine and use it directly there as well, without having to copy the original file.
Hope that will help. I'd appreciate it if you can write down your impressions of the speedup here after testing this.
from q.
thank you @harelba
Hope that will help. I'd appreciate it if you can write down your impressions of the speedup here after testing this.
I will let you know! thank you!
from q.
Related Issues (20)
- Can we support more glibc version ? HOT 1
- Setting default delimiter in .qrc config file? HOT 2
- Escape table name to avoid filename replacement
- local install fails due to setup.py HOT 3
- Feature request: replace spaces in column names with underline
- HOT 1
- Cannot run WIndows build HOT 4
- How do I use it in GitHub Actions?
- Header -O problem with -b beautify and -f format HOT 1
- Using q inside Windows PowerShell causes encoding issues
- q can't be installed with home-brew anymore HOT 3
- Warning about BOM suggest incorrect flag
- header mis-interpretation HOT 1
- Exception ignored in _io.TextIOWrapper when piping output to less HOT 1
- IS DISTINCT FROM not working HOT 1
- ENH: Would you consider making q available through conda? HOT 6
- How does the speed depend on the environment? HOT 2
- Always getting "query error: near "~": syntax error" error HOT 4
- BUG: Packaging in broken
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from q.