Random subsampling of reads from BAM files about biobear HOT 5 CLOSED

Regrettably one cannot write files yet. I need to finish supporting reading CRAM files, then the next major thing is supporting file output.

Glad the random sort worked, I'd just mention two things that may help in the future:

1. the BAM reader uses projection pushdown meaning that if you only want certain columns, if you specify only those columns in the query, only those fields will be deserialized from the BAM file which can really speed things up
2. SELECT * FROM bam_scan('file.bam') WHERE RANDOM() < .05 might be another good option if you want a proportion of the data and not a specific LIMIT cound. That wouldn't require a SORT and thus could return the data in a single scan.

from biobear.

@abearab Hey, I think you were very close... could you try, ORDER BY RANDOM(), though I don't think you can set the seed if that's critical.

Also, I should make this clearer, but biobear leverages Apache DataFusion for general purpose SQL functions, standard joins, etc, and you can see info about which functions are available here: https://arrow.apache.org/datafusion/user-guide/sql/index.html (in addition to the specialized ones I've added like reverse_complement). E.g. the docs for random.

from biobear.

@abearab Hey, I think you were very close... could you try, ORDER BY RANDOM(), though I don't think you can set the seed if that's critical.

I see, thanks for the correction. I just tried it and it's seems it's working now.

Also, I should make this clearer, but biobear leverages Apache DataFusion for general purpose SQL functions, standard joins, etc, and you can see info about which functions are available here: https://arrow.apache.org/datafusion/user-guide/sql/index.html (in addition to the specialized ones I've added like reverse_complement). E.g. the docs for random.

Cool, I'll look into datafusion docs. Thanks

One question: how can I write bam files after I made these kind of manipulation?

from biobear.

I think you answered my question here, I'm going to close this issue for now to cleanup your repo! Thanks

from biobear.

Thanks!

from biobear.