Comments (2)
thanks @Hoeze do you deal with a lot of malformed schemas?
You can also read a vcf that does conform and use df.schema to get the schema
The challenge is that schemas do vary considerably depending on what INFO, FORMAT and FILTER fields are present.
If VCFs are not conforming often it is prudent to reprocess the VCF files from BAM or FASTQ or write a script to modify them before loading into Spark
from glow.
@williambrandler Yes, that's exactly how I got my default schema.
My concrete problem was that some VCFs had (seemingly hand-edited) VCF headers where field data types differed.
It took me a while to find out that the best solution to fix this issue is to just specify .schema()
before loading them :)
from glow.
Related Issues (20)
- VCF files with spaces in the file name cannot be read HOT 4
- configure sparkSession with glow HOT 2
- VEP exone/intron annotations failure HOT 5
- Improve handling of environment variables in the pipe transformer HOT 5
- Installing and running glow outside of databricks HOT 11
- Release with spark3.2 support HOT 1
- Vulnerable shared library might make glow-spark3 vulnerable. Can you help upgrade to patch versions? HOT 5
- Docker build.sh script is not current in tagged release HOT 1
- Ask: add glow to chocolatey HOT 4
- VCF Infinity/NaN values are not handled according to VCF spec
- PySpark 3.3 support HOT 8
- Load glow mave package from Python HOT 5
- spark.read.format("vcf").load() fails for vcf.bgz files HOT 3
- Cannot write INFO fields with LongType to VCF HOT 2
- Python tests fail with KeyError: '_glow_regression_values' HOT 1
- Logistic regression ValueError: Null fit failed! HOT 2
- AnalysisException: Column 'num_workers' does not exist. HOT 1
- pipe transformer support for file input/output for command line apps HOT 2
- Interaction Tests with GLOW HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glow.