Importing less than half of nodes in CSV file about batch-import HOT 5 CLOSED

Hey @kylemarkwilliams,

can you provide head -100 nodes.csv of your nodes and rels file ?

Probably something about delimeters?

Also, are you sure that there are no duplicate id's ?

from batch-import.

Hi @jexp

Thanks for the response. I will send you the first 100 lines of the nodes and rels via email since I can't attach them here.

I suspect you're right about it maybe being related to delimiters. My data is very messy (it was automatically extracted) so there may be all sort of weird things going on with it. I've already stripped all non-ASCII characters since the import was failing due to bad characters before I did that.

I suspect that there may be some double quotes (") in my data so for now I will try adding batch_import.csv.quotes=false to my batch.properties and see if that helps.

Also, yes. I'm pretty sure about duplicate id's. I did a select count(id) and select count(distinct id) on my relational database and got the same number of rows.

Thanks

from batch-import.

I managed to resolve this issue and will post the solution here (feel free to remove it if it's not appropriate). I followed the following steps to successfully import all 23million nodes:

1. Exported my data from MySQL using the --default-character-set=utf8 option
2. Verified that my exported data had the correct number of tab separated columns: awk -F"\t" 'NF!=$n {print}' $file where $n was the number of columns and $file was the file I exported to.
3. I edited batch.properties to use a whole lot more memory
4. I added batch_import.csv.quotes=false to batch.properties to ignore quotes (I exported from MySQL without quotes)
5. I edited src/main/java/org/neo4j/batchimport/utils/Chunker.java and changed private static final int BUFSIZE = 32*1024; to private static final int BUFSIZE = 64*1024; since the import was failing on a node with particularly large properties
6. I ran batch-import with -Dfile.encoding=UTF-8 as mentioned in the config file.

By doing this I managed to import 23 million nodes successfully in under 10 minutes.

from batch-import.

Great! Well done.

Adapting batch.properties to your memory requirements is expected.

I can increase BUFSIZE to 1M or something

It would be cool if you could convert this comment into a small blog post, perhaps also describing your domain and how you exported the data from mysql? That would be awesome!

from batch-import.

Sure, will do that later this week and edit and update this post with a link when it's done.

from batch-import.