twentyfourg / developer-sdk Goto Github PK

PKG: model-data-generator

Right now the script will fail if it encounters any error from SQL. This is especially frustrating with these large datasets because the script may spend 30 minutes generating values, only to fail an insert almost immediately, wasting all that time.

I think making the script resilient to UNQ constraint violations would help mitigate this.

PKG: model-data-generator

When load testing I was able to consistently get script failures because of UNQ key violations on email columns, though I suspect this is an issue for more than just the email datatype. I was attempting to generate 100,000 users, but was not able to reliably get over 20,000.

I think the value-space for many of the data-generates needs to be widened, and perhaps more stringent checks put in place to ensure that inserted data will be unique. This is especially relevant if the script crashed during a previous populate attempt and there are existing values in the database. The script seems to assume that the database is empty, but if I am trying to generate 10s of thousands of values, I don't want to have to TRUNCATE tables to add more data because it can take several minutes to an hour to do this.