anatolyuss / frommysqltopostgresql Goto Github PK
View Code? Open in Web Editor NEW"FromMySqlToPostgreSql" - the database migration tool.
License: GNU General Public License v3.0
"FromMySqlToPostgreSql" - the database migration tool.
License: GNU General Public License v3.0
php index.php sample_config.json
"FromMySqlToPostgreSql" - the database migration tool
Copyright 2015 Anatoly Khaytovich <[email protected]>
-- Migration began...
-- Script is terminated.
In the logs I see:
-- FromMySqlToPostgreSql::createSchema
-- Cannot create a new schema...
-- PDOException code: 2002
-- File: xyz/mysqldump/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 300
-- Message: SQLSTATE[HY000] [2002] No such file or directory
In arrangeColumnsData()
a function is used for some special column types. These column names are used for select and re-used for the INSERT statements but functions are not valid column names.
The script will create false statements like:
INSERT INTO "public"."table" ("id","IF(date_column
IN('0000-00-00', '0000-00-00 00:00:00'), '-INFINITY', date_column
)"
Giving the functions an alias like the original column name, it will fix the issue.
Where can i put my username and password for db
log as follow:
"FromMySqlToPostgreSql" - the database migration tool
Copyright 2015 Anatoly Khaytovich <[email protected]>
-- Migration began...
-- FromMySqlToPostgreSql::createSchema
-- Cannot create a new schema...
-- PDOException code: 42P06
-- File: /home/lihao/s/go/mysql2pg_migration/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 460
-- Message: SQLSTATE[42P06]: Duplicate schema: 7 ERROR: schema "public" already exists
-- SQL: CREATE SCHEMA "public";
May be when schema is 'public', no need to call createSchema()
This is the only error message I get and it happens instantly. I have tied the error down the fact that the code is not able to Create the Schema, but I don't know if that is because it is not able to connect or it can not find the server or something else.
Any ideas?
Since standard SQL is case insensitive, pgsql force user to add " around mixed-case names as for not seeking them in lower case only.
As a migration tool, this script may expose an option to allow lower-case rewriting of all names took from mysql.
This will prevent user to rewrite all his requests like such :
(mysql) SELECT * FROM MyTable <=> (pgsql) SELECT * FROM "MyTable" (with appropriate schemas and search_path).
Using SELECT * FROM MyTable in pgsql won't work directly because pgsql will look for mytable instead of MyTable.
When I launch the script I receive the next error.
-- FromMySqlToPostgreSql::createSchema
-- Cannot create a new schema...
-- PDOException code: 0
-- File: /home/usuario/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 292
-- Message: could not find driver
What do you think it lefts?
Thanks
hi,
i had a lot of tables to import with a lot of foreign keys.
i couldn't import them because the table orders can't be controlled or i didn't found how.
most of the datas couldn't be imported because the data they were referring datas that were created after them.
i found a workaround :
changing line 1256 of migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
from :
. implode(',', $arrPKs) . ') ON UPDATE ' . $strUpdateRule . ' ON DELETE ' . $strDeleteRule . ';';
to:
. implode(',', $arrPKs) . ') ON UPDATE ' . $strUpdateRule . ' ON DELETE ' . $strDeleteRule . ' DEFERRABLE;';
Adding DEFERRABLE makes postgresql check the foreign keys integrity only at the end of the transaction so everything was migrated correctly.
Maybe there is a better way ?
thanks.
-- PDOException code: 42501
-- File: /root/FromMySqlToPostgreSql-master/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 791
-- Message: SQLSTATE[42501]: Insufficient privilege: 7 ERROR: could not open file "/root/FromMySqlToPostgreSql-master/temporary_directory/allactions0.csv" for reading: Permission denied
-- SQL: COPY "arierra"."allactions" FROM '/root/FromMySqlToPostgreSql-master/temporary_directory/allactions0.csv' DELIMITER ',' CSV;
Script was downloaded by root and is running by root.
It would be interesting migrate the functions also
Hi,
I'm trying to use FromMySqlToPostgreSql to migrate a zabbix database. The process mainly goes well, but I'm getting some erros like these:
Message: SQLSTATE[23502]: Not null violation: 7 ERROR: column "def_shortdata" contains null values
-- SQL: ALTER TABLE "public"."actions" ALTER COLUMN "def_shortdata" SET NOT NULL;
Then I noticed that the "default" clauses are not exported/imported. For ex, the table "condition" in mysql is defined as:
CREATE TABLE conditions
(
conditionid
bigint(20) unsigned NOT NULL,
actionid
bigint(20) unsigned NOT NULL,
conditiontype
int(11) NOT NULL DEFAULT '0',
operator
int(11) NOT NULL DEFAULT '0',
value
varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (conditionid
),
KEY conditions_1
(actionid
),
CONSTRAINT c_conditions_1
FOREIGN KEY (actionid
) REFERENCES actions
(actionid
) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8
But when exported to postgres, it is defined as:
CREATE TABLE conditions
(
conditionid numeric NOT NULL,
actionid numeric NOT NULL,
conditiontype integer NOT NULL,
operator integer NOT NULL,
value character varying(255),
CONSTRAINT conditions_pkey PRIMARY KEY (conditionid),
CONSTRAINT conditions_actionid_fkey FOREIGN KEY (actionid)
REFERENCES actions (actionid) MATCH SIMPLE
ON UPDATE RESTRICT ON DELETE CASCADE
)
Any help?
Thanks in advance.
Hello,
When i run the script, i have this error:
Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "D:\RECETTE\FromMySqlToPostgreSql-master/temporary_directory/t_referentiel0.csv" for reading: No such file or directory
I tried modifying the code by creating the temporary directory with 777 permissions, but i still have the same error.
mkdir($this->strTemporaryDirectory, 0777, true);
What do you think is the problem?
I am getting this:
-- FromMySqlToPostgreSql::createSchema
-- Cannot create a new schema...
-- PDOException code: 2002
-- File: /Volumes/User Data/Users/pbhowmick/Projects/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 285
-- Message: SQLSTATE[HY000] [2002] No such file or directory
-------------------------------------------------------
HI,
When trying to import a zabbix database from mysql to postgres using FromMySqlToPostgreSql, I'm finding the following error:
-- FromMySqlToPostgreSql::populateTableWorker
-- PDOException code: 22P04
-- File: /var/lib/pgsql/FromMySqlToPostgreSql-master/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 641
-- Message: SQLSTATE[22P04]: Bad copy file format: 7 ERROR: extra data after last expected column
CONTEXT: COPY items, line 13500: "40320,8,,,10292,"Jboss activeSessions","grpsum[""{$SERVER_GROUP}"",""jmx["jboss.web:type=Manager,pa..."
-- SQL: SELECT itemid
,type
,snmp_community
,snmp_oid
,hostid
,name
,key_
,delay
,history
,trends
,status
,value_type
,trapper_hosts
,units
,multiplier
,delta
,snmpv3_securityname
,snmpv3_securitylevel
,snmpv3_authpassphrase
,snmpv3_privpassphrase
,formula
,error
,lastlogsize
,logtimefmt
,templateid
,valuemapid
,delay_flex
,params
,ipmi_sensor
,data_type
,authtype
,username
,password
,publickey
,privatekey
,mtime
,flags
,interfaceid
,port
,description
,inventory_link
,lifetime
,snmpv3_authprotocol
,snmpv3_privprotocol
,state
,snmpv3_contextname
,evaltype
FROM items
LIMIT 0, 36445;
COPY "public"."items" FROM '/var/lib/pgsql/FromMySqlToPostgreSql-master/temporary_directory/items0.csv' DELIMITER ',' CSV;
Table definition is as follows:
CREATE TABLE items
(
itemid
bigint(20) unsigned NOT NULL,
type
int(11) NOT NULL DEFAULT '0',
snmp_community
varchar(64) NOT NULL DEFAULT '',
snmp_oid
varchar(255) NOT NULL DEFAULT '',
hostid
bigint(20) unsigned NOT NULL,
name
varchar(255) NOT NULL DEFAULT '',
key_
varchar(255) NOT NULL DEFAULT '',
delay
int(11) NOT NULL DEFAULT '0',
history
int(11) NOT NULL DEFAULT '90',
trends
int(11) NOT NULL DEFAULT '365',
status
int(11) NOT NULL DEFAULT '0',
value_type
int(11) NOT NULL DEFAULT '0',
trapper_hosts
varchar(255) NOT NULL DEFAULT '',
units
varchar(255) NOT NULL DEFAULT '',
multiplier
int(11) NOT NULL DEFAULT '0',
delta
int(11) NOT NULL DEFAULT '0',
snmpv3_securityname
varchar(64) NOT NULL DEFAULT '',
snmpv3_securitylevel
int(11) NOT NULL DEFAULT '0',
snmpv3_authpassphrase
varchar(64) NOT NULL DEFAULT '',
snmpv3_privpassphrase
varchar(64) NOT NULL DEFAULT '',
formula
varchar(255) NOT NULL DEFAULT '',
error
varchar(2048) NOT NULL DEFAULT '',
lastlogsize
bigint(20) unsigned NOT NULL DEFAULT '0',
logtimefmt
varchar(64) NOT NULL DEFAULT '',
templateid
bigint(20) unsigned DEFAULT NULL,
valuemapid
bigint(20) unsigned DEFAULT NULL,
delay_flex
varchar(255) NOT NULL DEFAULT '',
params
text NOT NULL,
ipmi_sensor
varchar(128) NOT NULL DEFAULT '',
data_type
int(11) NOT NULL DEFAULT '0',
authtype
int(11) NOT NULL DEFAULT '0',
username
varchar(64) NOT NULL DEFAULT '',
password
varchar(64) NOT NULL DEFAULT '',
publickey
varchar(64) NOT NULL DEFAULT '',
privatekey
varchar(64) NOT NULL DEFAULT '',
mtime
int(11) NOT NULL DEFAULT '0',
flags
int(11) NOT NULL DEFAULT '0',
interfaceid
bigint(20) unsigned DEFAULT NULL,
port
varchar(64) NOT NULL DEFAULT '',
description
text NOT NULL,
inventory_link
int(11) NOT NULL DEFAULT '0',
lifetime
varchar(64) NOT NULL DEFAULT '30',
snmpv3_authprotocol
int(11) NOT NULL DEFAULT '0',
snmpv3_privprotocol
int(11) NOT NULL DEFAULT '0',
state
int(11) NOT NULL DEFAULT '0',
snmpv3_contextname
varchar(255) NOT NULL DEFAULT '',
evaltype
int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (itemid
),
UNIQUE KEY items_1
(hostid
,key_
),
KEY items_3
(status
),
KEY items_4
(templateid
),
KEY items_5
(valuemapid
),
KEY items_6
(interfaceid
),
CONSTRAINT c_items_1
FOREIGN KEY (hostid
) REFERENCES hosts
(hostid
) ON DELETE CASCADE,
CONSTRAINT c_items_2
FOREIGN KEY (templateid
) REFERENCES items
(itemid
) ON DELETE CASCADE,
CONSTRAINT c_items_3
FOREIGN KEY (valuemapid
) REFERENCES valuemaps
(valuemapid
),
CONSTRAINT c_items_4
FOREIGN KEY (interfaceid
) REFERENCES interface
(interfaceid
)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
And the table line where it seems there is a problem follows:
*************************** 1. row ***************************
itemid: 40320
type: 8
snmp_community:
snmp_oid:
hostid: 10292
name: Jboss activeSessions
key_: grpsum["{$SERVER_GROUP}","jmx["jboss.web:type=Manager,path=/{$GRAU},host=localhost",activeSessions]",last,0]
delay: 30
history: 90
trends: 365
status: 0
value_type: 3
trapper_hosts:
units:
multiplier: 0
delta: 0
snmpv3_securityname:
snmpv3_securitylevel: 0
snmpv3_authpassphrase:
snmpv3_privpassphrase:
formula: 1
error:
lastlogsize: 0
logtimefmt:
templateid: NULL
valuemapid: NULL
delay_flex:
params:
ipmi_sensor:
data_type: 0
authtype: 0
username:
password:
publickey:
privatekey:
mtime: 0
flags: 0
interfaceid: NULL
port:
description:
inventory_link: 0
lifetime: 30
snmpv3_authprotocol: 0
snmpv3_privprotocol: 0
state: 0
snmpv3_contextname:
evaltype: 0
Any help?
Thanks in advance.
I keep getting these errors but I am not sure why, can you give me a hint? Thanks
-- FromMySqlToPostgreSql::populateTableWorker
-- PDOException code: 42501
-- File: /root/FromMySqlToPostgreSql-master/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 791
-- Message: SQLSTATE[42501]: Insufficient privilege: 7 ERROR: could not open file "/root/FromMySqlToPostgreSql-master/temporary_directory/action_log0.csv" for reading: Permission denied
-- SQL: COPY "database_name"."table_name" FROM '/root/FromMySqlToPostgreSql-master/temporary_directory/action_log0.csv' DELIMITER ',' CSV;
-------------------------------------------------------
I am trying to convert a database with your script.
it fails with the following error:
-- FromMySqlToPostgreSql::populateTableByPrepStmtWorker
-- PDOException code: 42601
-- File: /cygdrive/c/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 575
-- Message: SQLSTATE[42601]: Syntax error: 7 ERROR: INSERT has more expressions than target columns
LINE 1: ...sMode" ("id","name","index") VALUES( $1, $2, $3, '-INFINITY...
^
-- SQL: INSERT INTO "test"."Test" ("id","name","index") VALUES( :id, :name, :index, '-INFINITY', :index);
As you can see, it is trying to insert 5 columns in a table with only 3 columns. It is doing this for all the tables of the database.
Here is the create code for this table:
CREATE TABLE `Test` (
`id` BIGINT(20) NOT NULL AUTO_INCREMENT,
`name` CHAR(10) NOT NULL COLLATE 'utf8_bin',
`index` BIGINT(20) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE INDEX `name` (`name`),
UNIQUE INDEX `index` (`index`)
)
COLLATE='utf8_bin'
ENGINE=InnoDB
AUTO_INCREMENT=4
;
Hey AnatolyUss,
Please help!
What user should I run this php?
As you can see , I have a reading permission issue in csv file generated . I have already put the maximum permissions but still can not migrate.
[paulo@localhost temporary_directory]$ tail -f ../logs_directory/all.log
-- FromMySqlToPostgreSql::populateTable
-- PDOException code: 42501
-- File: /tmp/migrabanco/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 807
-- Message: SQLSTATE[42501]: Insufficient privilege: 7 ERRO: não pôde abrir arquivo "/tmp/migrabanco/temporary_directory/emp_categoriarelatorio.csv" para leitura: Permissão negada
-- SQL: COPY "demas_3"."emp_categoriarelatorio" FROM '/tmp/migrabanco/temporary_directory/emp_categoriarelatorio.csv' DELIMITER ',' CSV;
In the documentation you say:
PDO_MYSQL should be installed and enabled
PDO_PGSQL should be installed and enabled
mbstring should be installed and enabled
register_argc_argv should be enabled (check php.ini).
postgis should be installed and enabled to migrate spatial data (geometry type columns).
But how do I install this after installing the PHP CLI ?
PS: I'm on Windows and I don't know anything about PHP :)
Quote: Ease of use - the only thing needed to run this script is the PHP(CLI) interpreter.
Hi,
Currently, your script seems to move unique indexes from mysql to pgsql 'unique' indexes.
I think it's better to use constraints with query like :
ALTER TABLE public."test_table"
ADD CONSTRAINT test UNIQUE (columns) USING INDEX TABLESPACE pg_default;
instead of things like
CREATE UNIQUE INDEX
ON public."iRcom_networks_ipNodes_tags" (name ASC NULLS LAST, value ASC NULLS LAST);
Pgsql's docs say index creation for unique enforcing is a detail of implementation and shouldn't be accessed directly.
https://www.postgresql.org/docs/current/static/indexes-unique.html
I see two big advantages :
Thank you in advance to deal with this
Thanks for your great script. I discovered double single quotes in PostgreSQL whereas in MySQL there was only a single one. Presumably this happened in line 637 of migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php.
I created pull request #44 which should address this issue.
-- FromMySqlToPostgreSql::populateTableByPrepStmtWorker
-- PDOException code: HY093
-- File: /home/www/FromMySqlToPostgreSql-master/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 642
-- Message: SQLSTATE[HY093]: Invalid parameter number: :Хуясе_колонка9
-- SQL:
INSERT INTO "mobsted_2"."objects" ("id","PIN","Phone","untitled22","untitled229","untitled230","untitled231","untitled232","untitled233","untitled235","un
titled236","untitled237","untitled238","untitled239","untitled240","mycol00","Хуясе_колонка9","Organization","Device","Model","Expenditure","Photo","Manual","Video","unti
tled242","untitled243","untitled244","untitled246","Fuck_Taht_Columnd","untitled249dd00","untitled250","untitled251","untitled252","untitled253","untitled254","untitled25
50","jj","Denis99","untitled25512","NameAZS","AddressAZS","untitled25513","untitled25514","untitled25515ррр","Колонка_1","untitled25516","Name","untitled25517fgdfg","unti
tled25518dfgdfgdfg","untitled25519334","ФИО","Номер_водительского","Адрес","Дом","Улица","New_column","gfdhhgfdhfghfd","tryeyreyeryrety","ColdCounter1Serial","ColdCounter
2Serial","HotCounter1Serial","HotCounter2Serial","Квартира","PrinterName","Office","untitled2551935","Room_Number","Address","FIO","Contacts","RoomNumber","ClassOfService
","FirstLastName","sduntitled2551936sdf","HelloDenisMuyName","untitled2551937df","untitled2551938d","AZSAdrees","AZSNumber","untitled2551939","untitled2551940hh","Вода1",
"untitled2551941g","untitled2551942","House","CounterHot","CounterCold","BranchOfficeNumber","BranchOfficeCity","TotalRating","untitled2551943","surname","Personnel_Numbe
r","DateOfBirth","Модель","Производитель","Серийный_номер","Модель_картриджа","Организация","Ответственный_сотрудник_от_организации","Контакты_огранизации","Enabled","Tra
nsport","LastLogin","InviteStatus","untitled2551944","Тип_устройства","фыфы","untitled2551945","Last_Name","Surname2","DateOfOrder","untitled2551946","untitled2551947","P
hone2","Лицевой_счет","Email","sdgsdgsdgsdg","untitled25519335","untitled25519336","ident","Куда","Откуда","От_кого","Местонахождение_посылки","untitled25519337авп","unti
tled25519338","untitled25519339f","Объем_материала","Номер_АЗС","Адрес_АЗС","Контактные_данные","Город","untitled25519340d","untitled25519341xfg","Email1","untitled255193
42","Files","Наименование_Организации","ИНН","КПП","Телефон","Email_для_связи","Баланс","Задолженность","Контактное_лицо","untitled25519343","untitled25519344d","img","bo
ld","FileId","File5","untitled25519345","File3","asd","deded","1212","untitled25519348","untitled25519350sdfsdf","untitled25519351","untitled25519352","untitled25519353",
"untitled25519354","ываываыва","Selfie","Ебать_Ту_Люсю","untitled25519355","untitled25519356","untitled25519357","Tenant","Depends","untitled25519358","untitled25519359",
"untitled25519360","Тестируем_и_Яшеем_2","Depend","superColumn","mob","Sted","untitled25519363","Модель_фильтра","MAS_S","MAS_C","MAS_M","Mas_Z","Информация_о_минералах",
"Фото_места_установки","Дата_установки","Отзыв","untitled25519364","sexybaba","hhh","ebaka","untitled25519365","untitled25519366ываываываываываываываываываываываыва","unt
itled25519367","NameThisColumn2","NameThisColumn3","NameThisColumn4","NameThisColumn5","untitled25519368","NameThisColumn6","NameThisColumn7","NameThisColumn9","Customer"
,"NameThisColumn10","NameThisColumn11","NameThisColumn12","NameThisColumn13m","DateCreate","NameThisColumn14","Номер_кабинета","NameThisColumn16","NameThisColumn17","Name
ThisColumn18","NameThisColumn19","NameThisColumn20","NameThisColumn21","NameThisColumn22","NameThisColumn23","NameThisColumn24","NameThisColumn25","NameThisColumn26","Nam
eThisColumn27","NameThisColumn28","NameThisColumn30","NameThisColumn31","NameThisColumn32","NameThisColumn33","NameThisColumn34","Language","gorod","хим_адрес","хим_тел",
"html_pay","studio_phone","Fullname","html_zakaz","UID","NameThisColumn36","Demo","Номер_кабиента","Время_работы","Врач1","Врач2","Врач3","ОбслуживающийОфис","Адрес_офиса
","ТелефонОфиса","Лечащий_Врач","Телефон_Лечащего_врача","Возраст","отзыв777","отзыв999","варвар","StartScreen","sum","NameThisColumn37","NameThisColumn38","NameThisColum
n39","NameThisColumn40","NameThisColumn41","NameThisColumn42","NameThisColumn43","Город2ewr","Модель_домофона","Статус","УК","Фото_Домофон","Инструкция_Домофон","Подъезд"
,"Телефон_УК","apiPa","Test","Изображение_Устройства","Имя_Отчество","Видео_Инструкция","Инструкция_для_абонентского_устройства","Характеристики_устройства","Модель_абоне
нтского_устройства_","Адрес_установки","ЛС","NameThisColumn44","NameThisColumn45","NameThisColumn46","NameThisColumn47","Имя","Фамилия","Отчество","Модель_АУ","apiInfo","
Инструкция_для_АУ","Фото_АУ","Уведомление","Информирование","Микрорайон","Корпус","Счетчик_ГВ","Счетчик_ХВ","Счетчик_ЭЭ","Дата_ЭЭ","Дата_ХВ","Дата_ГВ","Картинка","Видео",
"PayOnline","Карта","Контактный_телефон","Логотип_организации","Адрес_доставки","Status777","newcolumn","хим_название","Cert_pay","NameThisColumn48","AutomaticalTimezone"
,"Timezone","NameThisColumn49","City","current_order","NewColonka","Kol1","Kol2") VALUES( :id, :PIN, :Phone, :untitled22, :untitled229, :untitled230, :untitled231, :unti
tled232, :untitled233, :untitled235, :untitled236, :untitled237, :untitled238, :untitled239, :untitled240, :mycol00, :Хуясе_колонка9, :Organization, :Device, :Model, :Exp
enditure, :Photo, :Manual, :Video, :untitled242, :untitled243, :untitled244, :untitled246, :Fuck_Taht_Columnd, :untitled249dd00, :untitled250, :untitled251, :untitled252,
:untitled253, :untitled254, :untitled2550, :jj, :Denis99, :untitled25512, :NameAZS, :AddressAZS, :untitled25513, :untitled25514, :untitled25515ррр, :Колонка_1, :untitled
25516, :Name, :untitled25517fgdfg, :untitled25518dfgdfgdfg, :untitled25519334, :ФИО, :Номер_водительского, :Адрес, :Дом, :Улица, :New_column, :gfdhhgfdhfghfd, :tryeyreyer
yrety, :ColdCounter1Serial, :ColdCounter2Serial, :HotCounter1Serial, :HotCounter2Serial, :Квартира, :PrinterName, :Office, :untitled2551935, :Room_Number, :Address, :FIO,
:Contacts, :RoomNumber, :ClassOfService, :FirstLastName, :sduntitled2551936sdf, :HelloDenisMuyName, :untitled2551937df, :untitled2551938d, :AZSAdrees, :AZSNumber, :untit
led2551939, :untitled2551940hh, :Вода1, :untitled2551941g, :untitled2551942, :House, :CounterHot, :CounterCold, :BranchOfficeNumber, :BranchOfficeCity, :TotalRating, :unt
itled2551943, :surname, :Personnel_Number, :DateOfBirth, :Модель, :Производитель, :Серийный_номер, :Модель_картриджа, :Организация, :Ответственный_сотрудник_от_организаци
и, :Контакты_огранизации, :Enabled, :Transport, :LastLogin, :InviteStatus, :untitled2551944, :Тип_устройства, :фыфы, :untitled2551945, :Last_Name, :Surname2, :DateOfOrder
, :untitled2551946, :untitled2551947, :Phone2, :Лицевой_счет, :Email, :sdgsdgsdgsdg, :untitled25519335, :untitled25519336, :ident, :Куда, :Откуда, :От_кого, :Местонахожде
ние_посылки, :untitled25519337авп, :untitled25519338, :untitled25519339f, :Объем_материала, :Номер_АЗС, :Адрес_АЗС, :Контактные_данные, :Город, :untitled25519340d, :untit
led25519341xfg, :Email1, :untitled25519342, :Files, :Наименование_Организации, :ИНН, :КПП, :Телефон, :Email_для_связи, :Баланс, :Задолженность, :Контактное_лицо, :untitle
d25519343, :untitled25519344d, :img, :bold, :FileId, :File5, :untitled25519345, :File3, :asd, :deded, :1212, :untitled25519348, :untitled25519350sdfsdf, :untitled25519351
, :untitled25519352, :untitled25519353, :untitled25519354, :ываываыва, :Selfie, :Ебать_Ту_Люсю, :untitled25519355, :untitled25519356, :untitled25519357, :Tenant, :Depends
, :untitled25519358, :untitled25519359, :untitled25519360, :Тестируем_и_Яшеем_2, :Depend, :superColumn, :mob, :Sted, :untitled25519363, :Модель_фильтра, :MAS_S, :MAS_C, :
MAS_M, :Mas_Z, :Информация_о_минералах, :Фото_места_установки, :Дата_установки, :Отзыв, :untitled25519364, :sexybaba, :hhh, :ebaka, :untitled25519365, :untitled25519366ыв
аываываываываываываываываываываыва, :untitled25519367, :NameThisColumn2, :NameThisColumn3, :NameThisColumn4, :NameThisColumn5, :untitled25519368, :NameThisColumn6, :NameT
hisColumn7, :NameThisColumn9, :Customer, :NameThisColumn10, :NameThisColumn11, :NameThisColumn12, :NameThisColumn13m, :DateCreate, :NameThisColumn14, :Номер_кабинета, :Na
meThisColumn16, :NameThisColumn17, :NameThisColumn18, :NameThisColumn19, :NameThisColumn20, :NameThisColumn21, :NameThisColumn22, :NameThisColumn23, :NameThisColumn24, :N
ameThisColumn25, :NameThisColumn26, :NameThisColumn27, :NameThisColumn28, :NameThisColumn30, :NameThisColumn31, :NameThisColumn32, :NameThisColumn33, :NameThisColumn34, :
Language, :gorod, :хим_адрес, :хим_тел, :html_pay, :studio_phone, :Fullname, :html_zakaz, :UID, :NameThisColumn36, :Demo, :Номер_кабиента, :Время_работы, :Врач1, :Врач2,
:Врач3, :ОбслуживающийОфис, :Адрес_офиса, :ТелефонОфиса, :Лечащий_Врач, :Телефон_Лечащего_врача, :Возраст, :отзыв777, :отзыв999, :варвар, :StartScreen, :sum, :NameThisCol
umn37, :NameThisColumn38, :NameThisColumn39, :NameThisColumn40, :NameThisColumn41, :NameThisColumn42, :NameThisColumn43, :Город2ewr, :Модель_домофона, :Статус, :УК, :Фото
_Домофон, :Инструкция_Домофон, :Подъезд, :Телефон_УК, :apiPa, :Test, :Изображение_Устройства, :Имя_Отчество, :Видео_Инструкция, :Инструкция_для_абонентского_устройства, :
Характеристики_устройства, :Модель_абонентского_устройства_, :Адрес_установки, :ЛС, :NameThisColumn44, :NameThisColumn45, :NameThisColumn46, :NameThisColumn47, :Имя, :Фам
илия, :Отчество, :Модель_АУ, :apiInfo, :Инструкция_для_АУ, :Фото_АУ, :Уведомление, :Информирование, :Микрорайон, :Корпус, :Счетчик_ГВ, :Счетчик_ХВ, :Счетчик_ЭЭ, :Дата_ЭЭ,
:Дата_ХВ, :Дата_ГВ, :Картинка, :Видео, :PayOnline, :Карта, :Контактный_телефон, :Логотип_организации, :Адрес_доставки, :Status777, :newcolumn, :хим_название, :Cert_pay,
:NameThisColumn48, :AutomaticalTimezone, :Timezone, :NameThisColumn49, :City, :current_order, :NewColonka, :Kol1, :Kol2);
mysql utf8 to pg utf8, cyrillic are: '???????????? ????????'
I am getting this error after migration:
App 555161 stderr: Completed 500 Internal Server Error in 936ms (ActiveRecord: 27.2ms)
App 555161 stderr:
App 555161 stderr: ActiveRecord::StatementInvalid (PG::UndefinedFunction: ERROR: operator does not exist: smallint = boolean
App 555161 stderr: LINE 1: ...) AND (members.user_id = 2 OR (projects.is_public = TRUE AND...
migration:
From Redmine 3.3.3 to 4.2.1 (MySql to MySql) by using common Redmine migration. Checked working.
Then from MySQL to Postgres - then this error.
Note: the same error was when tried another approach: first migrate from v.3.3.3. MySQL to Postgres then update Redmine.
What could be wrong?
brgds & looking forward
I was wondering why you are supporting both programs, this one and NMIG? Is the plan to eventually deprecate this one and continue this good work with NMIG? If not, can you elaborate on when one would choose this one over NMIG? In other words, when would it be BETTER to use this program instead of NMIG?
Thanks in advance.
MySQL 4.1 does not have visible system metadata tables. That's why I asked.
Hi,
Thank you to have implemented comment migration today.
Is there any special option to set to enable it?
For now, I see no "-- Comment on column" in the output and no comment is moved on the pgsql db.
It sounds the last update broke up something : currently the migration process freeze over a quite large table (32k rows). I see no pending request on mysql side but on pgsql there is the request built in arrangeColumnsData() method. Are you sure pgsql likes it ?
There is a csv file corresponding to the processed table in the temporary_directory.
Here what I see in shell :
"FromMySqlToPostgreSql" - the database migration tool
Copyright 2015 Anatoly Khaytovich <[email protected]>
-- Migration began...
-- New schema "public" was successfully created...
-- 93 tables detected
-- Currently processing table: ...
-- Table "public"."...." is created.
-- Populating table "public"."..."
-- Total rows to insert into "public"."...": 50
-- Currently processing table: ...
-- Table "public"."..." is created.
-- Populating table "public"."...."
-- Total rows to insert into "public"."....": 0
-- Currently processing table: ...
-- Table "public"."...." is created.
-- Populating table "public"."...."
-- Total rows to insert into "public"."....": 32774
And then it waits, output nothing for a while... maybe 15 minutes
The for loop at line 900 of FromMySqlToPostgreSql.php might know something about this issue :)
Therefore you have PHP notice like :
PHP Notice: Undefined variable: sql in /tmp/frommysqltopgsql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php on line 568
Thank you in advance for any feedback !
Hi!
I'm tried migrate my DB from mysql to postgreSQL and got an error.
Errors-only.log:
-- FromMySqlToPostgreSql::populateTable
-- PDOException code: 58P01
-- File: C:\1\FromMySqlToPostgreSql\migration\FromMySqlToPostgreSql\FromMySqlToPostgreSql.php
-- Line: 624
-- Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "temp/Comment.csv" for reading: No such file or directory
-------------------------------------------------------
-- FromMySqlToPostgreSql::populateTable
-- PDOException code: 58P01
-- File: C:\1\FromMySqlToPostgreSql\migration\FromMySqlToPostgreSql\FromMySqlToPostgreSql.php
-- Line: 624
-- Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "temp/Follow.csv" for reading: No such file or directory
-------------------------------------------------------
-- FromMySqlToPostgreSql::populateTable
-- PDOException code: 58P01
-- File: C:\1\FromMySqlToPostgreSql\migration\FromMySqlToPostgreSql\FromMySqlToPostgreSql.php
-- Line: 624
-- Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "temp/Installation.csv" for reading: No such file or directory
-------------------------------------------------------
-- FromMySqlToPostgreSql::populateTable
-- PDOException code: 58P01
-- File: C:\1\FromMySqlToPostgreSql\migration\FromMySqlToPostgreSql\FromMySqlToPostgreSql.php
-- Line: 624
-- Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "temp/Like.csv" for reading: No such file or directory
-------------------------------------------------------
-- FromMySqlToPostgreSql::populateTable
-- PDOException code: 58P01
-- File: C:\1\FromMySqlToPostgreSql\migration\FromMySqlToPostgreSql\FromMySqlToPostgreSql.php
-- Line: 624
-- Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "temp/Post.csv" for reading: No such file or directory
-------------------------------------------------------
-- FromMySqlToPostgreSql::createTable
-- Cannot create table tweather1."User".
-- PDOException code: 42601
-- File: C:\1\FromMySqlToPostgreSql\migration\FromMySqlToPostgreSql\FromMySqlToPostgreSql.php
-- Line: 409
-- Message: SQLSTATE[42601]: Syntax error: 7 ERROR: syntax error at or near ","
LINE 1: ...untryCode" CHARACTER VARYING(3) ,"deviceToken" ,"displayN...
^
-------------------------------------------------------
I'm not a database expert, therefore I'm not sure if it was caused by the conversion tool but I ended up with the following error:
gerrit> SELECT nextval('change_id') from changes;
ERROR: ERROR: "change_id" is not a sequence
gerrit> \d changes
Table changes
COLUMN_NAME | TYPE
---------------------+--------------------------------------------------------------------
change_key | varchar(60) DEFAULT ''::character varying NOT NULL
created_on | timestamp DEFAULT now() NOT NULL
last_updated_on | timestamp DEFAULT '-infinity'::timestamp without time zone NOT NULL
owner_account_id | int4 DEFAULT 0 NOT NULL
dest_project_name | varchar(255) DEFAULT ''::character varying NOT NULL
dest_branch_name | varchar(255) DEFAULT ''::character varying NOT NULL
status | bpchar(1) DEFAULT ''::bpchar NOT NULL
current_patch_set_id | int4 DEFAULT 0 NOT NULL
subject | varchar(255) DEFAULT ''::character varying NOT NULL
topic | varchar(255)
row_version | int4 DEFAULT 0 NOT NULL
change_id | int4 DEFAULT 0 NOT NULL
original_subject | varchar(255)
submission_id | varchar(255)
Indexes on changes:
changes_pkey UNIQUE (change_id)
gerrit>
Reindex didn't help. Data can be queried, however inserts are failing.
Do you have any hint on what could have caused this?
Thanks.
Testcase:
create table zero (
ID
bigint(20) NOT NULL,
DATA
varchar(255) NOT NULL,
PRIMARY KEY (ID
)
) ENGINE=InnoDB;
create table one (
ID
bigint(20) NOT NULL,
DATA
varchar(255) NOT NULL,
PRIMARY KEY (ID
),
CONSTRAINT one_FK1
FOREIGN KEY (ID
) REFERENCES zero
(ID
)
) ENGINE=InnoDB;
create table two (
ID
bigint(20) NOT NULL,
one_ID
bigint(20) NOT NULL,
DATA
varchar(255) NOT NULL,
PRIMARY KEY (ID
),
CONSTRAINT two_FK1
FOREIGN KEY (one_ID
) REFERENCES one
(ID
)
) ENGINE=InnoDB;
create table three (
ID
bigint(20) NOT NULL,
one_ID
bigint(20) NOT NULL,
DATA
varchar(255) NOT NULL,
PRIMARY KEY (ID
),
CONSTRAINT three_FK1
FOREIGN KEY (one_ID
) REFERENCES one
(ID
)
) ENGINE=InnoDB;
insert into zero values ( 1, 'One' );
insert into one values ( 1, 'One' );
insert into two values ( 1, 1, 'Two' );
insert into three values ( 1, 1, 'Three' );
type "longblob" does not exist
https://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL
BINARY(n)
VARBINARY(n)
TINYBLOB
BLOB
MEDIUMBLOB
LONGBLOB
must convert to BYTEA ?
Hi,
I'm running Windows 10, Node 5.9.1, MySQL 5.7 and PostGres 9.5
I've successfully migrated a non-trivial database, but to do so I had to comment out './migration/fmtp/FromMySQL2PostgreSQL' line 1021 global.gc();
as it was throwing an error when it got that line "global.gc() is not a function".
Before commenting out the line I tried adding the command line parameter --expose-gc but it seemed to have no effect.
Removing the garbage collection didn't seem to phase my PC (I have 16 GBs of RAM).
Just letting you know in case the issue crops up for other people.
thanks
James
Is it possible to use this in order to convert a MySQL dump to PG format?
Hello very powerfull tool so when i migrate from mysql to postgres object type boolean mysql are migrated in type bit variying so i have some 0 values or postgres is waitng for false value any suggestions.
Hi,
I've tried the migration table and it seems not to support the json column type from MySQL.
thanx.
-- Currently processing table: bs... PHP Notice: Undefined index: json in /root/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/MapDataTypes.php on line 276 PHP Notice: Undefined variable: sql in /root/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php on line 568 -- Script is terminated.
There're many variants of storing date and time in postgresql, while only few in mysql. It would be good, if i could point which of datetime type i want to convert to.
Also, I have columns in MySQL, where i store JSON text - and i would like to have an ability to choose JSON or JSONB filed types for those columns.
Hi,
I would find interesting to preserve comments put on columns or tables.
The COMMENT pgsql statement may help to do so.
Btw thank you for this extremely useful stuff ;)
I'm trying to convert a Zabbix database and am running into issues with columns declared as 'int(11)' in MySQL (actually MariaDB)...
Here's the DDL for the table
DROP TABLE acknowledges;
CREATE TABLE acknowledges (
acknowledgeid bigint(20) UNSIGNED NOT NULL,
userid bigint(20) UNSIGNED NOT NULL,
eventid bigint(20) UNSIGNED NOT NULL,
clock int(11) NOT NULL DEFAULT '0',
message varchar(255) NOT NULL,
PRIMARY KEY(acknowledgeid)
)
Here's what I get in response...
-- Migration began...
-- New schema "zabbix" was successfully created...
-- 104 tables detected
-- Currently processing table: acknowledges...
PHP Notice: Undefined variable: sql in /var/lib/pgsql/FromMySqlToPostgreSql-master/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php on line 471
-- Script is terminated.
And here's the log...
-- Migration began...
-- New schema "zabbix" was successfully created...
-- 104 tables detected
-- Currently processing table: acknowledges...
-- FromMySqlToPostgreSql::createTable
-- Cannot create table "zabbix"."acknowledges".
-- PDOException code: 42601
-- File: /var/lib/pgsql/FromMySqlToPostgreSql-master/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 463
-- Message: SQLSTATE[42601]: Syntax error: 7 ERROR: syntax error at or near "INT"
LINE 1: ...knowledgeid" ,"userid" ,"eventid" ,"clock" INT ,"mess...
^
-------------------------------------------------------
Notice INT
after "clock".
I looked at MapDataTypes but didn't see anything obvious.
Any suggestions?
Tested on my database but views were not migrated, tables worked fine. Thank you.
Hello,
I am converting a phpbb3 site - thank you for the excellent tool! I did have the migration process working well the other day with the same data, etc, but today it is failing after I pulled master.
commit 837a32a
The CSV file was created, but the actual table in mysql has no rows. Not sure if that can be the issue. I am going to try v1.3.0.
-- PDOException code: 58P01 -- File: /home/FSPMigrate/my2pg/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php -- Line: 801 -- Message: SQLSTATE[58P01]: Undefined file: 7 ERROR: could not open file "/home/FSPMigrate/my2pg/temporary_directory/phpbb_bbcodes.csv" for reading: No such file or directory -- SQL: COPY "public"."phpbb_bbcodes" FROM '/home/FSPMigrate/my2pg/temporary_directory/phpbb_bbcodes.csv' DELIMITER ',' CSV; -------------------------------------------------------
Thanks!
Perry Clark
When running the script, it reports success, but I see a number of errors and no database is created. Each time I run it, the database name is incremented.
My relevant JSON config (everything else remains unchanged):
"source" : "mysql:host=localhost;port=3306;charset=UTF8;dbname=development_db,root,",
"target" : "pgsql:host=localhost;port=5432;dbname=mysql_import;options=--client_encoding=UTF8,agbodike,",
all.log (some select errors):
"FromMySqlToPostgreSql" - the database migration tool
Copyright 2015 Anatoly Khaytovich <[email protected]>
-- Migration began...
-- New schema "development_db_6" was successfully created...
-- 45 tables detected
...
-- PDOException code: 22P02
-- File: /Users/agbodike/workspace/agbodike/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 800
-- Message: SQLSTATE[22P02]: Invalid text representation: 7 ERROR: invalid input syntax for integer: "''"
CONTEXT: COPY baselines, line 1, column attachment_file_size: "''"
-- SQL: COPY "development_db_6"."baselines" FROM '/Users/agbodike/workspace/agbodike/FromMySqlToPostgreSql/temporary_directory/baselines0.csv' DELIMITER ',' CSV;
...
-- PDOException code: 22007
-- File: /Users/agbodike/workspace/agbodike/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 800
-- Message: SQLSTATE[22007]: Invalid datetime format: 7 ERROR: invalid input syntax for type timestamp: "''"
CONTEXT: COPY credit_cards, line 1, column deleted_at: "''"
-- SQL: COPY "development_db_6"."credit_cards" FROM '/Users/agbodike/workspace/agbodike/FromMySqlToPostgreSql/temporary_directory/credit_cards0.csv' DELIMITER ',' CSV;
...
-- FromMySqlToPostgreSql::processForeignKey
-- Error occurred when tried to create foreign key for table "development_db_6"."baseline_users"...
-- PDOException code: 23503
-- File: /Users/agbodike/workspace/agbodike/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 1361
-- Message: SQLSTATE[23503]: Foreign key violation: 7 ERROR: insert or update on table "baseline_users" violates foreign key constraint "baseline_users_user_id_fkey"
DETAIL: Key (user_id)=(8) is not present in table "users".
-- SQL: ALTER TABLE "development_db_6"."baseline_users" ADD FOREIGN KEY ("user_id") REFERENCES "development_db_6"."users" ("id") ON UPDATE RESTRICT ON DELETE RESTRICT;
...
-- FromMySqlToPostgreSql::populateTableByPrepStmtWorker
-- PDOException code: 22001
-- File: /Users/agbodike/workspace/agbodike/FromMySqlToPostgreSql/migration/FromMySqlToPostgreSql/FromMySqlToPostgreSql.php
-- Line: 680
-- Message: SQLSTATE[22001]: String data, right truncated: 7 ERROR: value too long for type character varying(255)
-- SQL: INSERT INTO "development_db_6"."users" ("id","email","encrypted_password","reset_password_token","reset_password_sent_at","remember_created_at","sign_in_count","current_sign_in_at","last_sign_in_at","current_sign_in_ip","last_sign_in_ip","created_at","updated_at","company_id","is_passcode","first_name","last_name","phone","is_primary","deleted_at","role_id","activated","avatar_file_name","avatar_content_type","avatar_file_size","avatar_updated_at","personal_website","tour_visibility") VALUES(:0,:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,:15,:16,:17,:18,:19,:20,:21,:22,:23,:24,:25,:26,:27);
That last error looks like it is inserting totally wrong information as I would expect the VALUES to be very different.
I guess json type not supported?
Could you please confirm that this tool does not change anything on the source database instance, so tests/experiments can be done from a production source database instance.
Thanks.
Since i didnt check out this project and just downloaded it, i dont have a pull request, but here's the patch trough Winmerge:
869,870d868
< $escapedField = '`' . $arrColumn['Field'] . '`';
<
873c871
< $fieldName = "hex(ST_AsWKB(" . $escapedField . "))";
---
> $fieldName = "hex(ST_AsWKB(" . $arrColumn['Field'] . "))";
875c873
< $fieldName = $escapedField;
---
> $fieldName = $arrColumn['Field'];
Basically in FromMysqlToPostgreSql.php
method populateTable
at line 868
should escape the field names, since there might be reserved words and than the select
in populateTableWorker
breaks.
If a table has a column with SET type, the script crushes.
Message: SQLSTATE[42501]:
Insufficient privilege: 7
ERRO: não pôde abrir arquivo
"/home/vmoura/Downloads/FromMySqlToPostgreSql-master/temporary_directory/rh_atributo_benefico_colaborador0.csv"
para leitura: Permissão negada
I have columns in mysql, which are not defined - whether they NULL or NOT NULL - just nothing
. And theese columns after conversion get NOT NULL constraint. Why? How could i skip this rule?
Example:
Create table Temp(id bigint, name varchar(500)); - name field becomes NOT NULL in postgresql
I would suggest adding php-mbstring extension to the list of dependencies since it is not included in the default PHP installation.
Hi,
As seen recently, all comments put on mysql columns are now moved to pgsql descriptions.
Thank you for this really convenient update :)
I see that mysql tables' comments aren't moved to pgsql tables descriptions.
Can you look to implement it please ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.