Comments (8)
你好,我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件,使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况
./python a.py ../project.ibd Traceback (most recent call last): File "a.py", line 28, in sdiinfo = sdi_page_data(bdata) File "a.py", line 22, in sdi_page_data sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())] IndexError: list index out of range
sdi:
[root@aaaaa ~]# more project.json ["ibd2sdi" , { "type": 1, "id": 1773, "object": { "mysqld_version_id": 80034, "dd_version": 80023, "sdi_version": 80019,我在我的8.0.34环境测试也是可以的... -_- 从报错来看, 第4页不是sdi page或者不完整. 请使用如下python3代码验证下呢
import sys with open(sys.argv[1],'rb') as f: for i in range(16): bdata = f.read(16384) page_type = int.from_bytes(bdata[24:26],'big') page_type = 'SDI_PAGE' if page_type == 17853 else page_type print(f'PAGE_NO: {i} \tPAGE_TYPE:{page_type}')
我上面遇到的报错就是因为第4页不是SDI PAGE。对于升级上来的Mysql 8的ibd文件SDI PAGE位置不固定。
对于5.7升级到8.0的ibd文件, sdi page确实不在第四页. 已修复(参考ibd2sdi.cc). 请下载最新版本测试下呢
from ibd2sql.
python3 main.py --ddl /mnt/d/temp/sys_menus_admin.ibd Traceback (most recent call last): File "/data/source/ibd2sql-main/main.py", line 100, in print('\n',innodb_sdi.sdi(filename).get_ddl(),'\n') File "/data/source/ibd2sql-main/innodb_sdi.py", line 43, in get_ddl dic_info = self.get_dic() File "/data/source/ibd2sql-main/innodb_sdi.py", line 132, in get_dic unzbdata = zlib.decompress(self.bdata[offset+33:offset+33+dzip_len]) zlib.error: Error -3 while decompressing data: incorrect header check
目前只支持 mysql 8.0 的数据文件, 请确认下数据库版本信息
from ibd2sql.
使用ibd2sdi解析ibd文件出来的结果:
["ibd2sdi"
,
{
"type": 1,
"id": 290,
"object":
{
"mysqld_version_id": 80021,
"dd_version": 80021,
"sdi_version": 80019,
"dd_object_type": "Table",
"dd_object": {
"name": "sys_menus_admin",
"mysql_version_id": 80021,
from ibd2sql.
使用ibd2sdi解析ibd文件出来的结果: ["ibd2sdi" , { "type": 1, "id": 290, "object": { "mysqld_version_id": 80021, "dd_version": 80021, "sdi_version": 80019, "dd_object_type": "Table", "dd_object": { "name": "sys_menus_admin", "mysql_version_id": 80021,
从ibd2sdi解析结果来看是mysql 8.0.21版本的Ibd文件, 我也解析了下8.0.21版本的ibd文件是正常的
12:03:42 [root@ddcw21 ibd2sql_v0.2]#python3 main.py --ddl /data/mysql_3352/mysqldata/db1/t1.ibd
CREATE Table IF NOT EXISTS `db1`.`t1`(
`id` int NOT NULL ,
`name` varchar(200) DEFAULT NULL ,
PRIMARY KEY (`id`)
)ENGINE=InnoDB ;
12:03:45 [root@ddcw21 ibd2sql_v0.2]#/soft/mysql_3352/mysqlbase/mysql/bin/ibd2sdi /data/mysql_3352/mysqldata/db1/t1.ibd | head -10
["ibd2sdi"
,
{
"type": 1,
"id": 352,
"object":
{
"mysqld_version_id": 80021,
"dd_version": 80021,
"sdi_version": 80019,
12:04:16 [root@ddcw21 ibd2sql_v0.2]#
那么这个报错的原因可能为:
- python版本问题. 本次测试的是python3.6
- sdi page 解析有问题
- page_size不为16384
select @@innodb_page_size;
可以使用如下python代码进行验证第二个可能(暴力解析第4页)
import sys,zlib,json
# 参考https://cloud.tencent.com/developer/article/2272297
def sdi_page_data(bdata):
_sdi_offset = []
isok = False
for i in range(120,16384):
if isok:
break
for j in range(i,16384):
if isok:
break
try:
_test = zlib.decompress(bdata[i:j])
_sdi_offset.append(i)
i += len(_test)
if len(_sdi_offset) == 2:
isok = True #break 2
break
except:
pass
sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())]
return sdi_info
with open(sys.argv[1],'rb') as f:
f.seek(3*16384,0)
bdata = f.read(16384)
sdiinfo = sdi_page_data(bdata)
print(sdiinfo)
(venv) 12:15:39 [root@ddcw21 ei]#python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd
[{'mysqld_version_id': 80021, 'dd_version': 80021, 'sdi_version': 80019, 'dd_object_type': 'Tablespace', 'dd_object': {'name': 'db1/t1', 'comment': '', 'options': 'encryption=N;', 'se_private_data': 'flags=16417;id=2;server_version=80021;space_version=1;state=normal;', 'engine': 'InnoDB', 'engine_attribute': '', 'files': [{'ordinal_position': 1, 'filename': './db1/t1.ibd', 'se_private_data': 'id=2;'}]}}, {'mysqld_version_id': 80021, 'dd_version': 80021, 'sdi_version': 80019, 'dd_object_type': 'Table', 'dd_object': {'name': 't1', 'mysql_version_id': 80021, 'created': 20231011040312, 'last_altered': 20231011040312, 'hidden': 1, 'options': 'avg_row_length=0;encrypt_type=N;key_block_size=0;keys_disabled=0;pack_record=1;stats_auto_recalc=0;stats_sample_pages=0;', 'columns': [{'name': 'id', 'type': 4, 'is_nullable': False, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 1, 'ordinal_position': 1, 'char_length': 11, 'numeric_precision': 10, 'numeric_scale': 0, 'numeric_scale_null': False, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': True, 'default_value_null': False, 'srs_id_null': True, 'srs_id': 0, 'default_value': 'AAAAAA==', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': 'interval_count=0;', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 2, 'column_type_utf8': 'int', 'elements': [], 'collation_id': 33, 'is_explicit_collation': False}, {'name': 'name', 'type': 16, 'is_nullable': True, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 1, 'ordinal_position': 2, 'char_length': 600, 'numeric_precision': 0, 'numeric_scale': 0, 'numeric_scale_null': True, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': False, 'default_value_null': True, 'srs_id_null': True, 'srs_id': 0, 'default_value': '', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': 'interval_count=0;', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 1, 'column_type_utf8': 'varchar(200)', 'elements': [], 'collation_id': 33, 'is_explicit_collation': False}, {'name': 'DB_TRX_ID', 'type': 10, 'is_nullable': False, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 2, 'ordinal_position': 3, 'char_length': 6, 'numeric_precision': 0, 'numeric_scale': 0, 'numeric_scale_null': True, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': False, 'default_value_null': True, 'srs_id_null': True, 'srs_id': 0, 'default_value': '', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': '', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 1, 'column_type_utf8': '', 'elements': [], 'collation_id': 63, 'is_explicit_collation': False}, {'name': 'DB_ROLL_PTR', 'type': 9, 'is_nullable': False, 'is_zerofill': False, 'is_unsigned': False, 'is_auto_increment': False, 'is_virtual': False, 'hidden': 2, 'ordinal_position': 4, 'char_length': 7, 'numeric_precision': 0, 'numeric_scale': 0, 'numeric_scale_null': True, 'datetime_precision': 0, 'datetime_precision_null': 1, 'has_no_default': False, 'default_value_null': True, 'srs_id_null': True, 'srs_id': 0, 'default_value': '', 'default_value_utf8_null': True, 'default_value_utf8': '', 'default_option': '', 'update_option': '', 'comment': '', 'generation_expression': '', 'generation_expression_utf8': '', 'options': '', 'se_private_data': 'table_id=1059;', 'engine_attribute': '', 'secondary_engine_attribute': '', 'column_key': 1, 'column_type_utf8': '', 'elements': [], 'collation_id': 63, 'is_explicit_collation': False}], 'schema_ref': 'db1', 'se_private_id': 1059, 'engine': 'InnoDB', 'last_checked_for_upgrade_version_id': 0, 'comment': '', 'se_private_data': '', 'engine_attribute': '', 'secondary_engine_attribute': '', 'row_format': 2, 'partition_type': 0, 'partition_expression': '', 'partition_expression_utf8': '', 'default_partitioning': 0, 'subpartition_type': 0, 'subpartition_expression': '', 'subpartition_expression_utf8': '', 'default_subpartitioning': 0, 'indexes': [{'name': 'PRIMARY', 'hidden': False, 'is_generated': False, 'ordinal_position': 1, 'comment': '', 'options': 'flags=0;', 'se_private_data': 'id=143;root=4;space_id=2;table_id=1059;trx_id=1568;', 'type': 1, 'algorithm': 2, 'is_algorithm_explicit': False, 'is_visible': True, 'engine': 'InnoDB', 'engine_attribute': '', 'secondary_engine_attribute': '', 'elements': [{'ordinal_position': 1, 'length': 4, 'order': 2, 'hidden': False, 'column_opx': 0}, {'ordinal_position': 2, 'length': 4294967295, 'order': 2, 'hidden': True, 'column_opx': 2}, {'ordinal_position': 3, 'length': 4294967295, 'order': 2, 'hidden': True, 'column_opx': 3}, {'ordinal_position': 4, 'length': 4294967295, 'order': 2, 'hidden': True, 'column_opx': 1}], 'tablespace_ref': 'db1/t1'}], 'foreign_keys': [], 'check_constraints': [], 'partitions': [], 'collation_id': 33}}]
from ibd2sql.
你好,我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件,使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况
./python a.py ../project.ibd
Traceback (most recent call last):
File "a.py", line 28, in
sdiinfo = sdi_page_data(bdata)
File "a.py", line 22, in sdi_page_data
sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())]
IndexError: list index out of range
sdi:
[root@aaaaa ~]# more project.json
["ibd2sdi"
,
{
"type": 1,
"id": 1773,
"object":
{
"mysqld_version_id": 80034,
"dd_version": 80023,
"sdi_version": 80019,
from ibd2sql.
你好,我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件,使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况
./python a.py ../project.ibd Traceback (most recent call last): File "a.py", line 28, in sdiinfo = sdi_page_data(bdata) File "a.py", line 22, in sdi_page_data sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())] IndexError: list index out of range
sdi:
[root@aaaaa ~]# more project.json ["ibd2sdi" , { "type": 1, "id": 1773, "object": { "mysqld_version_id": 80034, "dd_version": 80023, "sdi_version": 80019,
我在我的8.0.34环境测试也是可以的... -_-
从报错来看, 第4页不是sdi page或者不完整. 请使用如下python3代码验证下呢
import sys
with open(sys.argv[1],'rb') as f:
for i in range(16):
bdata = f.read(16384)
page_type = int.from_bytes(bdata[24:26],'big')
page_type = 'SDI_PAGE' if page_type == 17853 else page_type
print(f'PAGE_NO: {i} \tPAGE_TYPE:{page_type}')
from ibd2sql.
你好,我在8.0.34上也出现了这个问题。ibd2sdi可以解析文件,使用 python t20231011.py /data/mysql_3352/mysqldata/db1/t1.ibd 出现异常. 帮忙看看啥情况
./python a.py ../project.ibd Traceback (most recent call last): File "a.py", line 28, in sdiinfo = sdi_page_data(bdata) File "a.py", line 22, in sdi_page_data sdi_info = [json.loads(zlib.decompress(bdata[_sdi_offset[0]:]).decode()), json.loads(zlib.decompress(bdata[_sdi_offset[1]:]).decode())] IndexError: list index out of range
sdi:
[root@aaaaa ~]# more project.json ["ibd2sdi" , { "type": 1, "id": 1773, "object": { "mysqld_version_id": 80034, "dd_version": 80023, "sdi_version": 80019,我在我的8.0.34环境测试也是可以的... -_- 从报错来看, 第4页不是sdi page或者不完整. 请使用如下python3代码验证下呢
import sys with open(sys.argv[1],'rb') as f: for i in range(16): bdata = f.read(16384) page_type = int.from_bytes(bdata[24:26],'big') page_type = 'SDI_PAGE' if page_type == 17853 else page_type print(f'PAGE_NO: {i} \tPAGE_TYPE:{page_type}')
我上面遇到的报错就是因为第4页不是SDI PAGE。对于升级上来的Mysql 8的ibd文件SDI PAGE位置不固定。
from ibd2sql.
可以了,感谢大佬!
from ibd2sql.
Related Issues (20)
- --sql 参数解析idb数据 varchar、text、int 、bigint类型解析出的数据有问题! HOT 15
- 能否可以支持自己指定表ddl创建语句 HOT 1
- 感激,分分钟解决问题 HOT 2
- 老铁可以支持导出功能吗,导出来txt或者.sql格式的文件呢,谢谢 HOT 1
- 老铁,不是我说你,遮遮掩掩的。 HOT 2
- 解析5.7 ibd文件失败,没有任何输出 HOT 4
- 执行命令的时候报错 row_format = compressed or its damaged or its mysql 5.7 file HOT 3
- --ddl 生成错的 `!hidden!_dropped_v1_p13_status` NULL, 这怎么解决 HOT 4
- 恢复数据时报错RecursionError: maximum recursion depth exceeded,网上找了解决方案后执行不报错,但数据有缺失 HOT 5
- 你好,没有看到任何数据,在8.0里面生成的ibd 库名也要对应吗,我是从5.7导出create语句,在8.0执行的 HOT 7
- first_blob 方法 解包异常 HOT 12
- 你好,mysql8.0 数据恢复有问题 HOT 4
- 说几个使用体验吧 HOT 1
- 转换的时候乱码了 HOT 4
- noob_loop HOT 1
- 一坨大便
- zlib.error: Error -3 while decompressing data: unknown compression method HOT 1
- 对于 mysqlfrm 解析 mysql 5.* 不准确的问题 可以使用 dbsake 工具进行解析 HOT 1
- 使用 5.6.16提取 sql 时 死循环了 HOT 5
- 是否支持undrop for innodb 磁盘扫描出来的ibd碎片解析? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ibd2sql.