Comments (18)
salsferrazza
commented on June 2, 2024
Real crux of issue appears to be varying group definitions (of the same group type, e.g. NoLegs) across different messages. So NoLegs as part of InstrumentLegGrp may have a different # of supported fields compared to NoLegs in QuotReqLegsGrp. fix2json is not grouping groups by message type, so the internal data dictionary representation entry for any particular group name key always reflects the last instance of NoLegs encountered in the dictionary file.
There is some evidence in support of this being a classic rite of passage on the FIX parsing circuit.
from fix2json.
jlroo
commented on June 2, 2024
I am having a similar issue it seems like the when the message entry type changes from "TRADE" to "BID/OFFER" before the last entry on the array the last sub-message is left out of the array.
{
"_id" : ObjectId("56a00655f70c63fa0d857ad5"),
"5797" : "2",
"5799" : "1",
"ApplVerID" : "FIX50SP2",
"BodyLength" : 370,
"MsgType" : "MARKETDATAINCREMENTALREFRESH",
"SenderCompID" : "CME",
"MsgSeqNum" : 939774,
"SendingTime" : "20130715145146403",
"TradeDate" : "20130715",
"NoMDEntries" : 3,
"MDEntries" : [
{
"MDUpdateAction" : "CHANGE",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1288121,
"SecurityDesc" : "ESU3",
"MDEntryType" : "OFFER",
"MDEntryPx" : 167525,
"MDEntrySize" : 1095,
"MDEntryTime" : "145146000",
"TradingSessionID" : "HALFDAY",
"NumberOfOrders" : 208,
"MDPriceLevel" : 1
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1288122,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167500,
"MDEntrySize" : 1,
"MDEntryTime" : "145146000",
"NetChgPrevDay" : 475,
"TradeVolume" : 417394
}
],
"MDUpdateAction" : "CHANGE",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1288123,
"SecurityDesc" : "ESU3",
"MDEntryType" : "BID",
"MDEntryPx" : 167500,
"MDEntrySize" : 46,
"MDEntryTime" : "145146000",
"TradingSessionID" : "HALFDAY",
"NumberOfOrders" : 4,
"MDPriceLevel" : 1,
"CheckSum" : "100"
}
Other Message
.
.
.
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1284133,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167500,
"MDEntrySize" : 1,
"MDEntryTime" : "145110000",
"NetChgPrevDay" : 475,
"TradeVolume" : 416347
}
],
"MDUpdateAction" : "CHANGE",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1284134,
"SecurityDesc" : "ESU3",
"MDEntryType" : "OFFER",
"MDEntryPx" : 167500,
"MDEntrySize" : 74,
"MDEntryTime" : "145110000",
"TradingSessionID" : "HALFDAY",
"NumberOfOrders" : 7,
"MDPriceLevel" : 1,
"CheckSum" : "132"
}
from fix2json.
salsferrazza
commented on June 2, 2024
Yes, indeed this looks to be stemming from the same fundamental issue. There have been a few steps taken toward fixing this but unfortunately there's still a little bit more work to be done. Stay tuned and we'll update the status on this issue as soon as a branch is created with a candidate fix.
from fix2json.
salsferrazza
commented on June 2, 2024
I checked some code into the grouptreatment branch that appears to fix this issue for MarketDataIncrementalRefresh but still seems to mis-process SecurityDefinitions. You might want to give it a shot for your purposes if limited to MarketDataIncrementalRefresh's immediately.
git checkout grouptreatment && git rebase origin grouptreatment
Turns out that much of the issue stemmed from additional required fields in the data dictionary for CME. Not sure if there is a canonical dictionary in QuickFix format that CME provides to members as part of its SDK, but if there is it might forestall future complications of this kind. I recall seeing some extension packs somewhere on FIX Protocol's official site but have not had a chance to delve into those quite yet.
After checking out the branch, you should see output resembling:
$ head -5 /tmp/MDFF_CBT_20130714-20130715_7813_0 | ./fix2json.js -p dict/FIX50SP2.CME.xml
{
"ApplVerID": "FIX50SP2",
"BodyLength": 156,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 431875,
"SendingTime": "20130715053428544",
"TradeDate": "20130715",
"NoMDEntries": 1,
"MDEntries": [
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "19590",
"RptSeq": 1491,
"SecurityDesc": "ZCN4-ZCU4",
"MDEntryType": "BID",
"MDEntryPx": 0.75,
"MDEntrySize": 7,
"MDEntryTime": "53428000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 3,
"MDPriceLevel": 3
}
],
"CheckSum": "233"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 104,
"MsgType": "SECURITYSTATUS",
"SenderCompID": "CME",
"MsgSeqNum": 6613,
"SendingTime": "20130714180206034",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "164576",
"TradeDate": "20130715",
"SecurityDesc": "ZC:BF H4-K4-N4",
"HighPx": 159,
"LowPx": -161,
"CheckSum": "138"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 1634,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 3177179,
"SendingTime": "20130715181451830",
"TradeDate": "20130715",
"NoMDEntries": 16,
"MDEntries": [
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "1647",
"RptSeq": 180151,
"SecurityDesc": "ZC:BF U3-Z3-H4",
"MDEntryType": "OFFER",
"MDEntryPx": 45,
"MDEntrySize": 195,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "21907",
"RptSeq": 118087,
"SecurityDesc": "ZCU3-ZCH4",
"MDEntryType": "OFFER",
"MDEntryPx": 21,
"MDEntrySize": 3,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 1
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "21907",
"RptSeq": 118088,
"SecurityDesc": "ZCU3-ZCH4",
"MDEntryType": "OFFER",
"MDEntryPx": 21.25,
"MDEntrySize": 10,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "504398",
"RptSeq": 29594,
"SecurityDesc": "ZCU3-ZCH5",
"MDEntryType": "OFFER",
"MDEntryPx": 11.25,
"MDEntrySize": 3,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 1
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "504398",
"RptSeq": 29595,
"SecurityDesc": "ZCU3-ZCH5",
"MDEntryType": "OFFER",
"MDEntryPx": 11.5,
"MDEntrySize": 2,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "21970",
"RptSeq": 50008,
"SecurityDesc": "ZCU3-ZCK4",
"MDEntryType": "OFFER",
"MDEntryPx": 13.5,
"MDEntrySize": 3,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 1
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "21970",
"RptSeq": 50009,
"SecurityDesc": "ZCU3-ZCK4",
"MDEntryType": "OFFER",
"MDEntryPx": 13.75,
"MDEntrySize": 8,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "558644",
"RptSeq": 48554,
"SecurityDesc": "ZCU3-ZCN4",
"MDEntryType": "OFFER",
"MDEntryPx": 7,
"MDEntrySize": 3,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 1
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "558644",
"RptSeq": 48555,
"SecurityDesc": "ZCU3-ZCN4",
"MDEntryType": "OFFER",
"MDEntryPx": 7.25,
"MDEntrySize": 8,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "22098",
"RptSeq": 52504,
"SecurityDesc": "ZCU3-ZCU4",
"MDEntryType": "OFFER",
"MDEntryPx": 15.25,
"MDEntrySize": 3,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 1
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "22098",
"RptSeq": 52505,
"SecurityDesc": "ZCU3-ZCU4",
"MDEntryType": "OFFER",
"MDEntryPx": 15.5,
"MDEntrySize": 1,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "504409",
"RptSeq": 16282,
"SecurityDesc": "ZCU3-ZCU5",
"MDEntryType": "OFFER",
"MDEntryPx": 25.25,
"MDEntrySize": 1,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "588275",
"RptSeq": 265779,
"SecurityDesc": "ZCU3-ZCZ3",
"MDEntryType": "OFFER",
"MDEntryPx": 33,
"MDEntrySize": 3,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 1
},
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "588275",
"RptSeq": 265780,
"SecurityDesc": "ZCU3-ZCZ3",
"MDEntryType": "OFFER",
"MDEntryPx": 33.25,
"MDEntrySize": 20,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "519645",
"RptSeq": 32899,
"SecurityDesc": "ZCU3-ZCZ4",
"MDEntryType": "OFFER",
"MDEntryPx": 9.75,
"MDEntrySize": 1,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "19282",
"RptSeq": 20689,
"SecurityDesc": "ZCU3-ZCZ5",
"MDEntryType": "OFFER",
"MDEntryPx": 29.75,
"MDEntrySize": 1,
"MDEntryTime": "181451000",
"QuoteCondition": "IMPLIED PRICE",
"TradingSessionID": "HALFDAY",
"MDPriceLevel": 2
}
],
"CheckSum": "007"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 164,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 3191367,
"SendingTime": "20130715181519884",
"TradeDate": "20130715",
"NoMDEntries": 1,
"MDEntries": [
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "19312",
"RptSeq": 52036,
"SecurityDesc": "ZCZ3-ZCK4",
"MDEntryType": "OFFER",
"MDEntryPx": -19.5,
"MDEntrySize": 1103,
"MDEntryTime": "181519000",
"TradingSessionID": "0",
"NumberOfOrders": 48,
"MDPriceLevel": 1
}
],
"CheckSum": "117"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 164,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 2050622,
"SendingTime": "20130715151758397",
"TradeDate": "20130715",
"NoMDEntries": 1,
"MDEntries": [
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "164114",
"RptSeq": 145,
"SecurityDesc": "ZC:CF K4N4U4Z4",
"MDEntryType": "BID",
"MDEntryPx": -9.75,
"MDEntrySize": 1,
"MDEntryTime": "151758000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 1,
"MDPriceLevel": 1
}
],
"CheckSum": "146"
}
Please let us know how the new code works out for you.
Thanks!
Sal
from fix2json.
jlroo
commented on June 2, 2024
Thank you Sal, I will try to follow up with the CME data people about the canonical dictionary for QuickFix format. Now I cloned and checkout the git repository and I am getting the following error.
/home/jlroo/cme/fix2json/fix2json.js:91
var type = TAGS[num].type;
^
TypeError: Cannot read property 'type' of undefined
at pluckGroup (/home/jlroo/cme/fix2json/fix2json.js:91:22)
at resolveFields (/home/jlroo/cme/fix2json/fix2json.js:155:28)
at processLine (/home/jlroo/cme/fix2json/fix2json.js:167:21)
at Interface.<anonymous> (/home/jlroo/cme/fix2json/fix2json.js:60:31)
at emitOne (events.js:90:13)
at Interface.emit (events.js:182:7)
at Interface._onLine (readline.js:211:10)
at Interface.<anonymous> (readline.js:341:12)
at Array.forEach (native)
at Interface._normalWrite (readline.js:340:11)
Any suggestions on how to fix it?
from fix2json.
salsferrazza
commented on June 2, 2024
Hi jlroo,
Were you using the grouptreatment branch when you got that error? Also was npm install run after the re-clone?
If you could share the full command that you were executing when you encountered that error, that'd be helpful as well.
Thanks!
Sal
from fix2json.
jlroo
commented on June 2, 2024
Sal, I did run install and switched branches. Below are all the commands that I used. Thank you!
git clone https://github.com/SunGard-Labs/fix2json.git
Initialized empty Git repository in ~/cme/fix2json/.git/
remote: Counting objects: 347, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 347 (delta 0), reused 0 (delta 0), pack-reused 344
Receiving objects: 100% (347/347), 307.95 KiB, done.
Resolving deltas: 100% (208/208), done.
$ cd fix2json/
[jlroo@fsba fix2json]$ git checkout grouptreatment && git rebase origin grouptreatment
Branch grouptreatment set up to track remote branch grouptreatment from origin.
Switched to a new branch 'grouptreatment'
Already on 'grouptreatment'
Current branch grouptreatment is up to date.
$ npm install
[email protected] ~/cme/fix2json
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
└─┬ [email protected]
├─┬ [email protected]
│ └── [email protected]
└─┬ [email protected]
├─┬ [email protected]
│ └── [email protected]
├── [email protected]
├─┬ [email protected]
│ └─┬ [email protected]
│ ├── [email protected]
│ └── [email protected]
└── [email protected]
$ ./fix2json.js
Usage: fix2json [-p] <data dictionary xml file> [path to FIX message file]
fix2json will use standard input in the absence of a message file.
$ ./fix2json.js dict/FIX50SP2.CME.xml ~/cme/futures/fixdata/MDFF_CME_20130714-20130715_7817_0 > ~/cme/futures/jsdata/futures.json
~/cme/fix2json/fix2json.js:91
var type = TAGS[num].type;
^
TypeError: Cannot read property 'type' of undefined
at pluckGroup (~/cme/fix2json/fix2json.js:91:22)
at resolveFields (~/cme/fix2json/fix2json.js:155:28)
at processLine (~/cme/fix2json/fix2json.js:167:21)
at Interface.<anonymous> (~/cme/fix2json/fix2json.js:60:31)
at emitOne (events.js:90:13)
at Interface.emit (events.js:182:7)
at Interface._onLine (readline.js:211:10)
at Interface.<anonymous> (readline.js:341:12)
at Array.forEach (native)
at Interface._normalWrite (readline.js:340:11)
from fix2json.
salsferrazza
commented on June 2, 2024
Thanks for sending that over. I executed the identical commands and received different results:
$ git clone https://github.com/SunGard-Labs/fix2json.git
Cloning into 'fix2json'...
remote: Counting objects: 347, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 347 (delta 0), reused 0 (delta 0), pack-reused 344
Receiving objects: 100% (347/347), 307.95 KiB | 0 bytes/s, done.
Resolving deltas: 100% (208/208), done.
Checking connectivity... done.
$ cd fix2json
$ git checkout grouptreatment && git rebase origin grouptreatment
Branch grouptreatment set up to track remote branch grouptreatment from origin.
Switched to a new branch 'grouptreatment'
Already on 'grouptreatment'
Your branch is up-to-date with 'origin/grouptreatment'.
Current branch grouptreatment is up to date.
SGCSNYMACSSFER:fix2json salvatore.sferrazza$ npm install
[email protected] node_modules/string_decoder
[email protected] node_modules/node-xml
[email protected] node_modules/underscore
[email protected] node_modules/xmldom
[email protected] node_modules/xpath
[email protected] node_modules/yamljs
├── [email protected] ([email protected], [email protected], [email protected], [email protected])
└── [email protected] ([email protected])
$ ./fix2json.js dict/FIX50SP2.CME.xml ../C*il/MDFF_CME_20130714-20130715_7817_0|more
{"ApplVerID":"FIX50SP2","BodyLength":136,"MsgType":"MARKETDATAINCREMENTALREFRESH","SenderCompID":"CME","MsgSeqNum":578,"SendingTime":"20130714180133577","TradeDate":"20130715","NoMDEntries":1,"MDEntries":[{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"111473","SettlDate":"20130712","RptSeq":1,"SecurityDesc":"ESU4","MDEntryType":"SETTLEMENT PRICE","MDEntryPx":164350,"MDEntryTime":"180133000"}],"CheckSum":"248"}
{"ApplVerID":"FIX50SP2","BodyLength":1417,"MsgType":"MARKETDATAINCREMENTALREFRESH","SenderCompID":"CME","MsgSeqNum":731,"SendingTime":"20130714190005961","TradeDate":"20130710","NoMDEntries":15,"MDEntries":[{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":2,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":166000,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":1},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":3,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":165675,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":2},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":4,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":165500,"MDEntrySize":2,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":3},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":5,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":165375,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":4},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":6,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":165025,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":5},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":7,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":164425,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":6},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":8,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":164075,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":7},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":9,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":163000,"MDEntrySize":2,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":8},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":10,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":162675,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":9},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":11,"SecurityDesc":"ESZ3","MDEntryType":"BID","MDEntryPx":162400,"MDEntrySize":3,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":10},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":12,"SecurityDesc":"ESZ3","MDEntryType":"OFFER","MDEntryPx":167000,"MDEntrySize":2,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":1},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":13,"SecurityDesc":"ESZ3","MDEntryType":"OFFER","MDEntryPx":167500,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":2},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":14,"SecurityDesc":"ESZ3","MDEntryType":"OFFER","MDEntryPx":167925,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":3},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":15,"SecurityDesc":"ESZ3","MDEntryType":"OFFER","MDEntryPx":170000,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":4},{"MDUpdateAction":"NEW","SecurityIDSource":"EXCHANGE SYMBOL","SecurityID":"28112","RptSeq":16,"SecurityDesc":"ESZ3","MDEntryType":"OFFER","MDEntryPx":172500,"MDEntrySize":1,"MDEntryTime":"190005000","TradingSessionID":"0","NumberOfOrders":1,"MDPriceLevel":5}],"CheckSum":"029"}
...
$ ./fix2json.js -p dict/FIX50SP2.CME.xml ../MDFF_CME_20130714-20130715_7817_0
{
"ApplVerID": "FIX50SP2",
"BodyLength": 136,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 578,
"SendingTime": "20130714180133577",
"TradeDate": "20130715",
"NoMDEntries": 1,
"MDEntries": [
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "111473",
"SettlDate": "20130712",
"RptSeq": 1,
"SecurityDesc": "ESU4",
"MDEntryType": "SETTLEMENT PRICE",
"MDEntryPx": 164350,
"MDEntryTime": "180133000"
}
],
"CheckSum": "248"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 1417,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 731,
"SendingTime": "20130714190005961",
"TradeDate": "20130710",
"NoMDEntries": 15,
"MDEntries": [
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "28112",
"RptSeq": 2,
"SecurityDesc": "ESZ3",
"MDEntryType": "BID",
"MDEntryPx": 166000,
"MDEntrySize": 1,
"MDEntryTime": "190005000",
"TradingSessionID": "0",
"NumberOfOrders": 1,
"MDPriceLevel": 1
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "28112",
"RptSeq": 3,
"SecurityDesc": "ESZ3",
"MDEntryType": "BID",
"MDEntryPx": 165675,
"MDEntrySize": 1,
"MDEntryTime": "190005000",
"TradingSessionID": "0",
"NumberOfOrders": 1,
"MDPriceLevel": 2
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "28112",
"RptSeq": 4,
"SecurityDesc": "ESZ3",
"MDEntryType": "BID",
"MDEntryPx": 165500,
"MDEntrySize": 2,
"MDEntryTime": "190005000",
"TradingSessionID": "0",
"NumberOfOrders": 1,
"MDPriceLevel": 3
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "28112",
"RptSeq": 5,
"SecurityDesc": "ESZ3",
"MDEntryType": "BID",
"MDEntryPx": 165375,
"MDEntrySize": 1,
"MDEntryTime": "190005000",
"TradingSessionID": "0",
"NumberOfOrders": 1,
"MDPriceLevel": 4
},
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "28112",
"RptSeq": 6,
"SecurityDesc": "ESZ3",
"MDEntryType": "BID",
"MDEntryPx": 165025,
"MDEntrySize": 1,
"MDEntryTime": "190005000",
"TradingSessionID": "0",
"NumberOfOrders": 1,
"MDPriceLevel": 5
},
...
Not entirely sure what might be contributing to this. I tested this with:
$ node -v
v0.12.5
Would you be able to check your node version and/or perhaps try executing on a different machine to see if this might be related to an environment discrepancy issue?
Thanks!
Sal
from fix2json.
jlroo
commented on June 2, 2024
Sal thank you for all the support. Its definitely a problem with the node version. It works fine with node v4.2.6 on CentOS 6.5
Yum Nodejs - CentOS 6.5
$ npm -v
3.3.12
$ node -v
v5.5.0
Nodejs from tar - CentOS 6.5
$ ~/node/bin/npm -v
2.14.12
$ ~/node/bin/node -v
v4.2.6
from fix2json.
salsferrazza
commented on June 2, 2024
Hi @jlroo,
I'm actually not so sure now whether it is the node version or not. Last night I ran a whole Datamine sample file through and before it was complete, it failed with the following error:
fix2json.js:91
var type = TAGS[num].type;
^
TypeError: Cannot read property 'type' of undefined
at pluckGroup (/Users/salvatore.sferrazza/Desktop/fix2json/fix2json.js:91:22)
at resolveFields (/Users/salvatore.sferrazza/Desktop/fix2json/fix2json.js:155:28)
at processLine (/Users/salvatore.sferrazza/Desktop/fix2json/fix2json.js:167:21)
at Interface.<anonymous> (/Users/salvatore.sferrazza/Desktop/fix2json/fix2json.js:60:31)
at Interface.emit (events.js:107:17)
at Interface._onLine (readline.js:214:10)
at Interface.<anonymous> (readline.js:344:12)
at Array.forEach (native)
at Interface._normalWrite (readline.js:343:11)
at ReadStream.ondata (readline.js:91:10)
That's just too much of a coincidence. Will isolate the failing record, but I suspect there's a field in the patched data dictionary still missing that the group parser is failing on. Does it stop for you right away or does it successfully process a number of records before the exception?
from fix2json.
jlroo
commented on June 2, 2024
Sal, I experienced the same issue in two different machines and versions of node. Initially I thought that the problem was just on CentOS 6.5 but I get the same error on my mac. Here are the three fix messages right after "MsgSeqNum": 1550140 the last processed message before getting the error.
1128=9^A9=1839^A35=X^A49=CME^A34=1126934^A52=20130715160235126^A75=20130715^A268=17^A279=0^A22=8^A48=17704^A83=1550137^A107=ESU3^A269=2^A270=167700^A271=3^A273=160235000^A451=675^A1020=535856^A5797=1^A279=0^A22=8^A48=17704^A83=1550138^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535857^A5797=1^A279=0^A22=8^A48=17704^A83=1550139^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535858^A5797=1^A279=0^A22=8^A48=17704^A83=1550140^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535859^A5797=1^A279=0^A22=8^A48=17704^A83=1550141^A107=ESU3^A269=2^A270=167700^A271=10^A273=160235000^A451=675^A1020=535869^A5797=1^A279=0^A22=8^A48=17704^A83=1550142^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535870^A5797=1^A279=0^A22=8^A48=17704^A83=1550143^A107=ESU3^A269=2^A270=167700^A271=3^A273=160235000^A451=675^A1020=535873^A5797=1^A279=0^A22=8^A48=17704^A83=1550144^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535874^A5797=1^A279=0^A22=8^A48=17704^A83=1550145^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535875^A5797=1^A279=0^A22=8^A48=17704^A83=1550146^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535876^A5797=1^A279=0^A22=8^A48=17704^A83=1550147^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535877^A5797=1^A279=0^A22=8^A48=17704^A83=1550148^A107=ESU3^A269=2^A270=167700^A271=54^A273=160235000^A451=675^A1020=535931^A5797=1^A279=0^A22=8^A48=17704^A83=1550149^A107=ESU3^A269=2^A270=167700^A271=15^A273=160235000^A451=675^A1020=535946^A5797=1^A279=1^A22=8^A48=17704^A83=1550150^A107=ESU3^A269=1^A270=167700^A271=375^A273=160235000^A336=2^A346=42^A1023=1^A279=0^A22=8^A48=17704^A83=1550151^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=535947^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550152^A107=ESU3^A269=1^A270=167700^A271=374^A273=160235000^A336=2^A346=42^A1023=1^A279=0^A22=8^A48=17704^A83=1550153^A107=ESU3^A269=2^A270=167700^A271=16^A273=160235000^A451=675^A1020=535963^A5797=1^A5799=1^A10=099^A
1128=9^A9=155^A35=X^A49=CME^A34=78311^A52=20130715070236593^A75=20130715^A268=1^A279=1^A22=8^A48=17704^A83=93283^A107=ESU3^A269=0^A270=167325^A271=80^A273=70236000^A336=2^A346=29^A1023=1^A10=100^A
1128=9^A9=1214^A35=X^A49=CME^A34=1126935^A52=20130715160235126^A75=20130715^A268=11^A279=1^A22=8^A48=17704^A83=1550154^A107=ESU3^A269=1^A270=167700^A271=358^A273=160235000^A336=2^A346=42^A1023=1^A279=0^A22=8^A48=17704^A83=1550155^A107=ESU3^A269=2^A270=167700^A271=5^A273=160235000^A451=675^A1020=535968^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550156^A107=ESU3^A269=1^A270=167700^A271=353^A273=160235000^A336=2^A346=42^A1023=1^A279=0^A22=8^A48=17704^A83=1550157^A107=ESU3^A269=2^A270=167700^A271=20^A273=160235000^A451=675^A1020=535988^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550158^A107=ESU3^A269=1^A270=167700^A271=333^A273=160235000^A336=2^A346=41^A1023=1^A279=0^A22=8^A48=17704^A83=1550159^A107=ESU3^A269=2^A270=167700^A271=40^A273=160235000^A451=675^A1020=536028^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550160^A107=ESU3^A269=1^A270=167700^A271=293^A273=160235000^A336=2^A346=41^A1023=1^A279=0^A22=8^A48=17704^A83=1550161^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=536029^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550162^A107=ESU3^A269=1^A270=167700^A271=292^A273=160235000^A336=2^A346=41^A1023=1^A279=0^A22=8^A48=17704^A83=1550163^A107=ESU3^A269=2^A270=167700^A271=1^A273=160235000^A451=675^A1020=536030^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550164^A107=ESU3^A269=1^A270=167700^A271=291^A273=160235000^A336=2^A346=41^A1023=1^A10=013^A
1128=9^A9=273^A35=X^A49=CME^A34=1126936^A52=20130715160235126^A75=20130715^A268=2^A279=0^A22=8^A48=17704^A83=1550165^A107=ESU3^A269=2^A270=167700^A271=15^A273=160235000^A451=675^A1020=536045^A5797=1^A5799=1^A279=1^A22=8^A48=17704^A83=1550166^A107=ESU3^A269=1^A270=167700^A271=276^A273=160235000^A336=2^A346=41^A1023=1^A10=095^A
from fix2json.
salsferrazza
commented on June 2, 2024
Hi @jlroo,
You can try grabbing the grouptreatment branch again. I took those FIX messages above and put them into testfiles/issue#4.txt, they now seem to be parsing out correctly.
localhost:fix2json s$ git fetch && git rebase origin grouptreatment
Already on 'grouptreatment'
Current branch grouptreatment is up to date.
localhost:fix2json s$ cat testfiles/issue#4.txt | ./fix2json.js -p dict/FIX50SP2.CME.xml
{
"ApplVerID": "FIX50SP2",
"BodyLength": 1839,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 1126934,
"SendingTime": "20130715160235126",
"TradeDate": "20130715",
"NoMDEntries": 17,
"MDEntries": [
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550137,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 3,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535856,
"AggressorSide": 1
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550138,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535857,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550139,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535858,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550140,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535859,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550141,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 10,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535869,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550142,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535870,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550143,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 3,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535873,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550144,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535874,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550145,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535875,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550146,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535876,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550147,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535877,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550148,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 54,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535931,
"AggressorSide": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550149,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 15,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535946,
"AggressorSide": 1,
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550150,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 375,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 42,
"MDPriceLevel": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550151,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535947,
"AggressorSide": 1,
"MatchEventIndicator": "1",
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550152,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 374,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 42,
"MDPriceLevel": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550153,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 16,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535963,
"AggressorSide": 1,
"MatchEventIndicator": "1"
}
],
"CheckSum": "099"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 155,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 78311,
"SendingTime": "20130715070236593",
"TradeDate": "20130715",
"NoMDEntries": 1,
"MDEntries": [
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 93283,
"SecurityDesc": "ESU3",
"MDEntryType": "BID",
"MDEntryPx": 167325,
"MDEntrySize": 80,
"MDEntryTime": "70236000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 29,
"MDPriceLevel": 1
}
],
"CheckSum": "100"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 1214,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 1126935,
"SendingTime": "20130715160235126",
"TradeDate": "20130715",
"NoMDEntries": 11,
"MDEntries": [
{
"MDUpdateAction": "NEW",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550154,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 358,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 42,
"MDPriceLevel": 1
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550155,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 5,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535968,
"AggressorSide": 1,
"MatchEventIndicator": "1",
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550156,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 353,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 42,
"MDPriceLevel": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550157,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 20,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 535988,
"AggressorSide": 1,
"MatchEventIndicator": "1",
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550158,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 333,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 41,
"MDPriceLevel": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550159,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 40,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 536028,
"AggressorSide": 1,
"MatchEventIndicator": "1",
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550160,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 293,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 41,
"MDPriceLevel": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550161,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 536029,
"AggressorSide": 1,
"MatchEventIndicator": "1",
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550162,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 292,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 41,
"MDPriceLevel": 1,
"MDUpdateAction": "NEW"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550163,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 1,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 536030,
"AggressorSide": 1,
"MatchEventIndicator": "1",
"MDUpdateAction": "CHANGE"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550164,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 291,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 41,
"MDPriceLevel": 1
}
],
"CheckSum": "013"
}
{
"ApplVerID": "FIX50SP2",
"BodyLength": 273,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"MsgSeqNum": 1126936,
"SendingTime": "20130715160235126",
"TradeDate": "20130715",
"NoMDEntries": 2,
"MDEntries": [
{
"MDUpdateAction": "CHANGE",
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550165,
"SecurityDesc": "ESU3",
"MDEntryType": "TRADE",
"MDEntryPx": 167700,
"MDEntrySize": 15,
"MDEntryTime": "160235000",
"NetChgPrevDay": 675,
"TradeVolume": 536045,
"AggressorSide": 1,
"MatchEventIndicator": "1"
},
{
"SecurityIDSource": "EXCHANGE SYMBOL",
"SecurityID": "17704",
"RptSeq": 1550166,
"SecurityDesc": "ESU3",
"MDEntryType": "OFFER",
"MDEntryPx": 167700,
"MDEntrySize": 276,
"MDEntryTime": "160235000",
"TradingSessionID": "HALFDAY",
"NumberOfOrders": 41,
"MDPriceLevel": 1
}
],
"CheckSum": "095"
}
Are these the results you are expecting from that input?
from fix2json.
jlroo
commented on June 2, 2024
Sal those messages are correct. I searched for other messages with errors in the data previous the group treatment and I found a duplicated message (same "MsgSeqNum") one with the correct groups, the other with the problem. I did the same for the new version and this issues seems to be corrected Here are the messages:
Old Version
{
"_id" : ObjectId("56ac2ac0f70c63fa0dfdea0f"),
"5797" : "2",
"5799" : "1",
"ApplVerID" : "FIX50SP2",
"BodyLength" : 583,
"MsgType" : "MARKETDATAINCREMENTALREFRESH",
"SenderCompID" : "CME",
"MsgSeqNum" : 934521,
"SendingTime" : "20130715145036109",
"TradeDate" : "20130715",
"NoMDEntries" : 5,
"MDEntries" : [
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281111,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414934
}
],
"MDUpdateAction" : "CHANGE",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281115,
"SecurityDesc" : "ESU3",
"MDEntryType" : "BID",
"MDEntryPx" : 167525,
"MDEntrySize" : 196,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414938,
"TradingSessionID" : "HALFDAY",
"NumberOfOrders" : 35,
"MDPriceLevel" : 1,
"CheckSum" : "248"
}
{
"_id" : ObjectId("56b2df276a92f0f1ff8356c0"),
"ApplVerID" : "FIX50SP2",
"BodyLength" : 583,
"MsgType" : "MARKETDATAINCREMENTALREFRESH",
"SenderCompID" : "CME",
"MsgSeqNum" : 934521,
"SendingTime" : "20130715145036109",
"TradeDate" : "20130715",
"NoMDEntries" : 5,
"MDEntries" : [
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281111,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414934,
"AggressorSide" : 2,
"MatchEventIndicator" : "1"
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281112,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414935,
"AggressorSide" : 2
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281113,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414936,
"AggressorSide" : 2
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281114,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 2,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414938,
"AggressorSide" : 2
},
{
"MDUpdateAction" : "CHANGE",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281115,
"SecurityDesc" : "ESU3",
"MDEntryType" : "BID",
"MDEntryPx" : 167525,
"MDEntrySize" : 196,
"MDEntryTime" : "145036000",
"TradingSessionID" : "HALFDAY",
"NumberOfOrders" : 35,
"MDPriceLevel" : 1
}
],
"CheckSum" : "248"
}
New Version
{
"_id" : ObjectId("56b9fdd56f227b70f2dcd776"),
"ApplVerID" : "FIX50SP2",
"BodyLength" : 583,
"MsgType" : "MARKETDATAINCREMENTALREFRESH",
"SenderCompID" : "CME",
"MsgSeqNum" : 934521,
"SendingTime" : "20130715145036109",
"TradeDate" : "20130715",
"NoMDEntries" : 5,
"MDEntries" : [
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281111,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414934,
"AggressorSide" : 2,
"MatchEventIndicator" : "1"
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281112,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414935,
"AggressorSide" : 2
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281113,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 1,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414936,
"AggressorSide" : 2
},
{
"MDUpdateAction" : "NEW",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281114,
"SecurityDesc" : "ESU3",
"MDEntryType" : "TRADE",
"MDEntryPx" : 167525,
"MDEntrySize" : 2,
"MDEntryTime" : "145036000",
"NetChgPrevDay" : 500,
"TradeVolume" : 414938,
"AggressorSide" : 2
},
{
"MDUpdateAction" : "CHANGE",
"SecurityIDSource" : "EXCHANGE SYMBOL",
"SecurityID" : "17704",
"RptSeq" : 1281115,
"SecurityDesc" : "ESU3",
"MDEntryType" : "BID",
"MDEntryPx" : 167525,
"MDEntrySize" : 196,
"MDEntryTime" : "145036000",
"TradingSessionID" : "HALFDAY",
"NumberOfOrders" : 35,
"MDPriceLevel" : 1
}
],
"CheckSum" : "248"
}
from fix2json.
salsferrazza
commented on June 2, 2024
OK great - I will likely merge this into the master branch sometime soon, but am still looking into how fix2json treats repeating groups within message types other than MarketData* (e.g. SecurityDefinition).
from fix2json.
jlroo
commented on June 2, 2024
Sal,
I tested the latest fix2json version on the most recent CME data ( 3/15/16 | FIX Format (MDP 3.0) ) and I am getting the same error as before (node Version - v5.10.0 ), any suggestions on how to fix this issue? Thank you!
fix2json.js:91
var type = TAGS[num].type ? TAGS[num].type : 'STRING';
^
TypeError: Cannot read property 'type' of undefined
at pluckGroup (fix2json.js:91:23)
at resolveFields (fix2json.js:130:22)
at processLine (fix2json.js:141:21)
at Interface.<anonymous> (fix2json.js:59:32)
at emitOne (events.js:90:13)
at Interface.emit (events.js:182:7)
at Interface._onLine (readline.js:211:10)
at Interface.<anonymous> (readline.js:341:12)
at Array.forEach (native)
at Interface._normalWrite (readline.js:340:11)
Last Two json Msgs
{
"ApplVerID":"FIX50SP2",
"BodyLength":182,
"MsgType":"MARKETDATAINCREMENTALREFRESH",
"SenderCompID":"CME",
"TradeDate":"20160315",
"MsgSeqNum":5916296,
"SendingTime":"20160314215959992455478",
"TransactTime":"20160314215959992368638",
"MatchEventIndicator":"10000100",
"NoMDEntries":1,
"MDEntries":[
{
"MDUpdateAction":"CHANGE",
"MDEntryType":"OFFER",
"SecurityID":"49705",
"Symbol":"ESH6",
"RptSeq":2434571,
"MDEntryPx":201900,
"MDEntrySize":11,
"NumberOfOrders":7,
"MDPriceLevel":1
}
],
"CheckSum":"018"
}
{
"ApplVerID":"FIX50SP2",
"BodyLength":131,
"MsgType":"SECURITYSTATUS",
"SenderCompID":"CME",
"MsgSeqNum":5916297,
"SendingTime":"20160314220000004659459",
"TransactTime":"20160314220000000973568",
"TradeDate":"20160315",
"MatchEventIndicator":"00000000",
"SecurityGroup":"ES",
"SecurityTradingStatus":"NEW PRICE INDICATION",
"HaltReasonInt":"NEWS DISSEMINATION",
"SecurityTradingEvent":0,
"CheckSum":"084"
}
Last processed FIX Msg
1128=9^A9=131^A35=f^A49=CME^A34=5916297^A52=20160314220000004659459^A60=20160314220000000973568^A75=20160315^A5799=00000000^A1151=ES^A326=15^A327=0^A1174=0^A10=084^A
Next Three Msg
1128=9^A9=259^A35=X^A49=CME^A75=20160315^A34=5916298^A52=20160314220000004759687^A60=20160314220000000973568^A5799=00000000^A268=1^A279=0^A269=2^A48=49705^A55=ESH6^A83=2434572^A270=201900.0^A271=6^A346=7^A5797=0^A37705=7^A37=0^A32=1^A37=0^A32=2^A37=0^A32=5^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A10=232^A
1128=9^A9=413^A35=X^A49=CME^A75=20160315^A34=5916300^A52=20160314220000005077121^A60=20160314220000000973568^A5799=00000001^A268=1^A279=0^A269=2^A48=6505^A55=ESM6^A83=4428283^A270=200925.0^A271=36^A346=22^A5797=0^A37705=22^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A37=0^A32=3^A37=0^A32=1^A37=0^A32=2^A37=0^A32=6^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A37=0^A32=1^A37=0^A32=10^A37=0^A32=10^A37=0^A32=2^A37=0^A32=8^A37=0^A32=8^A37=0^A32=4^A37=0^A32=6^A37=0^A32=2^A37=0^A32=1^A10=172^A
1128=9^A9=202^A35=X^A49=CME^A75=20160315^A34=5916301^A52=20160314220000005716580^A60=20160314220000000973568^A5799=00000010^A268=2^A279=0^A269=e^A48=49705^A55=ESH6^A83=2434573^A271=42^A279=0^A269=e^A48=6505^A55=ESM6^A83=4428284^A271=72^A10=031^A
from fix2json.
salsferrazza
commented on June 2, 2024
Hi Jlroo,
Thanks for sending this over, will pull down those new datasets and let you know if I can see anything.
Sal
from fix2json.
salsferrazza
commented on June 2, 2024
Hi jlroo,
I've updated the CME data dictionary and the error is gone although the parser seems to be omitting the value of the final OrderID in the last member of the repeating group NoOrderIDEntries. Can you have a look and let me know if it's better for the data sets you are running fix2json against?
{
"ApplVerID": "FIX50SP2",
"BodyLength": 259,
"MsgType": "MARKETDATAINCREMENTALREFRESH",
"SenderCompID": "CME",
"TradeDate": "20160315",
"MsgSeqNum": 5916298,
"SendingTime": "20160314220000004759687",
"TransactTime": "20160314220000000973568",
"MatchEventIndicator": "00000000",
"NoMDEntries": 1,
"MDEntries": [
{
"MDUpdateAction": "NEW",
"MDEntryType": "TRADE",
"SecurityID": "49705",
"Symbol": "ESH6",
"RptSeq": 2434572,
"MDEntryPx": 201900,
"MDEntrySize": 6,
"NumberOfOrders": 7,
"AggressorSide": 0,
"NoOrderIDEntries": 7,
"OrderIDEntries": [
{
"OrderID": "0",
"LastQty": 1
},
{
"LastQty": 2,
"OrderID": "0"
},
{
"LastQty": 5,
"OrderID": "0"
},
{
"LastQty": 1,
"OrderID": "0"
},
{
"LastQty": 1,
"OrderID": "0"
},
{
"LastQty": 1,
"OrderID": "0"
},
{
"LastQty": 1
}
]
}
],
"CheckSum": "232"
}
The parser is also putting NoOrderIDEntries underneath the NoMDEntries group, but it should be moved up one level, obviously based on the Merc's own MDP FIX documentation. Still working on this but may track it with a separate issue. Stay tuned. But in the interim you can just update the FIX50SP2.CME.xml dictionary currently committed into the master branch.
from fix2json.
jlroo
commented on June 2, 2024
Sal, Thank you for all your great work. In the future if I encounter this error I will refer to the dictionary.
from fix2json.
Related Issues (19)
- Validate FIX repeating groups handling HOT 11
- TODO: Remove mnemonic lookups of value data
- TODO: Spec says no dictionary should be necessary, then how to handle repeating groups + round-trip
- TODO: Remove manufactured group field names and store array under No* field directly HOT 1
- TODO: Create header, body and trailer objects for output JSON. Need to cache a map of field to section, i.e. CheckSum->Trailer.
- TODO: Consider Output and Input strategy implementation
- TODO: Implement round-trip of json2fix from a spec-compliant object. Should be bit-identical each side of round-trip, with no semantic decay
- As per FIX JSON encoding spec, remove Checksum field from output JSON
- /usr/bin/env: ‘node’: No such file or directory HOT 4
- CME Instruments File HOT 1
- Specify data dictionary name in lieu of full path to XML
- XML merge for data dictionaries HOT 3
- Relying on position of message attributes causes incompatibility with external dictionaries
- Dictionaries update HOT 5
- Return integer data type HOT 5
- Support for compress files HOT 2
- Add extension pack support for dictionaries
- TODO: Represent numeric values as strings (see: early versions of fix2json)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fix2json.