Python 3.11.0 is out. Add it to CI / PyPI.

No docstrings in FixParser.

Rather than requiring that a new parser be created, it'd be nice to be able to reset the state of the parser if, eg. a connection is dropped.

In some cases, the SOH isn't a field boundary.
Check 95/96, 212/213, and 347-365.
There might be others too.

I guess this will change the parser to need to look tag by tag, check for a length tag, and then when it hits the following body tag, to use the previously-acquired length, rather than searching ahead for the SOH?

Is it the case that length must occur before the data?
Does the length need to immediately precede the data?

I've been thinking about adding a socket connection class to a generic FIX IP, was curious if there's anything in beta for this? If not, I'm happy to fork and contribute what I can to it if I have some success -- I was going to connect to IB.

Let me know your thoughts!

Simplefix is INCREDIBLE, so simple -- exactly what I was looking for -- FIX formatting without the overhead mess of QuickFix -- just needs a Socket initiator : )

Zach

So now that Python 2.7 is officially unsupported, I'd like to flag a date beyond which simplefix won't commit to support it.

In common with a few other projects, I'm thinking maybe 31 December 2020 sounds like a good sunset date.

I won't deliberately break it on 1 Jan 2021, but I anticipate removing Python 2.7 from the automated builds, and any new features won't be tested on 2.7.

Any objections?
Anyone still using Python 2.7 today, and won't be migrating by then?

The CME (bless 'em) publishes a list of active product definitions using a text file containing (very) many FIX 35=a messages (secdef.dat). Or almost FIX messages, except that they don't have a trailing CheckSum (10) field: messages are terminated by an end-of-line.

It'd be useful to parse this file using simplefix, but FixParser is currently relying on finding a 10=xxx| field to identify the end of a message.

Add optional support to the parser to terminate messages with an end-of-line instead.

A function that takes tag number and tag name as input and stores it as dictionary or any other data type.
It can also return tag name when given the corresponding tag number.

Hi David!
Thank you so much for all of this! I'm trying to use this to parse "ICE (intercontinental exchange) Private order feed" FIX messages for my company. I'm able to use your modules to parse the ICE orders, but not the "security definitions" from ICE which are these huge messages that Ice sends us to describe the ICE commodity products. I attached a sample of one of these. Was hoping you could have a look when u have time. Thank you Thank you Thank you! - Jerry

definition.txt

Rather than requiring the application to futz around with formatting time/datetime values, a helper method might be useful. It should probably default to millisecond-precise current time, with an option for microseconds and use of a different timestamp value.

hi, maybe we could implement a group parsing object

it receive the group tags that exists in a group at construction
when init it receive the tag group name (nomdentries) and the group that will be read
it find the first tag and the last tag, with that we can recursive find group inside group
the point is find the start byte and final byte of a group

do you think it's a good idea? i could help implementing

Hello,
Thanks for the awesome library. One quick question - once i have the byte buffer after doing the encode(), how do I convert it to a string so that or "\x01" characters are converted to "^A" characters ? I tried various different ways to do this, like repr(byte_buffer) or (ord(x) in bye_buffer) etc. but could not.
My bytes array looks like below -
b'8=FIX.4.4\x019=65\x0135=0\x0152=20220809-21:37:06.893\x0149=TRADEWEB\x0156=ABFIXREPO\x01347=UTF-8\x0110=045\x01'
Any idea ? Thanks

the README refers to http://www.quickfixengine.org/ but that now redirects to some kind of shopping site: https://hexleet.pw/login.html

The FIX protocol specifies that the checjsum should be 3 ASCII characters. I am observing that simplefix (1.0.16) sometimes sends less, i.e., 2 characters:
2023-09-11 13:16:44,485 : process_buffer : INFO : Sending FIX heartbeat message 8=FIX.4.4|9=67|35=0|49=QAORG1_UAT|56=STP_ROUTER_UAT|34=2|52=20230911-17:16:44.485|10=88

Per https://www.fixtrading.org/groups/continuousmkts/
Sequence number resets are a bore.

The implementation proposed is to use an unsigned 64 bit integer for a sequence number.

Using 1 billion sequence numbers per second, that will last for 584 years until they run out.

So we can leave the problem to our great, great, great (more) grandchildren to solve!

Would be good to include a set of test cases so that testing with a counterpart engine that maybe uses an int32 or uint32 can be picked up quickly. And cover all the other firles that use the seqnum "datatype"

Describe alternatives you've considered
See attached "Continuous Markets WG- 24x7 FIX Best Practise Working Draft_v0.015.docx"
Continuous Markets WG- 24x7 FIX Best Practise Working Draft_v0.015.docx

I think we use this -- need to check (from What's New in Python 3.12):

datetime’s utcnow() and utcfromtimestamp() are deprecated and will be removed in a future version. Instead, use timezone-aware objects to represent datetimes in UTC: respectively, call now() and fromtimestamp() with the tz parameter set to datetime.UTC. (Contributed by Paul Ganssle in gh-103857.

In fact I do not know how to use it .

FIX uses the ASCII SOH character to separate (well, maybe terminate) fields. But for human consumption, these are often replaced with, eg. | when logging messages.

Add optional support for parsing messages using non-standard separators.

It's very common for FIX log files to be written with one message per line, and a direction (send/recv) and timestamp prior to the message.

It'd be nice if FixParser were able to parse such files "out of the box" without having to trim the lines manually first.

Add an optional FixParser behaviour to enable it to ignore spurious characters before the 8=FIX and after the 10=xxx|, such that it can read a FIX log file into a sequence of messages with a single call to append_buffer().

is this app packaged in pip (it's a python app right)? can I install it from an apt repo? do clone the project and compile? there is zero guidance on the README as to how to use this

when i try to create new message using : "fix_msg = simplefix.Message()" an error pop up as:
"File "C:/simplefix-master/simplefix/simple.py", line 4, in
fix_msg = simplefix.Message()
AttributeError: module 'simplefix' has no attribute 'Message'"

and please may you explain the difference between "SIMPLEFIX" and "QUICKFIX" or put any links for atricles

Hello,

I am looking for FAST protocol parser in Python.
Otherwise, an explicit documentation, to develop a parser.
Do you have any information to share?
Thank you

Elie

It'd be nice to have symbolic names available for tag numbers and enum values.
These could be snarfed from FTC (5.0sp2, I guess).

I grabbed simplefix/test/test_message.py and tried to run it but it fails the following test:

$ python3.9 t

........................................./usr/local/lib/python3.9/site-packages/simplefix/message.py:147: DeprecationWarning: simplefix.FixMessage.append_time() is deprecated. Use append_utc_timestamp() or append_tz_timestamp() instead.
warnings.warn("simplefix.FixMessage.append_time() is deprecated. "
........E...........................................

ERROR: test_to_String (main.MessageTests)
Test conversion to string with separator.

Traceback (most recent call last):
File "/Users/ekkis/dev/t", line 875, in test_to_String
buffer = msg.to_string('XXX')
AttributeError: 'FixMessage' object has no attribute 'to_string'

---

Ran 93 tests in 0.018s

FAILED (errors=1)

what does it need?

import simplefix

fix_msg = simplefix.Message()

Traceback (most recent call last):
File "C:/Users/irmsc/pp/simple_fix_test/main.py", line 3, in
fix_msg = simplefix.Message()
AttributeError: module 'simplefix' has no attribute 'Message'

Now that 3.6 is the minimum supported Python version, we can start using f-strings.
Check through all string formatting, and see if there are places it makes sense to change.

I am trying to use simplefix to create a very simple FIX engine implementation in pure Python.

Would you be accepting of some code that allows the user, when building a message, to add tags to the front of the tag list (i.e. header tags). To actually use this in a FIX engine implementation, one needs to be able to add the sequence number (34) and sending time (52) right before the message is sent (not when the bulk of the message is created).

I'm happy to write the code, but wanted to know if you'd accept it as a PR or if you'd rather I simply forked things.

I am using simplefix version 1.0.14. And it gives me error calling get with the nth parameter.

    Traceback (most recent call last):
      File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 954, in _bootstrap_inner
        self.run()
      File "/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 892, in run
        self._target(*self._args, **self._kwargs)
      File "/Users/recapital/Git/tt-fix/client.py", line 59, in run
        self._reader.on_message(msg)
      File "/Users/recapital/Git/tt-fix/message_handler.py", line 31, in on_message
        self.handle_logon(message)
      File "/Users/recapital/Git/tt-fix/message_handler.py", line 39, in handle_logon
        logon.decode(message)
      File "/Users/recapital/Git/tt-fix/tt_messages.py", line 9517, in decode
        if (val := message.get(FIELD_ENCRYPT_METHOD, nth)) is not None:
    TypeError: <lambda>() takes 1 positional argument but 2 were given

Here is the code called:

    def decode(self, message: FixMessage, nth: int = 1):
        if (val := message.get(FIELD_ENCRYPT_METHOD, nth)) is not None:
            self.encrypt_method = EncryptMethod(int(val))

Here is the field constant:

FIELD_ENCRYPT_METHOD = b'98'

And here is the get method in simplefix:

    def get(self, tag, nth=1):
        """Return n-th value for tag.

        :param tag: FIX field tag number.
        :param nth: Index of tag if repeating, first is 1.
        :return: None if nothing found, otherwise value matching tag.

        Defaults to returning the first matching value of 'tag', but if
        the 'nth' parameter is overridden, can get repeated fields."""

        tag = fix_tag(tag)
        nth = int(nth)

        for t, v in self.pairs:
            if t == tag:
                nth -= 1
                if nth == 0:
                    return v

        return None

It works if I don't pass the nth parameter, but providing this (that I have to due to repetitions) gives this weird error.

Message that I received before the decode error.
8=FIX.4.4|9=00080|35=A|49=NOTHING|56=CAPITALAS_MD|34=1|52=20210622-13:59:02.752|98=0|108=30|141=Y|10=085

Seems that distutils is now deprecated, and we'll need to ditch setup.py and switch to pyproject.toml plus something.
Figure out what, and switch over.

The CME sends execution reports over drop copy with the format:

8=FIX.4.2|35=n|...|213=<RTRF>8=FIX.4.2|...|10=XXX</RTRF>|10=YYY

If this message is split inside the XML then FixParser will interpret 10=XXX as the end of the message instead of 10=YYY.

Test Case:

import unittest
import os

import simplefix

PART1 = b'8=FIX.4.2\x019=606\x0135=n\x0134=18\x01369=XX\x0152=XXXXXXXX-XX:XX:XX.XXX\x0143=Y\x0149=CME\x0150=G\x0156=XXXXXXX\x0157=NULL\x01122=XXXXXXXX-XX:XX:XX.XXX\x01143=XX\x01212=481\x01213=<RTRF>8=FIX.4'

PART2 = b'.2\x019=XXX\x0135=8\x0134=2087\x01369=2122\x0152=XXXXXXXX-XX:XX:XX.XXX\x0149=CME\x0150=G\x0156=XXXXXXX\x0157=XXXXXXXXXX\x01143=XXXXX\x011=XXXXXXXX\x016=X\x0111=XXXXX\x0114=XX\x0117=XXXXXXXXXXXXXXXXXXXXXXX\x0120=X\x0131=XXXXXX\x0132=X\x0137=XXXXXXXXXXXX\x0138=XXX\x0139=X\x0140=X\x0141=X\x0144=XXXXXX\x0148=XXXXXX\x0154=X\x0155=XX\x0159=X\x0160=XXXXXXXX-XX:XX:XX.XXX\x0175=XXXXXXXX\x01107=XXXX\x01150=X\x01151=XX\x01167=FUT\x01337=TRADE\x01375=CME000A\x01432=XXXXXXXX\x01442=1\x01527=XXXXXXXXXXXXXXXXXXXXXXXX\x011028=Y\x011057=Y\x015979=XXXXXXXXXXXXXXXXXXX\x019717=XXXXX\x0137711=XXXXXX\x0110=171\x01</RTRF>\x0110=169\x01'

class TestSimpleFix(unittest.TestCase):
    def test_simplefix_on_split_execution_report(self):

        # msg parses correctly if both byte strings
        # appended before calling get_message()
        parser = simplefix.FixParser()
        parser.append_buffer(PART1)
        parser.append_buffer(PART2)
        msg = parser.get_message()
        self.assertEqual(msg.get(10), b'169')

        # call get_message() after first byte string
        parser = simplefix.FixParser()
        parser.append_buffer(PART1)
        msg = parser.get_message()
        self.assertEqual(msg, None)  # correct behavior

        # append second byte string
        parser.append_buffer(PART2)
        msg = parser.get_message()
        checksum = msg.get(10).decode('utf-8')
        self.assertEqual(checksum, '169', "Expected 169, probably got 171 indicating incorrect parsing of split execution report message.")

Hi There,

This project is still using an old implementation of LGTM's automated code review, which has now been disabled. To continue using automated code review, and receive checks on your Pull Requests, please install the GitHub App on this repository.

Thanks,
The LGTM Team

There's no (good) way to discover what fields are set in a message.
It'd be nice to be able to iterate over the fields that are present.

4.2 is quite common, but I notice in the tests that there doesn't seem to be any mention of version 4.4. I understand that this is a simple version, but even my employer now requires 4.4 as do many liquidity providers.

Hi,

I noticed one issue with the parser, and would like to submit a patch. I could not attach the patch file to this message (GitHub does not allow the patch extension), so I am sending it below.

From: Humberto Naves <[email protected]>
Date: Thu, 3 Feb 2022 12:52:57 -0500
Subject: [PATCH] Parser now accepts empty string as the value in the tag/value
 pair.

Previously the parser would parse the FIX message "TAG1=|TAG2=VAL|" as a
single pair with tag "TAG1" and corresponding value "|TAG2=VAL", which
is incorrect. It should instead parse this message into two pairs:
("TAG1", ""), and ("TAG2", "VAL"). This commit fixes this issue.

Note that the character `|` is being used as a representation of the
EOH byte in the message above.
---
 simplefix/parser.py |  3 +++
 test/test_parser.py | 13 +++++++++++++
 2 files changed, 16 insertions(+)

diff --git a/simplefix/parser.py b/simplefix/parser.py
index 2d59ffa..32e0ae7 100644
--- a/simplefix/parser.py
+++ b/simplefix/parser.py
@@ -192,6 +192,9 @@ class FixParser(object):  # skipcq: PYL-R0205
                     in_tag = False
                     start = point
 
+                # To avoid skipping a pair when the value is empty
+                point -= 1
+
             elif c == self.stop_char:
                 if start != point:
                     value = self.buf[start:point]
diff --git a/test/test_parser.py b/test/test_parser.py
index 775b069..1977392 100644
--- a/test/test_parser.py
+++ b/test/test_parser.py
@@ -317,6 +317,19 @@ class ParserTests(unittest.TestCase):
         self.assertIsNotNone(m)
         self.assertEqual(b"3", m.get(34))
 
+    def test_empty_value(self):
+        """Test empty value in message."""
+
+        buf = b'8=FIX.4.2\x019=9\x0135=D\x0129=\x0110=098\x01'
+
+        p = FixParser()
+        p.append_buffer(buf)
+
+        m = p.get_message()
+        self.assertIsNotNone(m)
+        self.assertEqual(b"D", m.get(35))
+        self.assertEqual(b"", m.get(29))
+
 
 if __name__ == "__main__":
     unittest.main()
-- 
2.35.1

There is some reason to return null instead of raising exceptions in parser.get_message ?