answer to #45 (comment)

I implemented it using a high level approach, should be good enough to be used with pymongo dictionaries, but I didn't think about supporting other kind of objects (that would lead again to deepcopy, unless we are able to drop the abstractions and go low level, but I didn't find anything about copy on write on the python ecosystem (this is an interesting read but unrelated) so i don't know how to proceed in that direction, i'm not a kernel developer)

Proof Of Concept

from collections.abc import Mapping, ValuesView


class CopyOnWrite(Mapping):
    def __init__(self, data):
        self.__dict__['_diff'] = {}
        self.__dict__['_data'] = {}
        for k, v in data.items():
            self._data[k] = v if not isinstance(v, dict) else self.__class__(v)

    def __getitem__(self, key):
        if key in self._diff:
            return self._diff[key]
        return self._data[key]

    def __setitem__(self, key, value):
        if not isinstance(value, dict):
            self._diff[key] = value
        else:
            self._diff[key] = self.__class__(value)

    __setattr__ = __setitem__
    __getattr__ = __getitem__

    @property
    def original(self):
        res = {}
        for k, v in self._data.items():
            if isinstance(v, self.__class__):
                res[k] = v.original
            else:
                res[k] = v
        return res

    def __len__(self):
        return len(self._data | self._diff)

    def __iter__(self):
        return iter(self._data | self._diff)

    def values(self):
        # TODO: unsure about this, docs says a view should reflect the changes
        # TODO: does this work at deeper levels?
        values = self._data | self._diff
        for k, v in values.items():
            if isinstance(v, self.__class__):
                values[k] = v._data | v._diff
        return ValuesView(values)
        


if __name__ == '__main__':
    original = {
        'a': 'A',
        'd': {'z': 'Z'},
        'l': [1, 2, 3],
    }
    cow = CopyOnWrite(original)

    assert cow.original == original, cow.original
    
    cow.a = 'AA'
    assert cow.original == original, cow.original
    assert cow.a == 'AA'
    assert cow['a'] == 'AA'

    cow.d.x = 'X'
    cow.d.z = 'ZZZ'
    assert cow.original == original, cow.original
    assert cow.d.x == 'X'
    assert cow.d.z == 'ZZZ'

    cow.l.append(4)
    assert cow.original == original, cow.original
    assert cow.l[-1] == 4

    cow.n = 'new'
    assert cow.original == original, cow.original
    
    assert len(cow) == 4, len(cow)

    for i, n in enumerate(cow):
        assert n == ('a', 'd', 'l', 'n')[i]

    assert list(cow.keys()) == list(original.keys()) + ['n']

    assert list(cow.values()) == ['AA', {'x': 'X', 'z': 'ZZZ'}, [1, 2, 3, 4], 'new']

    assert dict(cow.items()) == dict(zip(list(original.keys()) + ['n'],
                                         ['AA', {'x': 'X', 'z': 'ZZZ'}, [1, 2, 3, 4], 'new']))

How to apply to the current Object/ObjectState/Tracker

I need to check.

class WikiPage(MappedClass):
class mongometa:
session = session
name = 'wiki_page'

_id = FieldProperty(schema.ObjectId)
title = FieldProperty(schema.String(required=True))
text = FieldProperty(schema.String(if_missing=''))

I am developing a multiprocess app . For each process, I would like to create a new ThreadLocalODMSession.

Given the above declarative method, how would I bind the MappedClass to different session?

I have already tried numerous ways but none seem to work.

Hi,

We have been advised by MongoDB that we are hitting a condition in the Pymongo code that doesnt recover from instance failover's very well and that we need to upgrade from Pymongo 3.6 to Pymongo 3.10.

Unfortunately, ming, does not yet support PyMongo as far back as 3.8
https://github.com/TurboGears/Ming/blob/master/setup.py#L38.

This dependency is also preventing us from upgrading our MongoDB Cluster.

Are there any plans to move this dependency forward? I can see a similar issue was raised for version 3.8 support (#12)

Thanks in advance

It'd be nice to test a matrix of mongodb major versions. Seems like that's not possible with GH action's services: and image: values. But https://github.com/supercharge/mongodb-github-action appears to support it.

While working on the fix of #25, I found that sometimes the wrong behavior occurs. On branch mim_dot_dollar_wrong this is illustrated by test_inc_dotted_dollar_middle2 with a $ in the middle of dotted field notation, and selecting for the 2nd subdoc in the list:

AssertionError: {'b': {'f': [{'g': 11}, {'g': 2}]}} != {'b': {'f': [{'g': 1}, {'g': 12}]}}
- {'b': {'f': [{'g': 11}, {'g': 2}]}}
?                    -

+ {'b': {'f': [{'g': 1}, {'g': 12}]}}
?

That branch has expanded tests where the $ is at the end of dotted field notation, and they do all pass.

I haven't looked into this any further, seems kinda tricky. And not actually a blocking problem for me right now.

A couple errors like this from the Allura test suite, when going from pymongo 3.7.2 to pymongo 3.8 (on ming master):

  File "/Users/brondsem/sf/allura/Allura/allura/ext/admin/admin_main.py", line 412, in update
    c.project.save_icon(icon.filename, icon.file, content_type=icon.type)
  File "/Users/brondsem/sf/allura/Allura/allura/model/project.py", line 376, in save_icon
    icon_orig_img = PIL.Image.open(icon_orig.rfile())
  File "/Users/brondsem/sf/env-allura/lib/python2.7/site-packages/PIL/Image.py", line 2804, in open
    im = _open_core(fp, filename, prefix)
  File "/Users/brondsem/sf/env-allura/lib/python2.7/site-packages/PIL/Image.py", line 2789, in _open_core
    fp.seek(0)
  File "/Users/brondsem/sf/env-allura/lib/python2.7/site-packages/gridfs/grid_file.py", line 643, in seek
    self.__chunk_iter.close()
  File "/Users/brondsem/sf/env-allura/lib/python2.7/site-packages/gridfs/grid_file.py", line 778, in close
    self._cursor.close()
AttributeError: 'Cursor' object has no attribute 'close'

I have replicated the example in https://ming.readthedocs.io/en/latest/polymorphism.html.

I have added a Bike and can reproduce the queries below as in the documentation.

from ming import schema
from ming.odm import FieldProperty
from ming.odm.declarative import MappedClass

class Transport(MappedClass):
    class __mongometa__:
        session = session
        name = 'transport'
        polymorphic_on = '_type'
        polymorphic_identity = 'base'

    _id = FieldProperty(schema.ObjectId)
    origin = FieldProperty(schema.String(required=True))
    destination = FieldProperty(schema.String(if_missing=''))
    _type = FieldProperty(schema.String(if_missing='base'))

    def move(self):
        return 'moving from {} to {}'.format(self.origin, self.destination)

class Bus(Transport):
    class __mongometa__:
        polymorphic_identity = 'bus'

    _type=FieldProperty(str, if_missing='bus')
    passengers_count = FieldProperty(schema.Int(if_missing=0))

    def move(self):
        return 'driving from {} to {}'.format(self.origin, self.destination)


class AirBus(Bus):
    class __mongometa__:
        polymorphic_identity = 'airbus'

    _type=FieldProperty(str, if_missing='airbus')
    wings_count = FieldProperty(schema.Int(if_missing=2))

    def move(self):
        return 'flying from {} to {}'.format(self.origin, self.destination)

class Bike(Transport):
    class __mongometa__:
        polymorphic_identity = 'bike'

    _type=FieldProperty(str, if_missing='bike')

    def move(self):
        return 'cycling from {} to {}'.format(self.origin, self.destination)


Bus(origin='Rome', destination='London', passengers_count=20))
Bus(origin='Turin', destination='London', passengers_count=20))
AirBus(origin='Turin', destination='London', passengers_count=60, wings_count=3))
Bike(origin='Newcastle', destination='London'))
session.flush()
session.clear()

print(Transport.query.find().count())  # 4
print(Transport.query.find({'_type': 'bus'}).count())  # 2
print(Transport.query.find({'_type': 'airbus'}).count()) # 1

However I have also added next queries and would expect this to produce 3 and 1:

print(Bus.query.find().count())  # expected 3, got 4
print(AirBus.query.find().count())  # expected 1, got 4

but I get 4 and 4.

https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.find_and_modify

Is there currently any chance to support the relatively new multi-document transactions provided by MongoDB > 4.0?

Next code works with M but fails with MappedClass:

#!/usr/bin/env python

from ming.odm import MappedClass

# class M:
#     pass

# class A(M):
class A(MappedClass):

    def __init_subclass__(cls, constant=1, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.prop = constant

class B(A, constant=2):
    pass

print(B().prop)

It seems to be fixed just by adding a **kwargs parameter to __new__ (and * and / parameters if appropriate).

Some refs:

• pydantic/pydantic#867
• https://stackoverflow.com/questions/42281697/typeerror-when-combining-abcmeta-with-init-subclass-in-python-3-6

Ming doesn't support connection string with ssl true.
Got this error while calling create_datastore() with connecting string having ssl true

ServerSelectionTimeoutError: SSL handshake failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841),SSL handshake failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841),SSL handshake failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)

The listed breaking changes in the release are:

• Commands that fail with server error codes 10107, 13435, 13436, 11600, 11602, 189, 91 (NotMaster, NotMasterNoSlaveOk, NotMasterOrSecondary, InterruptedAtShutdown, InterruptedDueToReplStateChange, PrimarySteppedDown, ShutdownInProgress respectively) now always raise NotMasterError instead of OperationFailure.
• parallel_scan() no longer uses an implicit session. Explicit sessions are still supported.
• Unacknowledged writes (w=0) with an explicit session parameter now raise a client side error. Since PyMongo does not wait for a response for an unacknowledged write, two unacknowledged writes run serially by the client may be executed simultaneously on the server. However, the server requires a single session must not be used simultaneously by more than one operation. Therefore explicit sessions cannot support unacknowledged writes. Unacknowledged writes without a session parameter are still supported.

While the latter two don't seem to be a problem, we might need to catch the new exceptions in the ming.session.annotate_doc_failure function where we're only catching OperationFailure and BSONErrors. any thoughts? @amol- @percious

I dropped py2 support in 6c7ed0a but only did a bit of cleanup.

We can remove __future__ imports, various six usage, etc. https://github.com/asottile/pyupgrade can help.

MongoDB 3.4 introduced support for the Decimal type (https://docs.mongodb.com/manual/tutorial/model-monetary-data/#numeric-decimal). Pymongo supports it from v3.7, I think Ming should be able to use it as well.
The main issue I see with this is the ming.base._safe_bson function, that explicitly casts decimal.Decimal values to float. Refactoring it would probably be a breaking change for some application relying on that cast to ensure consistency in the value type of that field.
@amol- any suggestions on how to safely implement this, considering also retrocompatibility with MongoDB < 3.4?

Hello,

I'd like to know if there's a reason ming 0.9.1 requires pymongo < 3.8 and not <= 3.8, was there a problem running ming with pymongo 3.8.0 or was it only not tested?

I'm willing to help if there's anything I can do to fix/test ming support with pymongo 3.8.0.

Thank you,

Yaniv

Formencode no longer supports Python3.9 and seems to have been unmaintained for a few years.
We should remove the need for it from Ming.

Mim does not match mongodb's behavior concerning matching sub-lists with $ne when the field is an empty list.

    self.bind.db.coll.insert(
        {'_id':'foo', 'hist': [{'int': 2}]}
    )
    self.bind.db.coll.insert(
        {'_id':'foo2', 'hist': []}
    )
    self.bind.db.coll.insert(
        {'_id':'foo3'}
    )
    self.bind.db.coll.insert(
        {'_id':'foo4', 'hist': [{'foo': 'bar'}]}
    )

    objs = self.coll.find({'hist.int': {'$ne': 5235}}).all()
    # this returns foo, foo3, foo4 but should also return foo2

Note https://docs.mongodb.com/manual/reference/operator/query/ne/index.html doesn't specify behavior for empty lists, but we should match how mongodb actually works.

I am unable to find any license statement in pypi tarball. Could you please add one?

The CI runs from #53 show that python 3.6 isn't available on the default ubuntu runner. With python 3.6 past end-of-life I think its fine & good to drop support for it.

I'm sure we could keep supporting it for a while, and tweak our github CI run to use an ubuntu version that has 3.6 available. But seems good to support only current python versions

Pymongo 4.x has a lot of breaking changes. See https://pymongo.readthedocs.io/en/stable/changelog.html#changes-in-version-4-0 and https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html
Certainly Ming will be affected significantly by save/insert/update/remove/find_and_modify being replaced with _one and _many versions

Pymongo 3.x last release was 3.13 in Nov 2022 https://pypi.org/project/pymongo/#history and there is no 3.x branches on github so probably 3.x is done being supported

3.x does support mongodb 5.0 and python 3.10 so that'll last for a while.