Maintain insert order for multi-valued properties about followthemoney HOT 4 CLOSED

@pudo I’ve added this as an issue here mostly to keep track of it as it’s part of a bigger project in Aleph. I’d be happy to put in a PR to resolve this though!

from followthemoney.

One thing I'd consider before making this guarantee to downstream consumers is how this will intersect with entity fragment aggregation. I've looked at this before a bit in the context of making the pages of a document show up in proper order in the indexText/bodyText props, and could never quite conceptualise all the places in the pipeline that would need to be updated for this guarantee to really be held. It's also something that will be really hard to do wrt. to the statement-based data model I use downstream in nomenklatura, but that's more of a "me" problem :)

from followthemoney.

One thing I'd consider before making this guarantee to downstream consumers is how this will intersect with entity fragment aggregation. I've looked at this before a bit in the context of making the pages of a document show up in proper order in the indexText/bodyText props, and could never quite conceptualise all the places in the pipeline that would need to be updated for this guarantee to really be held.

Thanks for mentioning this, I wasn’t aware of the (potentially) incorrect order of indexText property values before. My (maybe naive?) assumption though is that preserving order of property values shouldn’t make anything worse (or might even be necessary in order to be able to tackle them in the future), right?

While it won’t automatically solve the potentially incorrect order of indexText values (because the values are stored in different fragments), it will solve the issue with previews of multipart emails (because all parts of the original email end up in the same fragment).

from followthemoney.

@pudo Do you have a preference regarding the alternative data structure?

1. Using dict keys (or a small wrapper around dicts)
2. Using lists (and manually checking if a value already exists when adding new values)
3. Using a third-party ordered set implementation.

I have a personal preference for 2 to keep it simple, and I don’t think the perf overhead for O(n) membership tests would be relevant in Aleph. Not sure though about other FtM use cases like OpenSanctions?

from followthemoney.