When are you going to adopt Python 3? about kuiper HOT 4 OPEN

hello
there is a lot of security concerns with Kuiper not only vulnerable libraries (such as not proper authentication and authorization), if security is important i recommend to not connect it to enterprise network (use it in lab network and only allow specific users to access the lab network).

for the python3, I'm working on new release with complete rebuild from scratch with proper structure and new features, of course it is python3, currently I'm testing it and checking the licensing.

from kuiper.

Hey @salehmuhaysin,

It's not directly related to this issue but I wanted to bring something to your attention that could be worthwhile to add to your new release before it's out (in case you are still using MongoDB and didn't change the collection layouts).

We noticed a critical issue with the MongoDB files collection. Since metadata about all files that belong to a machine is stored in a single document, large machines with around 30k files become highly problematic. The max document size in MongoDB is 16 MB and it cannot be increased without resorting to tricks.

So I put some work into restructuring the collection to represent each file as a single document. This has the following advantages:

• Document size limit is never hit.
• The flattened design allows all queries to be directly executed on the database without having to do processing/filtering on the Python side, greatly increasing performance in many scenarios.
• The old code has concurrency problems if multiple tasks are running for the same machine, e.g., updates can be lost. The new code does not have this problem because it expresses everything through atomic db operations.

You can find the changes here: https://github.com/GDATAAdvancedAnalytics/Kuiper/tree/mongodb-revamp - feel free to apply it to your code base or take inspiration from it. :)

I don't want to open a PR for this for Kuiper v2 because it's a breaking change for existing deployments. Theoretically it would be feasible to integrate it, but it'd require some fancy automatic migration mechanism for existing data.

from kuiper.

hello,
There are a lot of issues with the structure of Kuiper, specially the mongoDB database and flask which make adding new features of fixing issue very hard.
mongoDB is no longer in the new release since it is not efficient to do simple things like relationship and has other issues for scaling,
the new release will be commercial in cloud with more features added including full API, user management, etc. it should be released soon.

still will continue the Kuiper as open source but the new one will be different since it is restructured from scratch

from kuiper.