ayeo-flex-org / pulsar-flex Goto Github PK

Is your feature request related to a problem? Please describe.
Currently, pulsar-flex can't recognize ProxyToBrokerUrl field, it can't support connect to pulsar-proxy

Describe the solution you'd like
Adapt pulsar-proxy protocol, recognize ProxyToBrokerUrl field, and send it to correct addr.

Thanks for advance. And thanks for built this library. It's great for peolple who wants to use pure js.
:)

Describe the bug
The test for Should read the unacknowledged message again before the rest of the flow, in batch, in shared subscription, checks that there are 4 messages received while in the pulsar documentation it states that if you negatively acknowledge a message in a batch it resends the whole batch.

Reproduce
Please provide reproduce, For example:

1. Run the consumer tests
2. The test sometimes fails and says it has more messages than expected in the expected queue
   Expected behavior
   The test should pass every time
   Observed behavior
   A clear and concise description of what did happen. Please include any relevant logs.

Is your feature request related to a problem? Please describe.
Just found your interesting looking pulsar client!
You are writing in About:

• Pulsar Flex is a modern Apache Pulsar client for Node.js, developed to be independent of C++.
• It was developed because the dependency in the official c++ external libraries does not fit some use cases.

Describe the solution you'd like
Would be pretty handy to give some more information on what is better/more modern and what use cases you have in mind ... in comparison to official node client.

Maybe one could also

• link directly to the repository of the official pulsar node client https://github.com/apache/pulsar-client-node and
• refer to missing feature in the official client https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1784579914

Additional context
btw: to connect communities, I opened an issue to link in Doc directly to interesting 3rd party clients like pulsar-flex
apache/pulsar#17113

Describe the bug
Consumer only gets the number of messages in receiveQueueSize and then stops reading.

Reproduce
Please provide reproduce, For example:

1. Run a producer that continuously produces messages to a topic
2. Run a consumer that subscribes to that topic and logs each topic offset
3. After the consumer has consumed receiveQueueSize messages, it stops.
   Expected behavior
   I expect it to read that number of messages as a sort of batch (only that amount can fit in the queue at once) but to still keep reading after reaching that amount.

Observed behavior
After the consumer has consumed receiveQueueSize messages, it just stops consuming messages.

Environment:

• OS: CentOS 7.8
• PulsarFlex version 1.0.1-beta.4
• Pulsar version 2.8.1
• NodeJS version 13.1.0

Additional context

Is your feature request related to a problem? Please describe.
Currently the system written in native Node js, we would like to add some @types library for other developers.

Describe the solution you'd like
Create a types library alongside this one.

Describe the bug
Producing in batch didn't work because it didn't return the command.

Reproduce
Please provide reproduce, For example:

1. Run a producer that continuously produces messages to a topic in batch

Expected behavior
Send batch messages successfully

@ronfarkash

Is your feature request related to a problem? Please describe.
When creating a producer / consummer, we have to specify a discovery server.

Describe the solution you'd like
I would like to directly specify the broker, and not a discovery server.

Additional context
I'm quite new to pulsar, so maybe i'm completly wrong.
But currently, when trying to establish a connexion, it successfuly connect to the remote server :
"Authentication to discovery service established, now will lookup topic", and then find me a localhost url, that can't work because everything is on the remote server.

{"level":"INFO","timestamp":"2022-10-06T09:04:03.388Z","logger":"pulsar-flex","message":"Creating client connection for producer to topic: persistent://public/default/MY_TOPIC"}
{"level":"INFO","timestamp":"2022-10-06T09:04:03.389Z","logger":"pulsar-flex","message":"Starting to lookup topic persistent://public/default/MY_TOPIC on __________________________________:6650"}        
{"level":"INFO","timestamp":"2022-10-06T09:04:03.417Z","logger":"pulsar-flex","message":"Connected successfully __________________________________:6650, now sending connect command."}
{"level":"INFO","timestamp":"2022-10-06T09:04:03.444Z","logger":"pulsar-flex","message":"Authentication to discovery service established, now will lookup topic"}
{"level":"INFO","timestamp":"2022-10-06T09:04:03.467Z","logger":"pulsar-flex","message":"Closing connection to discovery connection"}
{"level":"INFO","timestamp":"2022-10-06T09:04:03.469Z","logger":"pulsar-flex","message":"Lookup succeeded, owner is localhost:6650"}
Error: connect ECONNREFUSED 127.0.0.1:6650

@ronfarkash

Describe the bug
When a topic is unloaded the producer will keep reconnecting forever.

Reproduce
Please provide reproduce, For example:

1. Run a producer that continuously produces messages to a topic
2. Send unload command to the broker to unload the mentioned topic.
3. Watch the producer enter a reconnect loop.

Expected behavior
It's expected from the producer to reconnect once.

Observed behavior
Mentioned above.

Environment:

• OS: [Windows 10]
• PulsarFlex version [1.0.1-beta.4]
• Pulsar version [2.8.0]
• NodeJS version [v12.18.4]

Additional context
Nope.

@galrose

Describe the bug
Consume on high throughput sturggles to keep up and eventually leads to memory usage rising until crashing

Reproduce

1. Run a consumer that subscribes to that topic and logs each topic offset
2. Run a producer that continuously produces messages to a topic at 5MB/s (split between 70-100 messages per second)
3. After a certain amount of time (for me around 10/15 minutes) memory usage starts rising rapidly and the subscription's backlog size also rises steadily.
   Expected behavior
   I expect the consumer to be able to handle these kinds of throughput and keep up with the consumer with memory usage staying steady.

Observed behavior
Every time the dequeue function is called, I logged the length of receiveQueue. At some point, it barely manages to empty the queue, reaching thousands of messages rapidly (even though I set receiveQueueSize to 1000).
This can be seen even within seconds if the consumer starts with a few thousand messages in its backlog.

Environment:

• OS: CentOS 7.8
• PulsarFlex version 1.0.1-beta.7
• Pulsar version 2.8.1
• NodeJS version 16.13.1

Describe the bug
Flow does not get called when receive queue is empty

Describe the bug
The consumer does not have the properties field that the producer has

Is your feature request related to a problem? Please describe.
The feature is not related to a problem

Describe the solution you'd like
Implement negative acknowledgement for consumer to enable re-consumption of acknowledged messages.

Describe the bug
When consuming messages produced in a batch there aren't properties for the message
Reproduce
Please provide reproduce, For example:

1. Run a producer that produces a batch to a topic with properties
2. Run a consumer that subscribes to that topic
3. There aren't properties when reading the messages
   Expected behavior
   There should be properties

Describe the solution you'd like
I would like to be able to listen to events of state changes for the consumer (producer can also be nice). Say the consumer state changes to RECONNECTING, I want to catch it via an event and perform some logic.

Describe the bug
There is a warning that messages are going to the pending queue although they should not.
Reproduce
Please provide reproduce, For example:

1. Run a producer that continuously produces messages to a topic
2. Run a consumer that subscribes to that topic and logs each topic offset
3. After a while you will start seeing warning that the messages are entering the pending queue
   Expected behavior
   When there is nothing wrong with the connection the messages should not enter the pending queue

Observed behavior

Environment:

• OS: The docker compose
• PulsarFlex version [1.12.0]
• Pulsar version [2.8.1]
• NodeJS version [14.18.1]

@ronfarkash

@danielsinai

@danielsinai

Is your feature request related to a problem? Please describe.
Add the redelivery count to the base object to use

Describe the solution you'd like
The redelivery count in the basic object when reading a message

Describe the bug
When consuming messages, a memory leak is created due to unresolved promises. After leaving the consumer on for a while, a heap snapshot will show an increasing number of promise references.

Reproduce

1. Run a Consumer that continuously reads messages.
2. Watch the process memory go up as time passes.

Expected behavior
The process memory to stay stable.

Observed behavior
Explained above.

Environment:

• OS: Windows 10
• PulsarFlex version: 1.0.1-beta.8
• Pulsar version: 2.8.1
• NodeJS version: 16.11.1

Is your feature request related to a problem? Please describe.
The feature is not related to a problem.

Describe the solution you'd like
Implement redeliver of unacknowledged messages.