dashersw / cote Goto Github PK
View Code? Open in Web Editor NEWA Node.js library for building zero-configuration microservices.
Home Page: http://cote.js.org
License: MIT License
A Node.js library for building zero-configuration microservices.
Home Page: http://cote.js.org
License: MIT License
Hello,
I want to use JWT between microservices (so between Requester
<=> Responder
, and Publisher
<=> Subscriber
). It is possible? Or something else to authenticate call?
Awesome project by the way: easy to understand and to use :)
I'm new to cote and maybe I'm missing something but I can't make more then 2 services to work.
Here is an isolated example:
index.js
var cote = require('cote');
var SERVERS = [
'server0',
'server1',
'server2'
];
var client = new cote.Requester({
name: 'index.js',
requests: SERVERS
});
function nextRequest(i) {
i = (i % SERVERS.length);
console.log('Request to server' + i);
client.send({type: 'server' + i}, function (response) {
console.log('response', response);
});
setTimeout(function () {
nextRequest(i + 1);
}, 1000);
}
client.on('ready', function () {
nextRequest(0);
});
server0.js
var cote = require('cote');
var server0 = new cote.Responder({name: 'server0', respondsTo: ['server0']});
server0.on('server0', function (req, cb) {
cb('server0 response ' + new Date());
});
server1.js
var cote = require('cote');
var server1 = new cote.Responder({name: 'server1', respondsTo: ['server1']});
server1.on('server1', function (req, cb) {
cb('server1 response ' + new Date());
});
server2.js
var cote = require('cote');
var server2 = new cote.Responder({name: 'server2', respondsTo: ['server2']});
server2.on('server2', function (req, cb) {
cb('server2 response ' + new Date());
});
To run it do:
npm install cote
node index.js &
node server0.js &
node server1.js &
node server2.js &
Or just run in different terminal windows.
When I change SERVERS
list in index.js file to just one server (server0) for example and run only node index.js
and node server0.js
then I got response for every request. Whenever I'm enabling more servers I'm starting to loose responses.
I'm using [email protected] and [email protected]
Am I missing something?
The readme features basic documentation for Sockends, and doesn't give a complete example for how and why it works. Therefore we need more detailed documentation in the readme. Certain things like simple security mechanisms via respondsTo
and broadcasts
configurations should also be taken into account.
A feature to return the list of currently connected cote nodes would be great for applications. Especially mine with hotswapping nodes.
Currently the Monitor component works in CLI, which is not the best place to inspect tens of other components. There should be a Sockend component in Monitor to push the network information to a management frontend. The frontend should draw the mesh network in a dynamic fashion, such as with D3.
Hi. I want to ask about events 'added' and 'removed'
When I add this snipped of code to responder:
responder.on( (req, info) => console.log(req, info) );
in console I see such outcome (when some component is found):
cote:added { isMaster: false,
isMasterEligible: true,
weight: -0.1512131192202,
address: '192.168.56.1',
advertisement:
{ name: 'REQUESTER',
key: '$$',
axon_type: 'req',
type: 'service' },
id: 'c63f7216-06ba-472a-9a37-6dcf3547b4bf',
processId: '7e12d231-3cb3-4d38-8b62-eda680cfbfa3',
processCommand: 'D:\\Projects\\cote-study\\ms\\requester',
lastSeen: 1512131202220,
hostName: 'DESKTOP-CC122GO',
port: 12345 }
Can anyone explain me what is it about and how can I utilize it?
Also I want to know when callback in 'on' method accepts more than 1 argument?
Thanks
@dashersw This is somewhat a two part question:
The documentation lists out which protocols are not used but does not really state what protocol is used in cotejs. It sort of implies that it is a custom light weight protocol. Is that true? If that is the case, how do I go about testing my work independently on a request/response level without necessarily having to setup another service to test my work?
I have seen the express JS library example from a request stand point. That example makes a good API gateway example. But how would you incorporate the Responder
to expressJs
?
In Readme.md,
const responder = new cote.Requester({ name: 'currency conversion responder' });
should be changed to
const responder = new cote.Responder({ name: 'currency conversion responder' });
This seems pretty interesting but it also looks like in order to give it a spin will take a no trivial effort.
It would be nice to see a script that pull up a few services.
Maybe you can use a docker-compose
file that can spin up a few docker containers with the services running inside.
It would be also nice to see the output of the monitoring tool.
There's an issue involving the use of 'dashersw/axon' as it should be just 'axon'. If that can be fixed back that would be helpful as the old naming scheme isn't in npm.
npm ERR! 404 'dashersw/axon' is not in the npm registry.
Hi, is possibile send a request to a specific node?
Hello I'm using cote with Redis to discovery.
When I use in the same machine, all works ok, but when I use 2 or more machines nothing works.
Looking pub messages inside redis I see the IP address field 127.0.0.1, I thought that should be the network address 10.14.0.44 or 10.14.1.163 at this case. What's wrong?
responder1 ip 10.14.0.44
1511981478.601845 [0 10.14.0.44:38922] "publish" "cote" "{\"event\":\"hello\",\"pid\":\"dc136bc2-c818-4e00-a215-044348875b2d\",\"iid\":\"22d137fd-4ee9-437d-9920-d74f10078da9\",\"hostName\":\"HPDEV\",\"data\":{\"isMaster\":false,\"isMasterEligible\":true,\"weight\":-0.1511976109907,\"address\":\"127.0.0.1\",\"advertisement\":{\"name\":\"Estabelecimentos:Responder\",\"key\":\"$$Estabelecimentos\",\"axon_type\":\"rep\",\"port\":8005,\"type\":\"service\"},\"id\":\"22d137fd-4ee9-437d-9920-d74f10078da9\",\"processId\":\"dc136bc2-c818-4e00-a215-044348875b2d\",\"processCommand\":\"projects/mpre_estabs\"}}"
responder2 ip 10.14.1.163
1511982019.531203 [0 10.14.1.163:37152] "publish" "cote" "{\"event\":\"hello\",\"pid\":\"1a8d3288-c8aa-4218-b683-eb36f7c1b562\",\"iid\":\"9ff0f993-d8c7-4bc1-a272-86f04b05a658\",\"hostName\":\"ecelepar16853\",\"data\":{\"isMaster\":false,\"isMasterEligible\":true,\"weight\":-0.1511981997474,\"address\":\"127.0.0.1\",\"advertisement\":{\"name\":\"Estabelecimentos:Responder\",\"key\":\"$$Estabelecimentos\",\"axon_type\":\"rep\",\"port\":8002,\"type\":\"service\"},\"id\":\"9ff0f993-d8c7-4bc1-a272-86f04b05a658\",\"processId\":\"1a8d3288-c8aa-4218-b683-eb36f7c1b562\",\"processCommand\":\"menorpreco_estabelec/index.js\"}}
I have Requester that sends many requests (for example over some interval).
I want to make Responder that initiliaze then answer one request and die. I did it in several ways, but always when i start next Responder i miss some of requests.
What proper way to do that?
I couldn't find any details on how its achieving Fault tolerance ; I think it would be help full if we had a brief description of how that is achieved.
Cote components connect to each other via IP addresses in the advertisement packages, but under certain circumstances (Docker Cloud, for example) IP addresses may be misleading since the advertisement packets are routed through different routers.
One solution to the problem is letting the daemons use the advertised hostnames. If proper DNS configuration is made, hostnames can enable communication in a multi router setup.
It's somewhat inconvenient to set the same discovery options, such as the multicast/broadcast address, for more than one component. Cote should provide a way to set such default settings.
Hi Armagan, this is not an issue, just an invitation to discuss your vision on the subject.
First of all, I want to thank your for a great project, looks very sleak, I am definitely giving it a try.
As I brushed through your repo, I noticed that you have patched the Axon library to queue up (indefinitely) the outstanding requests until the connection is up. Although viable for some use-cases, this decision looks very opinionated, hence my friendly inquiry.
Do you have any plans for advancing the mechanism with configurable failover behavior? Think response timeouts, limiting the queue length, defaults for failed requests, or may be even some state machinery like in circuit breaker.
Could you share your general opinion on this topic?
Thank you!
Does it have tests?
Is there any plans to add TypeScript support to cote ? (With types?)
Hi,
Thanks for this library!!. This library really simplifies many things.
I was wondering if this has a python lib. I have some node-js apps and some python apps. If there is no python lib available, what is the best way to communicate between these two apps? Redis? NATS? GRPC? PUB-SUB?
Certain functions such as auth or request decoration would make use of middlewares, as in, virtually everywhere (Express & co).
The following code...
responder.on('request', function(req, cb) {
// do something;
});
should still work with a middleware:
responder.on('request', middleware, function(req, cb) {
// do something if middleware allows
});
Publishers are currently blind. We could have a component that allows subscribers to selectively subscribe (and unsubscribe) to messages from publishers. That would increase network efficiency, since publishers would ignore publish commands if no listeners matched.
Sometimes multicast, broadcast and key configs are not feasible enough to specify different environments in a connected network. Then there's a deliberate need for an environment setting which should be set at require-level; so that no components need to know about (and set) this setting.
Wouldn't it be easier for the community to adopt cote.js(since it needs environments with multicast or broadcast enabled) if the service discovery was made through something like Redis? I know you have this philosophy of zero-dependency, but since you are already advising people to setup docker environments to get a overlay network, they could spin a simple redis instance to handle this stuff
Currently we have a few load-balancing scenarios. Above all these, to let developers create stateful services, cote requesters should support sticky sessions, in that, it should be configurable that certain requests could be directed to certain responders.
So that the projects that are using cote would not have to install socket.io as a dependency.
We should be able to monitor the mesh network and its connections.
Hi @dashersw,
I just need more light be shed on this part of the documentation
Note: By default, every
Requester
will connect to everyResponder
it
discovers, regardless of the request type. This means, everyResponder
should
respond to the exact same set of requests, becauseRequester
s will
load-balance requests between all connectedResponder
s regardless of
their capabilities, i.e, whether or not they can handle a given request.If you have multiple
Responder
s with varying response handlers, you will
experience lost requests. In cote, this separation between responsibilities is
called segmentation, or partitioning. If you wish to segment your requests in
groups, you can usekey
s. Check out keys for a detailed guide on how
and when to use segmentation.
Also, could you explain more on key, namespace, and environment? When to use each.
Seems to me that key and environment are interchangeable
Components try to connect to their previous peer at 5 seconds intervals for ever, if noone starts listening on that port.
I am trying to implement cote in kubernetes on IBM bluemix and am having alot of trouble getting it to work.
The bluemix kubernetes implementation has Calico plugin at network layer and Pods can definitely access each other via IP.
I have put a redis service in place and set cote to use that for discovery which works:
2017-09-13T05:23:41.920741203Z core space requester > service.online idea service space responder#7528508b-a087-414e-ade8-2a95825472e1 on 8002
however once it starts actually messaging it seems to drop back to hostnames that do not resolve inside kubernetes, e.g.
2017-09-13T05:23:46.925401243Z Error: getaddrinfo EAI_AGAIN idea:8000
2017-09-13T05:23:46.925441235Z at Object._errnoException (util.js:1041:11)
2017-09-13T05:23:46.925447197Z at errnoException (dns.js:58:15)
2017-09-13T05:23:46.925452669Z at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:95:26)
I've trawled through the implementation of node-discover and can see the hostname getting encoded in the message but I'm unclear if that hostname is used to respond.
Is there a way to insist on using IP addresses for communication rather than hostnames or am I looking down the wrong path here?
Are there any plans to draw up a spec for the various communication elements (particularly service discovery)? To perhaps allow implementations in other languages? A key aspect of micro-services for me (and I expect many others) is the ability to use different languages for different services (as appropriate). I appreciate this library is primarily aimed at Node.js but if there was at least a spec it would open up the possibility of using it in a multi-language environment.
We are rewriting cote with a modern, ES6-based approach. It's currently on the es6 branch.
I try cote for the first time. I already run the time-service.js and client.js also with the monitoring tool. But nothing happened. Do i miss something?
I know it can sounds weird (it's about multiple processes and microservice I know! :) ), but I'm trying to subscribe and publish in the same process but that does not work with this example:
var Subscriber = require('cote').Subscriber;
var randomSubscriber = new Subscriber({
name: 'randomSub',
// namespace: 'rnd',
subscribesTo: ['randomUpdate']
});
randomSubscriber.on('randomUpdate', function(req) {
console.log('notified of ', req);
});
var Publisher = require('cote').Publisher;
// Instantiate a new Publisher component.
var randomPublisher = new Publisher({
name: 'randomPub',
// namespace: 'rnd',
broadcasts: ['randomUpdate']
});
// Wait for the publisher to find an open port and listen on it.
randomPublisher.on('ready', function() {
setInterval(function() {
var val = {
val: ~~(Math.random() * 1000)
};
console.log('emitting', val);
// publish an event with arbitrary data at any time
randomPublisher.publish('randomUpdate', val);
}, 3000);
});
Is there something I need to change?
Responders should be able to work with Promises.
Hi, I'm instantiating a cote Requester and I try to use it multiple times (pointing to the same Responder in another process), but only the first Request is getting a response, the following ones don't. What could be happening ? Thanks!
Hi!
I'm trying to use cote.js in my express app.
Here my route code:
...
router.get('uuid', (req, res, next) => mainSvc.getUuid()
.then( data => res.json(data) )
.catch( next )
);
Here mainSvc code:
const cote = require('cote'),
uuidRequester = new cote.Requester({ name: 'uuid requester' });
module.exports = {
...
getUuid: ()=> {
const request = { type: 'uuid' };
return uuidRequester.send( request )
.then( data => {
console.log('Data:', data);
return data;
})
.catch( err => {
console.error('Error:', err);
return Promise.reject( err );
})
;
}
...
};
Here my service to generate uuid:
const uuid = require('uuid/v4'),
cote = require('cote'),
responder = new cote.Responder({ name: 'uuid responder' });
responder.on('uuid', (res, cb) => {
cb({ uuid: uuid() });
});
And in console I got error:
uuid requester > service.online uuid responder#47cdd6ce-ccfb-4043-bcf7-e01b54b1b626 on 8000
Error: { uuid: 'e917380b-8128-4bd7-b087-3ed136987289' }
What I did wrong and why I got Error instead Data?
Thanks
cote components are great for one-way communication (pub/sub, or req/res).
In practice, some microservices need a two-way communication channel. Since cote components work as TCP servers and clients, currently we have to implement this two-way communication via two different components, which means we would have two TCP servers for this. This is a waste.
Requesters have to wait for the ready
event before they can send a request. Otherwise the requests will just be lost.
Implement an internal queue for the requesters so that messages will be saved until the ready event occurs and sent automatically when the ready event is received.
I created 2 services on different projects and another project to call this:
const cote = require('cote')
//responder 1
const responder1 = new cote.Responder({
name: 'resp1'
});
//responder 2
const responder2 = new cote.Responder({
name: 'resp2'
});
//responder1 -> mtd1
responder1.on('mtd1', (req, cb) => {
cb('OK1!');
});
//responder2 -> mtd2
responder2.on('mtd2', (req, cb) => {
cb('OK2!');
});
//creating requester
const requester = new cote.Requester({
name: 'req1'
});
const request = {
type: 'mtd1',
val: 'test'
};
setInterval(() => {
console.log('calling...')
requester.send(request, (res) => {
console.log(res);
});
}, 1000);
When I call mtd1, only half of the answers come back.
If I delete responder2 all responses come back.
If I create responder3, only 33% of requests come back.
What am I doing wrong? Thank you.
I have a pretty simple Requester/Responder setup running on a single dev machine at the moment. There is a set of main app instances with a requester and a set of workers with a responder.
statusLogsEnabled: false
for the options it only disables logging for offline status; online status is still logged (repeatedly).Hi,
I've just created two new services based on the sample code on the website and it just doesn't work. I've tried running as separate processes on the same machine and on two separate VM's on the same network. I've opened the ports on iptables and turned off the firewall, but still no joy.
The fact that it doesn't even work on the same machine tells me something is up.
It works fine on a single instance of unbuntu. What is different with CentOS?? Any ideas?
Requesters queue messages until a responder is online...but how is this done?
Hello,
Is possible to create Express Docker Containers, exposing an API Restful that responds to
CoteJS services, but those services are in anothers Docker containers?
If I use locally, run express at 3000 port, and them run cotejs services, it works.
Expresss routes send to cotejs that responds automatically.
But and I tried use those projects using Docker as I said above, the Express router doesnt "find"
the CoteJS that are running in another Docker container.
Can you explain me for how can I make it work?
Thanks ;)
Hi there!
I currently works on a CRM project (side project)
There is 1 front app (Vue.js), which calls 1 HTTP API (Express). This API calls Services (cote.js) and render results to the front app.
Is it better to have an API between front app and Services, or use Sockend possibilities (remove the API and call Services directly from front app)? API is usefull for authentication, but add a HTTP call layer
We should be able to use the monitor for silent operation. Adding disableScreen would improve the usefulness of this component a lot.
Hello.
How can I transfer ZIP file (10MB) from Requester to Responder? Is Cote allows to do this?
Thanks.
Until Kubernetes and Docker Swarm get native support for IP broadcast/multicast, it makes sense to enable service discovery over another tool.
This is fundamentally at odds with cote's zero configuration approach; but in order to enable cote on limited environment such as public clouds, it's viable to introduce an optional alternative. This will also lead to greater adoption of cote.
I am getting empty error messages from the Requester:
responder
authService.on('getAdminToken', req => {
console.log('getAdminToken');
const adminToken = {
role: 'admin'
};
const options = {
audience: 'http://domain.com/audience',
expiresIn: '6 month'
}
return new Promise((resolve, reject) => {
const jwt = sign(adminToken, config.jwtSecret, options);
if(!jwt) reject(new Error('failed to sign token'));
resolve(jwt);
});
});
requester
requester.send({ type: 'getAdminToken' })
.then((token) => {
winston.info(token);
})
.catch(e => winston.error(e));
output
{
"level": "error",
"message": ""
}
I can't see the problem. What did I get wrong with cote?
It also sends the empty message when I do:
authService.on('getAdminToken', req => {
return new Promise((resolve, reject) => {
reject(new Error('test error message'));
});
});
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.