Set of helpers designed for a Microservice architecture.
Available helpers:
- Service Loader
- AMQ publisher/consumer
- Redis cache
- Express common middlewares
- Google Cloud Monitoring (Trace, Debug, Errors)
- Logger
- Database helpers
- Database tasks
Start different services (sequentially) with graceful exit handler.
Supported features:
- Check if TCP hosts are reachable
- Start Redis cache manager
- Start AMQ publisher/consumer
- Start postgres database connection using knex (check if it's alive every 30 sec)
- Start HTTP server (express or http)
- Gracefully exit all services on exceptions
const serviceLoader = require('petitservice/lib/serviceLoader');
const logger = require('petitservice/lib/logger');
const config = require('./config');
return serviceLoader()
.ping([
config.apiUrlHost,
config.redisHost,
config.amqpHost,
])
.cache(config.redisUrl)
.amq({
amqUrl: 'amqp://guest:guest@localhost:5672',
consumerQueue: 'my-consumer-queue',
assertQueues: ['my-publisher-queue'],
consumer: (data, ack) => {
logger.info(data);
ack(); // acknowledge the message
},
})
.then(() => {
logger.info('Do something during starting...');
})
.db({
pgUrl: config.pgUrl,
pgDatabase: config.pgDatabase,
})
.express(() => require('./express_app.js'), config.httpPort)
.onExit(() => {
logger.info('Do something during exit...');
})
.done(() => {
logger.info('Everything is started!');
});
- ping(hostList, [options]): Check if hostnames are alive
hostList
(array of host to ping) host format:hostname:port
options.failureMax
(optional integer, how many attempts should we try before we exit the process, default: 5)options.frequency
(optional integer, how many milliseconds should wait before checking again hostnames, default: 30000)
- db([options]): Initiate database, checks if database is alive and destroy knex on exit
options.pgUrl
(optional string, postgres url, default: set in./lib/db/config.js
)options.pgDatabase
(optional string, database to query, default: postgres)options.failureMax
(optional integer, how many attempts should we try before we exit the process, default: 5)options.frequency
(optional integer, how many milliseconds should wait before checking again the database, default: 30000)
- cache(redisUrl): Start cache for
petitservice/lib/cache
redisUrl
(required string, redis url)
- amq(options): Connect to RabbitMQ using amqp.node, and close it when the program exit.
options
is required.options.amqUrl
(required string, amq url)options.assertQueues
(optional array, list of queue to create if they haven't been created before. if consumerQueue is set, it will be asserted as well automatically.options.consumerCb(data, ack)
(required function): handles a message to consume. It gets as parameter respectively the receiveddata
(already JSON parsed) andack
function to run when we want to acknowledge the message.options.consumerQueue
(required string): consumer queue nameoptions.consumerPrefetch
(optional integer): how many message we consume simultaneously - default: 1options.onError(err, msg)
(optional function): error handler when there's an exception on the consumer. Gets as parametererr
as error object andmsg
as raw message. Defaults a warning message.
- express(expressApp, port): Start express HTTP server, and close it when exit
expressApp
(required function that returns express app, https://github.com/expressjs/express) - We advice you to use the require inside this function.port
(integer, HTTP port. default:80
)
- then(cb): Run a function during starting
cb
(function that performs action, can return a promise as well)
- done([callback]): Add this at the end of the chain to start the service. it can take a callback function as parameter that executes when everything is loaded.
- onExit(cb): Action to perform when closing Gracefully
cb
(function that performs action, can return a promise as well)
Connect to RabbitMQ using amqp.node, Asserts queues, and consume messages from a queue.
const serviceLoader = require('petitservice/lib/serviceLoader');
const amq = require('petitservice/lib/amq');
serviceLoader()
.amq({
amqUrl: 'amqp://guest:guest@localhost:5672',
consumerQueue: 'my-consumer-queue',
assertQueues: ['my-publisher-queue'],
consumer: (data, ack) => {
logger.info(data);
ack(); // acknowledge the message
},
})
.done(() => {
amq.publish({ myKey: 'myValue' }, 'my-publisher-queue');
});
const amq = require('petitservice/lib/amq');
// Using serviceLoader
amq.start({
amqUrl: 'amqp://guest:guest@localhost:5672',
consumerQueue: 'my-consumer-queue',
assertQueues: ['my-publisher-queue'],
consumer: (data, ack) => {
logger.info(data);
ack(); // acknowledge the message
},
})
.then(() => {
// publish a message
amq.publish({ myKey: 'myValue' }, 'my-publisher-queue');
});
- start(options): Connect to RabbitMQ and assert publisher/consumer. Documentation on the options is available on serviceLoader
- publish(payload, queueName, [persistent]): Publish a payload to a queue.
- payload: (object) data to publish
- queueName: (string) where we'd like to publish the data
- persistent: (optional boolean) if we want to persist the message, default false
- close(): Close amq connection
- getChannel(): returns active channel
- getConnection(): returns active connection
Cache manager using Redis
const cache = require('petitservice/lib/cache');
const logger = require('petitservice/lib/logger');
cache.start(config.redisUrl);
const userId = 12;
const getUser = (userId) => models.getUser(userId);
// Get / Set / Delete
const cacheKey = 'bob';
cache.getValue(cacheKey)
.then((cachedValue) => {
if (!cachedValue) {
logger.info(`Setting value for ${cacheKey}`);
return cache.getValue(cacheKey, 'alice', 60);
}
logger.info(`I remember ${cachedValue}`);
})
.then((cachedValue) => {
logger.info(`Bye ${cacheKey}`);
return cache.delValue(cacheKey);
});
// Wrap a function
cache.wrap(userId, () => models.getUser(userId), 10)
.then((cacheUser) => {
logger.info(`Bonjour ${cacheUser.firstName}`);
});
// Delayed Execution
const id = 'abc';
const prm = () => {
return models.insertKeystroke(id, Math.random());
}
cache.delayedExec(id, prm, 10); // <= prm will be discarded after 10 sec
cache.delayedExec(id, prm, 10); // <= prm will be resolved after 10 sec
- start(redisUrl): Instantiate the cache so that we can call get/set data.
- getValue(key): Get value by its key
- setValue(key, value, ttl): Set a value using a key (ttl - Time to live - is in seconds)
- delValue(key): Delete a value by its key
- wrap(key, fallbackPromise, ttl, isJSON): Wrap a promise in cache.
- key: (string) identifier
- fallbackPromise: (function that returns a promise) How we get the data to read/write
- ttl: (integer) Time to live in seconds
- isJSON: (boolean - default:false) encode objects to a JSON before saving into the cache
- delayedExec(identifier, prm, delayTime): Delay an promise execution of a promise across different microservices. The promise is resolved only if not another delayedExecution has been trigged during the same timeframe (delayTime).
- identifier: (string) how we identify this execution
- prm: (function that returns a promise) the promise that we want to execute
- delayTime: (integer - in seconds) timeframe when there wasn't any delayed execution with the same identifier
Set common middlewares for an express app
const express = require('express');
const expressMiddleWare = require('petitservice/lib/expressMiddleWare');
const app = express();
expressMiddleWare.addStandard(app);
expressMiddleWare.addCompression(app);
expressMiddleWare.addLogs(app);
app.get('/', (req, res) => {
res.send('Bonjour!');
});
expressMiddleWare.addErrorHandlers(app);
- addStandard(app): Add the following middlewares:
- Disable 'x-powered-by' header
- Define 'trsut proxy' to accept forwared ip
- Body parser (json and form data)
- Set a health check endpoints '/~health' that returns 'ok'
- Remove trailing slashes on urls
- addCompression(app): Adds GZIP compression middleware
- addLogs(app): logs http request using the logger module (See section about logger)
- addErrorHandlers(app, isHTML):
- Report errors to google cloud engine if it's defined
- Endpoint to handle not found pages (if isHTML is set to true it will render the view
404
) - Endpoint to handle internal errors (if isHTML is set to true it will render the view
500
)
Monitoring using Google Stackdriver: Debug, Trace, Errors.
const gcloud = require('petitservice/lib/gcloud');
// Environment variable:
// - ENABLE_GCLOUD_TRACE: "1"
// - ENABLE_GCLOUD_ERROR: "1"
// - ENABLE_GCLOUD_DEBUG: "1"
// - GCLOUD_PROJECT: "my-project"
// - GCLOUD_STACKDRIVER_CREDENTIALS: "xxxx"
gcloud.init(process.cwd(), {
trace: {
ignoreUrls: [/^\/asserts/, /\/~*health/],
}
});
- init(projectRootDirectory, [options]): Initiate gcloud
- options: (optional object) more details below
- projectRootDirectory: (required, string) Project root directory (where package.json is located)
- reportError(): Report an error to gcloud-errors, error must be an Error object
- expressMiddleWare(): gcloud-errors express middleware
- startSpan(): gcloud-trace startSpan (see trace documentation)
- endSpan(): gcloud-trace endSpan (see trace documentation)
- runInSpan(): gcloud-trace runInSpan (see trace documentation)
- runInRootSpan(): gcloud-trace runInRootSpan (see trace documentation)
- credentials: object, gcloud credentials (default: base64decode(GCLOUD_STACKDRIVER_CREDENTIALS))
- trace: object, options to override default configuration: https://github.com/GoogleCloudPlatform/cloud-trace-nodejs/
- debug: object, options to override default configuration: https://github.com/GoogleCloudPlatform/cloud-debug-nodejs/
- error: object, options to override default configuration: https://github.com/GoogleCloudPlatform/cloud-errors-nodejs/
- GCLOUD_STACKDRIVER_CREDENTIALS: required string, base64 of the gcloud json key
- GCLOUD_PROJECT: required string: gcloud project name
- ENABLE_GCLOUD_TRACE: option binary: Enable gcloud trace
- ENABLE_GCLOUD_ERROR: option binary: Enable gcloud error reporting
- ENABLE_GCLOUD_DEBUG: option binary: Enable gcloud debug
- GCLOUD_DEBUG_LOGLEVEL: Log level for gcloud/debug (default: 1)
- GCLOUD_TRACE_LOGLEVEL: Log level for gcloud/trace (default: 1)
- GCLOUD_ERRORS_LOGLEVEL: Log level for gcloud/error (default: 1)
Log data on the console (using winston), and report errors to gcloud/error if enabled
The default LogLevels depends on the NOD_ENV:
debug
fordevelopment
envinfo
forproduction
enverror
fortest
env
// You may also set the log level using the environment variable: LOG_LEVEL: 'debug'
const logger = require('petitservice/lib/logger');
logger.debug('bonjour');
logger.info('un café et un croissant chaud');
logger.error(new Error('Something broke'));
// You can use middlewares for express
// Request logs
app.use(logger.requestLogger);
// Error logs
if (logger.gcloudErrorsMiddleWare) {
app.use(logger.gcloudErrorsMiddleWare);
}
app.use(logger.errorLogger);
- requestLogger: Express middleware to log requests
- errorLogger: Express middleware to log errors
- gcloudErrorsMiddleWare: Express middleware to report express errors to gcloud
- error
- outputError (like error, but without reporting that to gcloud)
- warn
- info
- log
- verbose
- debug
- silly
Connect to a Postgres database using knex.
Check section Service Loader for details to initiate the database
const serviceLoader = require('petitservice/lib/serviceLoader');
const logger = require('petitservice/lib/logger');
const db = require('petitservice/lib/db');
serviceLoader()
.db({
pgUrl: 'postgres://root:@localhost',
pgDatabase: 'postgres',
})
.done(() => {
const db = db.getKnexObject();
db.raw('SELECT 1;')
.then((data) => {
logger.info(data);
});
});
- init([pgUrl], [pgDatabase]): Initiate knex object in memory
- pgUrl: (option string, postgres url, default: set in
./lib/db/config.js
) - pgDatabase: (option string, postgres database, default: set in
./lib/db/config.js
) - getKnexObject(): returns knex object
- POSTGRES_PORT_5432_TCP_ADDR: Postgres hostname
- POSTGRES_PORT_5432_TCP_PORT: Postgres port
- POSTGRES_ENV_POSTGRES_USER: Postgres username
- POSTGRES_ENV_POSTGRES_PASSWORD: Postgres password
- DB_NAME: Postgres database
- run(action, database): Run a task to the database
- action: can be
createdb
,dropdb
,migrate
,seed
,init
(createdb + migrate + seed),refresh
(dropdb + init) - database: database name
- action: can be
- createdb(database): Create database
- dropdb(database): Drop database
- migrate(database): Migrate database
- seed(database): Seed database
node ./dbTasks.js createdb development
// dbTasks.js
const tasks = require('petitservice/lib/db/tasks');
const pgDatabases = {
production: 'myDb',
test: 'myDb_test',
development: 'myDb_dev',
};
const env = process.argv[process.argv.length - 1];
const action = process.argv[process.argv.length - 2];
tasks.run(action, pgDatabases[env]);
You are welcomed to fork the project and make pull requests. Or just file an issue or suggestion 😊