Giter VIP home page Giter VIP logo

microphone's Introduction

Microphone - Self announcing services

Microphone is a lightweight framework to run self hosting REST services using Web Api or NancyFx ontop of a Consul or ETCD cluster. Each service will start out by allocating a free port to run on, once the service is started, it will register itself in the local cluster provider.

Install from Nuget

WebApi bootstrapper

PM> Install-Package Microphone.AspNet

NancyFx bootstrapper

PM> Install-Package Microphone.Nancy

Create a service

Web Api

    class Program
    {
        static void Main(string[] args)
        {
            var options = new ConsulOptions();
            var loggerFactory = new LoggerFactory();
            var logger = loggerFactory.CreateLogger("logger");
            var provider = new ConsulProvider(loggerFactory, Options.Create(options));
            Cluster.RegisterService(new Uri($"http://localhost"), provider, "orders", "v1", logger);
            Console.ReadLine();
        }
    }

    public class OrdersController : ApiController
    {
        public string Get()
        {
            return "WebApi Service";
        }
    }

NancyFx

    class Program
    {
        private static void Main(string[] args)
        {
            Cluster.Bootstrap(new NancyProvider(), new ConsulProvider(), "customers", "v1");
            Console.ReadLine();
        }
    }

    public class CustomersService : NancyModule
    {
        public MyService()
        {
            Get["/"] = _ => "Nancy Service";
        }
    }

Cluster providers

Consul

Cluster.Bootstrap(new WebApiProvider(), new ConsulProvider(), "my-service", "v1");

The Consul provider also works together with Ebays "Fabio" load balancer https://github.com/eBay/fabio

Cluster.Bootstrap(new WebApiProvider(), new ConsulProvider(useEbayFabio:true), "my-service", "v1");

ETCD

Cluster.Bootstrap(new WebApiProvider(), new EtcdProvider(), "my-service", "v1");

Service Discovery

If one of your services needs to communicate with another service in the same Consul cluster you can query it for active instances.

//inside some WebApi/Nancy endpoint:

//automatically load balanced over service instances
var instance = await Cluster.FindServiceInstanceAsync("orders"); 

//Use Rest# or similar to call into the remote service
MakeSomeCall("/api/orders",instance.ServiceAddress, instance.ServicePort);

Running your services

Before you start your services, make sure you have an active cluster running on the host machine.

Consul Cluster

If you are new to Consul, you can bootstrap your test environment using this command:

consul agent -server -bootstrap -data-dir /tmp/consul -bind=127.0.0.1

This will give you a single server Consul cluster, this is not recommended for production usage, but it will allow you to use service discovery on your dev machine.

Diagnostics using Consul REST API

Check service health on Consul agent:

GET

http://localhost:8500/v1/agent/checks

Check all services registered on Consul agent:

GET

http://localhost:8500/v1/agent/services

ETCD Cluster

If you are using the ETCD cluster provider, make sure you have a local ETCD cluster running on your dev machine.

etcd.exe

Key/Value storage

    class Person
    {
        public string Name { get; set; }
        public int Age { get; set; }
    }

    class Program
    {
        private static void Main(string[] args)
        {
            Cluster.Bootstrap(new WebApiProvider(), new ConsulProvider(), "orders", "v1");
            Cluster.KVPutAsync("val1", new Person { Name = "Name", Age = 25 }).Wait();
            var res = Cluster.KVGetAsync<Person>("val1").Result;
            Console.ReadLine();
        }
    }

microphone's People

Contributors

alexpantyukhin avatar antoinega avatar dan-mcelroy avatar franklin89 avatar madhon avatar rogeralsing avatar saifmd4u avatar thecodejunkie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

microphone's Issues

Automate service cleanup

Consul does not unregister services with status "Critical" by itself, I have included a demo project that polls Consul and unregisters dead services.
This should be automated instead

Looking for maintainers

I would like to find a few developers that could take on the maintenance of Microphone.
I have a bit too much on my plate and I don't do a good job of giving this lib the attention it needs.

So if there is anyone interested, let me know.

Registered critical service

Using the ConsulProvider, it is correct that a service cannot be both registered and have critical status? Because whenever the status code is critical it gets unregistered. If that is the intention I would've thought if I have a service that is responding but is critical for another reason, consul should still know about it? (I'm new to consul if this question seems obvious) Thanks

Why so many dependencies? Causing problems for ILRepack

I don't understand why there are so many dependencies when I install Microphone.AspNet in a Framework 4.5.1 project. It's even pulling in things like Microsoft.CodeAnalysis.dll. I discovered this when I tried to run ILRepack on the output assembly.

I haven't followed the dnx thing closely...but if I am installing Microphone in a project that targets a framework version shouldn't that stuff be pulled from the GAC?

WARN: Method reference is used with definition return type / parameter. Indicates a likely invalid set of assemblies, consider one of the following WARN: - Remove the assembly defining Microsoft.CodeAnalysis.OptimizationLevel from the merge WARN: - Add assembly defining Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions::WithOptimizationLevel(Microsoft.CodeAnalysis.OptimizationLevel) to the merge WARN: Method reference is used with definition return type / parameter. Indicates a likely invalid set of assemblies, consider one of the following WARN: - Remove the assembly defining Microsoft.CodeAnalysis.OptimizationLevel from the merge WARN: - Add assembly defining Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions::WithOptimizationLevel(Microsoft.CodeAnalysis.OptimizationLevel) to the merge WARN: Method reference is used with definition return type / parameter. Indicates a likely invalid set of assemblies, consider one of the following WARN: - Remove the assembly defining Microsoft.CodeAnalysis.DocumentationMode from the merge WARN: - Add assembly defining System.Void Microsoft.CodeAnalysis.CSharp.CSharpParseOptions::.ctor(Microsoft.CodeAnalysis.CSharp.LanguageVersion,Microsoft.CodeAnalysis.DocumentationMode,Microsoft.CodeAnalysis.SourceCodeKind,System.Collectio ns.Generic.IEnumerable1<System.String>) to the merge
WARN: Method reference is used with definition return type / parameter. Indicates a likely invalid set of assemblies, consider one of the following
WARN: - Remove the assembly defining Microsoft.CodeAnalysis.OutputKind from the merge
WARN: - Add assembly defining System.Void Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions::.ctor(Microsoft.CodeAnalysis.OutputKind,System.Boolean,System.String,System.String,System.String,System.Collections.Generic.IEnumerable1<Sys

Mono compatability

I tried building it in mac. Look's like the framework version 4.6 isn't available in Mono. Is there a specific feature of 4.6 that is being used.

Cluster.RegisterService request canceled while waiting for connection

I am running Consul on Windows in Development mode. However, I have a problem in registering a service.

Code

    class Program
    {
        static void Main(string[] args)
        {
            var options = new ConsulOptions();
            var loggerFactory = new LoggerFactory();
            var logger = loggerFactory.CreateLogger("logger");
            var provider = new ConsulProvider(loggerFactory, Options.Create(options));

            Cluster.RegisterService(new Uri($"http://localhost:5011"), provider, "values", "v1", logger);

            var res = Microphone.Cluster.Client.GetServiceInstances("values");
        
            Console.ReadLine();
        }
    }

public class ValuesController : Controller
{
    public string Get()
    {
        return "WebApi Service";
    }
}

}

Problem:

However after running the app to register the service, and checking the consul

image

This is also the output of http://localhost:8500/v1/agent/checks:

{
    "service:values_localhost_5011": {
        "Node": "hsalman-pc",
        "CheckID": "service:values_localhost_5011",
        "Name": "Service 'values' check",
        "Status": "critical",
        "Notes": "",
        "Output": "Get http://localhost:5011/status: net/http: request canceled while waiting for connection",
        "ServiceID": "values_localhost_5011",
        "ServiceName": "values",
        "CreateIndex": 0,
        "ModifyIndex": 0
    }
}

Why its failing to register?

Last release and custom Consul host+port

Just downloaded the latest version from Nuget (testing for WebApi) and couldn't connect to my Consul server which is not on the same server using Microphone lib.
The config includes the following:

  <appSettings>
    <add key="Consul:Host" value="172.18.1.164" />
    <add key="Consul:Port" value="8300" />
  </appSettings>

The error message clearly says thats it can't connect to 127.0.0.1:8500 which is not picking up the config file.

But I have checked the code for the last release and Consul provider only accepts useEbayFabio parameter. The current code in the repository has different parameters (string consulHost = "localhost", int consulPort = 8500)

Can someone explains!

Https support for ConsulProvider

When registering a new service:

app.UseMicrophone("idSrv", "1.0", "https://idSrv.example.com:5106");

The servive will be registered with the uri: http://idSrv.example.com:5106

Went through the source code and saw that inside of ConsulProvider.cs the uri schma is ignored and it will always use http as a schema. Also for checks.

Is this on purpose?

Add configurability to all providers

All providers needs to be configurable.
e.g. consul needs to be configurable what port it listens to.
ETCD needs to be configurable on frequency on heartbeats and TTL.

Documentation

Can you update the documentation for ASPNet. I am trying to find usage for .AddMicrophoneKeyValueStore() and have not been successful. Is this to update and read from key value store in consul? how do i store my current configuration to the microphone key value store when the app boots up?

Add monitorng support

My initial thought is to build logging support that defaults to ElasticSearch logging, then incorporate this log into some of the Microphone core features.
This will allow us to see what services interact with each other and how often.

Port allocation failed, retrying.

consul_1 | ==> Starting Consul agent...
consul_1 | ==> Starting Consul agent RPC...
consul_1 | ==> Consul agent running!
consul_1 | Version: 'v0.7.5'
consul_1 | Node ID: '50357575-f0d8-4cb5-bbd6-f855776d999a'
consul_1 | Node name: '9ce5af27ecef'
consul_1 | Datacenter: 'dc1'
consul_1 | Server: true (bootstrap: false)
consul_1 | Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
consul_1 | Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
consul_1 | Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
consul_1 | Atlas:
consul_1 |
consul_1 | ==> Log data will now stream in as it occurs:
consul_1 |
consul_1 | 2017/03/29 21:48:37 [DEBUG] Using unique ID "50357575-f0d8-4cb5-bbd6-f855776d999a" from host as node ID
consul_1 | 2017/03/29 21:48:37 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
consul_1 | 2017/03/29 21:48:38 [INFO] raft: Node at 127.0.0.1:8300 [Follower] entering Follower state (Leader: "")
consul_1 | 2017/03/29 21:48:38 [INFO] serf: EventMemberJoin: 9ce5af27ecef 127.0.0.1
consul_1 | 2017/03/29 21:48:38 [INFO] consul: Adding LAN server 9ce5af27ecef (Addr: tcp/127.0.0.1:8300) (DC: dc1)
consul_1 | 2017/03/29 21:48:38 [INFO] serf: EventMemberJoin: 9ce5af27ecef.dc1 127.0.0.1
consul_1 | 2017/03/29 21:48:38 [INFO] consul: Adding WAN server 9ce5af27ecef.dc1 (Addr: tcp/127.0.0.1:8300) (DC: dc1)
consul_1 | 2017/03/29 21:48:43 [WARN] raft: Heartbeat timeout from "" reached, starting election
consul_1 | 2017/03/29 21:48:43 [INFO] raft: Node at 127.0.0.1:8300 [Candidate] entering Candidate state in term 2
consul_1 | 2017/03/29 21:48:43 [DEBUG] raft: Votes needed: 1
consul_1 | 2017/03/29 21:48:43 [DEBUG] raft: Vote granted from 127.0.0.1:8300 in term 2. Tally: 1
consul_1 | 2017/03/29 21:48:43 [INFO] raft: Election won. Tally: 1
consul_1 | 2017/03/29 21:48:43 [INFO] raft: Node at 127.0.0.1:8300 [Leader] entering Leader state
consul_1 | 2017/03/29 21:48:43 [INFO] consul: cluster leadership acquired
consul_1 | 2017/03/29 21:48:43 [DEBUG] consul: reset tombstone GC to index 3
consul_1 | 2017/03/29 21:48:43 [INFO] consul: New leader elected: 9ce5af27ecef
consul_1 | 2017/03/29 21:48:43 [INFO] consul: member '9ce5af27ecef' joined, marking health alive
consul_1 | 2017/03/29 21:48:43 [INFO] agent: Synced service 'consul'
consul_1 | 2017/03/29 21:48:43 [DEBUG] agent: Node info in sync
consul_1 | ==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=windows&signature=&version=0.7.5: dial tcp: lookup checkpoint-api.hashicorp.com: getaddrinfow: No such host is known.
usermanagementwebservice_1 | Configuration Result:
usermanagementwebservice_1 | [Success] Name UserManagementService
usermanagementwebservice_1 | [Success] DisplayName User Management Service
usermanagementwebservice_1 | [Success] Description Service endpoint to serve requests for user management
usermanagementwebservice_1 | [Success] ServiceName UserManagementService
usermanagementwebservice_1 | Topshelf v4.0.0.0, .NET Framework v4.0.30319.42000
usermanagementwebservice_1 | Port allocation failed, retrying.
usermanagementwebservice_1 | Port allocation failed, retrying.
usermanagementwebservice_1 | Port allocation failed, retrying.

Am trying to run a nancyfx self host in a docker container but it fails to register with the consul agent.

Service dies if consul is down

Hi

Great work. I'm evaluating Microphone at the moment.. I ran into this exception after stopping my local consul agent. Is this as expected?

2016-10-13 14:56:59 [Information] OK
2016-10-13 14:57:07 [Error] Crashed
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:8500
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microphone.Core.ClusterProviders.ConsulRestClient.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microphone.Core.ClusterProviders.ConsulProvider.<>c.<b__9_0>d.MoveNext()

'Cluster' does not contain a definition for 'Bootstrap'

using Microphone;
using Microphone.WebApi;

image

Trying to setup Microphone etc but the only method that exists on Cluster is RegisterService. Is the documentation outdated or something? Can't get anything set up at the moment.

Support for dotnetcore and ASP.NET 5

Are you planning on supporting dotnetcore (and ASP.NET 5)? Seems like a logical step since you can easily run those in Docker containers, where things like Consul and etcd are most useful (especially etcd because it's baked into CoreOS).

Consol Service Registration Fails With Post

Looks like consul now does not like POST requests to it's api and it seems PUT should now be used instead. Cluster.RegisterService now fails due to this as it's POSTing to the endpoint.

[ERR] http: Request POST /v1/agent/service/register, error: method POST not allowed

Custom Consul client

I'm not sure a better way to get in touch with you, but I noticed that in c95caca you replaced Consul.NET with your own Consul client. If you could leave a quick reply saying why, that would be great - I'm currently in the process of redesigning a lot of the internals of Consul.NET to allow for DNXCore50 and if there were any specific issues that you ran into that caused you to not use it, I'd like to get them fixed.

Add WCF support

I like the idea of Microphone. In our enterprise we have a lot of WCF services. Therefore I am investigating if it is possible to add WCF support to Microphone.
I would appreciate it if you would be able to help me getting started by giving me some high level info how to best approach this.

Configuration to change the status check url

How do i change the status check url to use my application map? Currently the health check for my application gets registered as http://172.18.16.72:5000/status. I need it configurable to be changed as http://172.18.16.72:5000/api/status. I keep seeing exceptions in consul trying to reach the former rather than latter.

2016/07/30 11:55:18 [WARN] agent: http request failed 'http://172.18.16.72:5000/status': Get http://172.18.16.72:5000/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2016/07/30 11:55:20 [WARN] agent: http request failed 'http://172.18.16.72:5000/status': Get http://172.18.16.72:5000/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Error Crashed

I tried to make 100000 requests call to WebApi that has been registered in Consul cluster for 2 instances that running in the same local machine. The server return successful response but suddenly in the middle of the request I received Error Crashed exception "Unable to connect to remote server"

[WARN]agent check service is now critical

I created the simple web api service as below
var options = new ConsulOptions();
options.Host = "198.160.161.XXX";
options.Port = 8500;
options.NameResolution = ConsulNameResolution.HttpApi;
options.NameResolution = ConsulNameResolution.EbayFabio

         var loggerFactory = new LoggerFactory();
        var logger = loggerFactory.CreateLogger("logger");
        var provider = new ConsulProvider(loggerFactory, Options.Create(options));          
        Cluster.RegisterService(new Uri($"http://localhost"), provider, "WebApi", "v1", logger);   

When i ran the service I am getting following warning in consul

2016/12/07 12:56:05 [INFO] agent: Synced service 'WebApi_localhost_80'
2016/12/07 12:56:05 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:06 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:07 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:08 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:09 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:10 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:11 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:12 [WARN] agent: check 'service:WebApi_localhost_80' is now critical
2016/12/07 12:56:13 [WARN] agent: check 'service:WebApi_localhost_80' is now critical

can someone please guide me on how can i make fix

Thanks,
Vijay

4.6 only

It appears Microphone was build and compiled for .NET 4.6 only. This requires updates for many servers which might be prohibitive in some environments.

If 4.6 features are not needed, it might be better for exposure to target 4.5.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.