I am currently attempting to use your Redis membership provider for a development environment on my local machine, where I would essentially start silos at
127.0.0.1:11101
and 127.0.0.1:30001
127.0.2.1:11102
and 127.0.2.1:30002
127.0.3.1:11103
and 127.0.3.1:30003
as well as a client connecting to the same Redis instance.
Now whichever silo hits Redis first creates an entry that looks like this:
127.0.0.1:6379> HGETALL dev:MyCluster
1) "S127.0.0.1:11101:277414322"
2) "{\"Entry\":{\"SiloAddress\":{\"Generation\":277414322,\"Endpoint\":{\"Address\":\"127.0.0.1\",\"Port\":11101}},\"Status\":3,\"SuspectTimes\":[],\"ProxyPort\":30001,\"HostName\":\"chillkroete\",\"SiloName\":\"Silo_69578\",\"RoleName\":\"TrackerService\",\"UpdateZone\":0,\"FaultZone\":0,\"StartTime\":\"2018-10-16T19:32:02.8708864Z\",\"IAmAliveTime\":\"2018-10-16T19:32:07.7225187Z\"},\"TableVersion\":{\"Version\":2,\"VersionEtag\":\"1\"},\"ResourceVersion\":\"1\"}"
That silo then logs information like these:
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
info: Orleans.Runtime.SiloControl[0]
GetSimpleGrainStatistics
info: Orleans.Runtime.Management.ManagementGrain[0]
GetDetailedHosts onlyActive=True
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
Interestingly, the other silos attempt to InsertRow
and then still only find the first entry:
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.2.1:11102:277414326
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414326,"Endpoint":{"Address":"127.0.2.1","Port":11102}},"Status":2,"SuspectTimes":[],"ProxyPort":30002,"HostName":"chillkroete","SiloName":"Silo_ace4d","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:06.6316528Z","IAmAliveTime":"2018-10-16T19:32:11.0376885Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.2.1:11102:277414326
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414326,"Endpoint":{"Address":"127.0.2.1","Port":11102}},"Status":2,"SuspectTimes":[],"ProxyPort":30002,"HostName":"chillkroete","SiloName":"Silo_ace4d","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:06.6316528Z","IAmAliveTime":"2018-10-16T19:32:18.0667248Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.2.1:11102:277414326
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414326,"Endpoint":{"Address":"127.0.2.1","Port":11102}},"Status":2,"SuspectTimes":[],"ProxyPort":30002,"HostName":"chillkroete","SiloName":"Silo_ace4d","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:06.6316528Z","IAmAliveTime":"2018-10-16T19:32:32.9006762Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.2.1:11102:277414326
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414326,"Endpoint":{"Address":"127.0.2.1","Port":11102}},"Status":2,"SuspectTimes":[],"ProxyPort":30002,"HostName":"chillkroete","SiloName":"Silo_ace4d","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:06.6316528Z","IAmAliveTime":"2018-10-16T19:32:39.0281563Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
and
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.3.1:11103:277414328
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414328,"Endpoint":{"Address":"127.0.3.1","Port":11103}},"Status":2,"SuspectTimes":[],"ProxyPort":30003,"HostName":"chillkroete","SiloName":"Silo_14d0e","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:08.2988266Z","IAmAliveTime":"2018-10-16T19:32:16.2606885Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.3.1:11103:277414328
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414328,"Endpoint":{"Address":"127.0.3.1","Port":11103}},"Status":2,"SuspectTimes":[],"ProxyPort":30003,"HostName":"chillkroete","SiloName":"Silo_14d0e","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:08.2988266Z","IAmAliveTime":"2018-10-16T19:32:31.1208846Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadRow: S127.0.3.1:11103:277414328
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
InsertRow: {"SiloAddress":{"Generation":277414328,"Endpoint":{"Address":"127.0.3.1","Port":11103}},"Status":2,"SuspectTimes":[],"ProxyPort":30003,"HostName":"chillkroete","SiloName":"Silo_14d0e","RoleName":"TrackerService","UpdateZone":0,"FaultZone":0,"StartTime":"2018-10-16T19:32:08.2988266Z","IAmAliveTime":"2018-10-16T19:32:34.6838954Z"}, {"Version":1,"VersionEtag":"0"}
dbug: Orleans.Clustering.Redis.RedisMembershipTable[0]
ReadAll
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S127.0.0.1:11101:277414322 SiloName=Silo_69578 Status=Active]
Moreover, stopping one silo doesn't make the others fail over. I can verify in the OrleansDashboard that exactly one silo is registered in the cluster.
I feel like I'm doing something wrong, but I can provoke the same behavior with the example you provided:
127.0.0.1:6379> HGETALL testcluster:testcluster
1) "S172.17.0.1:10025:277414741"
2) "{\"Entry\":{\"SiloAddress\":{\"Generation\":277414741,\"Endpoint\":{\"Address\":\"172.17.0.1\",\"Port\":10025}},\"Status\":3,\"SuspectTimes\":[],\"ProxyPort\":20025,\"HostName\":\"chillkroete\",\"SiloName\":\"Silo_d2d31\",\"RoleName\":\"RedisSiloHost\",\"UpdateZone\":0,\"FaultZone\":0,\"StartTime\":\"2018-10-16T19:39:01.4353724Z\",\"IAmAliveTime\":\"2018-10-16T19:39:02.9488986Z\"},\"TableVersion\":{\"Version\":2,\"VersionEtag\":\"1\"},\"ResourceVersion\":\"1\"}"
SiloHost
is saying
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 0 are Active, 0 are Dead, 1 are Joining, Version=<1, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Joining]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Runtime.MembershipService.MembershipOracleData[100634]
-ReadAll (called from BecomeActive) Membership table 1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Runtime.MembershipService.MembershipOracleData[100605]
-Finished BecomeActive.
and RedisGatewayHost
is just repeating
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Runtime.Catalog[100507]
Before collection#2: memory=9MB, #activations=0, collector=<#Activations=0, #Buckets=0, buckets=[]>.
info: Orleans.Runtime.Catalog[100508]
After collection#2: memory=9MB, #activations=0, collected 0 activations, collector=<#Activations=0, #Buckets=0, buckets=[]>, collection time=00:00:00.0255795.
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
info: Orleans.Clustering.Redis.RedisMembershipTable[0]
1 silos, 1 are Active, 0 are Dead, Version=<2, 1>. All silos: [SiloAddress=S172.17.0.1:10025:277414741 SiloName=Silo_d2d31 Status=Active]
Do you have any idea where the issue could be? I'm on .NET Core 2.1, Orleans 2.1.1 and Redis 4.0.11.