Connect fails: domain lookups on IP fail about aerospike-client-go HOT 16 CLOSED

We are internally using go-client on Amazon and haven't encountered this problem. To help us pin down the problem, we would appreciate if you could:

1. Try connecting to the same database, from the same machine, using asinfo?
2. Try using the Java benchmark tool to connect to the same database from the same machine?
3. Provide more information about how you've configured your environment. Any particular settings?

from aerospike-client-go.

I can connect to the cluster from the same machine using asinfo and the benchmark tool. I gave the benchmark tool a fake namespace, which is – I'm guessing – why it logs errors on almost every subsequent line, but it does connect to the cluster. I can provide sample output from both if needed.

We're running a two-node cluster running Aerospike CE 3.1.3. Both hosts are in the private subnet of our VPC and are accessible from the relevant boxes with tools like asinfo and awscli. We mainly use the Java lib to connect to our Aerospike servers, and that works fine, but I'm looking into this for a couple tools I want to write.

I'm using aerospike-client-go @ 834d627, but I can pull down the latest commit on master and try again.

from aerospike-client-go.

I did some more digging into the Go source to figure out what's going on here. This is some sort of issue with the Go DNS stack. I'm have been building this program on OSX and running it on Linux with CGO_ENABLED set to 0. I built this on Linux – the same EC2 instance I'm testing with – and it works fine.

I'll do some further investigation and file a bug elsewhere if necessary. Sorry for the confusion!

from aerospike-client-go.

Interesting! Thanks for letting us know about cross compilation issues.

from aerospike-client-go.

Specifically this is an issue with the Go DNS fallback code. I'm running go 1.3.3, and the error I saw is coming from this line: https://code.google.com/p/go/source/browse/src/pkg/net/dnsclient_unix.go?name=go1.3.3#216

That code is used in cases where you apply the netgo tag when building or with CGO_ENABLED=0, which is the default for cross-compilation: https://code.google.com/p/go/source/browse/src/go/build/build.go#307

That behavior doesn't exist with the cgo code that is run if enabled, which eventually calls getaddrinfo and other C DNS functions.

Just posting this here in case anyone else stumbles on this problem.

from aerospike-client-go.

@hopkinsth is there any workaround for this issue ? we're adopting aerospike and we love cross compilation

from aerospike-client-go.

@quexer Not really, I'm sad to say!

The easiest solution is to compile your program on linux rather than whatever you're doing now. Even if you force cgo (something like CGO_ENABLED=1 go build), the linker might link against a library file that doesn't exist on linux. I can't even get my old test program to build now (with CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build) because the OSX (i.e. BSD) linker has different command line options, or, specifically, it spits this out:

ld: unknown option: --build-id=none
clang: error: linker command failed with exit code 1 (use -v to see invocation)

So, I would say build this on linux! You may need to work your program architecture around that need. Or convince the authors of this library or the Go DNS fallback code to behave differently.

You could always mess around with your local environment, but you would probably need – and someone should correct me, because I'm not 100% sure if this would work on, e.g. OSX – glibc and the GNU linker installed and functional on your machine. This might be a little easier if you're building on Windows, but really, just build in Linux.

You could set up a VM with Vagrant (or by hand, even) to take care of that. It's a pain, but it's the easiest option.

from aerospike-client-go.

@hopkinsth
We've been using mgo ( golang driver of mongodb ) for some time and it doesn't have this issue.
So I guess it can be fixed by the author of this library although I haven't dug into the code.
Maybe you can re-open this issue, let the author know.

from aerospike-client-go.

@khaf I know aerospike is optimized for Linux, but it will be great if cross compilation support could be added

from aerospike-client-go.

@quexer That mgo works with cross compilation doesn't really have any bearing on this library, though. mgo probably doesn't do DNS lookups on IP addresses or whatever aerospike-client-go is actually doing—calling some DNS function ( the net package's LookupHost, I think) with an IP address where the fallback DNS code only expects to operate on hostnames.

aerospike-client-go is relying on the fact that some DNS function is OK with being passed an IP address instead of a hostname when it's compiled with underlying C headers. The Go DNS fallback – which is what you're going to be using when compiling for another architecture – fails because it doesn't operate on an IP address... or something like that! It's been a while since I've looked into it.

So, yes, either this library or the Go DNS code needs to change if this library would work in compilation for different architectures. There's a good case for either and neither. Aerospike only runs on Linux, so it is reasonable for this client library to ultimately rely on linking against a library available on linux, even though the way it does this lookup strikes me as a little weird—why run a DNS function for an IP address when you don't need to?

On the other hand, that the Go DNS fallback seems to be inconsistent with behavior in shared libraries on at least two target platforms: OSX and Linux. Maybe it should be more consistent with those platforms.

It was easy enough for my team to build this program on Linux – either through a local VM or on a remote server – when deploying production code, so that's what we chose to do at the time. Unfortunately we're not using that service (or the aerospike DB) anymore.

I would be glad to compile more information about what the call stack looks like when the code hits this condition, but I'm several months and versions of Go removed from it.

from aerospike-client-go.

Thank you @quexer and @hopkinsth for the very educating discussion.

It is correct that the issue here that Go's LookupHost on Mac doesn't like an IP to be passed to it (or rather what it does is undefined).

We had another user in our forums who came up with a solution.

I've been experimenting with that solution, and will publish it in a branch tomorrow for experimentation.

Thoughts?

from aerospike-client-go.

@quexer and @hopkinsth: the post referenced by @khaf on our forums is here: https://discuss.aerospike.com/t/failing-to-connect-to-cluster/1006

from aerospike-client-go.

Thanks, I'm ready to try and test this fix

from aerospike-client-go.

In Master now; Please give it a try and let me know.

from aerospike-client-go.

It works in my environment: cross compile on mac, deploy to 64bit Linux.
Thanks!

from aerospike-client-go.

Cool! Thank you everyone for the productive discussion.

Closing the ticket for good :)

from aerospike-client-go.