sock5 proxy support about hdfs HOT 6 CLOSED

Sorry for the late response here!

net already has an interface for this: net.Dialer. It should be an option for Client - and then used for both the namenode connection and the blockwriter connections - but we're getting to kind of an awkward place with so many options. Might be time to add a ClientOptions thingy, and deprecate the old constructors.

from hdfs.

Sorry, as soon as I hit post I realized net.Dialer is a struct, not an interface. That makes it a bit trickier, but I think we can take net/https approach and have a DialFunc func(network, address string) (net.Conn, error) option.

(net.Dialer lets you specify a timeout which is used when you call Dial; DialTimeout is just a convenience method for that.)

from hdfs.

i think we should also use rather net.DialContext since that is the recommended way now.

for the interface

we could do something like this (since New(....) is already taken):

type ClientOptions struct {
	Address     string
	Username    string
	DialFunc    func(ctx context.Context, network, addr string) (net.Conn, error)
}

func NewClient(options ClientOptions) (*Client, error)

alternatively i also like the functional options pattern that would looke something like this:

type ClientOption func(*Client) error

func Address(address string) ClientOption
func Username(username string) ClientOption
func DialFunc(dialFn func(ctx context.Context, network, addr string) (net.Conn, error)) ClientOption

func NewClient(options ..ClientOption) (*Client, error)

and it could be used like this:

func main(){
	dialer, err := proxy.SOCKS5("tcp", "localhost:8157", nil, proxy.Direct)
	if err != nil {
		panic(err)
	}


	hdfsCli, err := hdfs.NewClient(
		hdfs.Address("127.0.0.1:8020"), 
		hdfs.Username("hadoop"),
		hdfs.DialFunc(dialer.Dial),
	)

}

this does not entirely work for my usecase since proxy does not have DialContext yet , but it will eventually, and for the time being it can be wrapped

in both cases one would have to pass the DialFunc to the BlockWriter. It could be done by passing it to NamenodeConnection.

what do you think ?

from hdfs.

The former sounds right - and you're totally correct about DialContext rather than Dial. Let me take a stab at the options thing without a dialfunc first.

from hdfs.

Ok, this is going to be a bit painful. We need to thread the dialer down into:

• BlockReader and ChecksumReader (and deprecate NewBlockReader, or something)
• BlockWriter (this takes the NamenodeConnection, though, so the signature doesn't need to change there)
• NamenodeConnection (which means deprecating NewNamenodeConnection)
• and, of course, Client

from hdfs.

With #77, there now exists a ClientOptions struct, so this should now be implementable.

from hdfs.