why is hdfs throws unexpected EOF error using golang? about hdfs HOT 7 CLOSED

from hdfs.

Hi Colinmark,
I've changed this but got the following error
dial tcp 192.168.0.38:8020: getsockopt: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x467f5e]

goroutine 1 [running]:
panic(0x565260, 0xc42000c100)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
audience/vendor/github.com/colinmarc/hdfs.(*Client).getFileInfo(0x0, 0x5960f9, 0x8, 0xc420067e70, 0x45c807, 0x64eac0, 0xc420046010)
/export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/stat.go:35 +0xfe
audience/vendor/github.com/colinmarc/hdfs.(*Client).Open(0x0, 0x5960f9, 0x8, 0x3b, 0x0, 0x0)
/export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/file_reader.go:34 +0x49
main.main()
/export/bzhang/devCodeForPrd/common/src/go/src/audience/new_audience/file_translator/test2.go:11 +0xe2
It seems that the port is not opened.

I can contact hdfs using github.com/vladimirvivien/gowfs

from hdfs.

github.com/vladimirvivien/gowfs uses WebHDFS, so that makes sense. I'd guess either you're using a nonstandard port (I've seen 9000 used elsewhere) or you have a firewall blocking access.

That panic doesn't look great, though, so I'll leave this open for now.

from hdfs.

Hm, can you post more of the traceback, actually? It looks like you're calling Open on a nil Client.

from hdfs.

I made some changes to this piece of code:
`package main

import (
"fmt"
"github.com/colinmarc/hdfs"
)

func main() {
client, err := hdfs.New("192.168.0.38:9000")
fmt.Printf("client:%v err:%v\n", client, err)
file, err := client.Open("testfile")
fmt.Printf("file:%v err:%v\n", file, err)
}`

and its output:

client:<nil> err:dial tcp 192.168.0.38:9000: getsockopt: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x467c3e]

goroutine 1 [running]:
panic(0x565260, 0xc42000c100)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
audience/vendor/github.com/colinmarc/hdfs.(*Client).getFileInfo(0x0, 0x5960f9, 0x8, 0x64ea80, 0xc420048010, 0x597f0f, 0x11)
        /export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/stat.go:35 +0xfe
audience/vendor/github.com/colinmarc/hdfs.(*Client).Open(0x0, 0x5960f9, 0x8, 0x2, 0x2, 0x4c)
        /export/bzhang/devCodeForPrd/common/src/go/src/audience/vendor/github.com/colinmarc/hdfs/file_reader.go:34 +0x49
main.main()
        /export/bzhang/devCodeForPrd/common/src/go/src/audience/new_audience/file_translator/test2.go:11 +0x10b

when I use 50070 as its port to open:

client:&{0xc4200ec000 <nil>} err:<nil>
1213486160 &{{0xc4200740e0}}
unexpected EOF
file:<nil> err:open testfile: unexpected EOF

I added one line in namenode.go

func (c *NamenodeConnection) readResponse(method string, resp proto.Message) error {
    var packetLength uint32
    err := binary.Read(c.conn, binary.BigEndian, &packetLength)
    if err != nil {
        return err
    }

    packet := make([]byte, packetLength)
    fmt.Println(packetLength, c.conn)
    ...
}

and the Println outputs "1213486160 &{{0xc4200740e0}}"

So I think 50070 is ok, but the response decode failed. And I've tried 8020 and 9000, but both these 2 ports can not be connected to generate a valid client as you can see.

from hdfs.

Thank you very much. I've found the reason for the problem. I should use 127.0.0.1 as the address to open because HDFS opens its IPC port on 127.0.0.1 instead of 0.0.0.0
I use 127.0.0.1:9000 and the error is gone

from hdfs.

Great - I'm going to close this, but feel free to reopen if you find any new issues.

from hdfs.