FileWriter doesn't renew file leases about hdfs HOT 9 CLOSED

Hi @hustchensi, thanks for the report. Do you mind submitting a test case?

from hdfs.

Going to close this since I haven't heard anything, but please feel free to reopen with more information!

from hdfs.

Hi @colinmarc. I have the same problem and I have more information.

The lease manager maintains a soft limit (1 minute) and hard limit (1 hour) for the expiration time (these limits are currently non-configurable), and all leases maintained by the lease manager abide by the same soft and hard limits. Before the soft limit expires, the client holding the lease of a file has exclusive write access to the file. If the soft limit expires and the client has not renewed the lease or closed the file (the lease of a file is released when the file is closed), another client can forcibly take over the lease. If the hard limit expires and the client has not renewed the lease, HDFS assumes that the client has quit and will automatically close the file on behalf of the client, thereby recovering the lease.

http://blog.cloudera.com/blog/2015/02/understanding-hdfs-recovery-processes-part-1/

Test case:

• write to the hdfs file longer than 1 hour
• get an error: "Invalid ack sequence number"

hdfs logs:

2017-03-14 05:42:03,455 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.internalReleaseLease: File /xxx
2017-03-14 05:42:06,467 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /xxx
...
2017-03-14 06:42:07,422 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease. Holder: go-hdfs-dPLA3zTobFBdVx3q, pendingcreates: 1], src=/xxx

The NameNode tracks the files each client has open for write. It is not necessary for a client to enumerate each file it has opened for write when renewing leases. Instead, it periodically sends a single request to the NameNode to renew all of them at once. (The request is an org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos.RenewLeaseResponseProto message, which is an RPC protocol between HDFS client and NameNode.)

from hdfs.

That's helpful! Seems like it should be pretty straightforward to automatically renew all leases with RenewLeaseResponseProto or whatever.

from hdfs.

@hustchensi I meet this problem too, How do you reslove this? thanks.

from hdfs.

@xxh2000
kockockockoc@a2359b0

from hdfs.

ok，thanks, Have you encountered this problem? #135

from hdfs.

No.
BTW, looks not related to colinmarc/hdfs issue.

from hdfs.

Fixed in #153 and 574b0b.

from hdfs.