This is to ingest csv file into HDFS

Create a demo for streaming

• real-time client
• reporting from Kafka

resolve is as follows

1. remove handshake from while loop since for each request we only do handshake once
2. use regular expression .+

• VM setup
• Step for demo

Make the data is archived into Hadoop and/or a file storage web service before it expires from Kafka.
This will be a far away feature. Put it here as placeholder.

DF Agent is now unblocking with verx.
When 1st thread is not fininshed streaming while 2nd thread starts. There are chances to get both threads's data mixed up. As result, the Kafka will have bad data.

The resolution is to make it blocking to stream file one by one.

Stream file function requires following improvement

• While loop to watch folder changes with timeout
• Need to support file filters so that we do not stream arriving files
• Need to archive the streamed files somewhere so that we do not messed up

Test cases are needed for both client and server

• Need to update the job status in terms of metadata
• For mongo, we can update
• For Kafaka, we can send another message

• We need a better logging system, such as log4j.
• We also need to log all jobs seperately

We need a better logging system, such as log4j.
We also need to log all jobs seperately

Current, the streamed data is saved to local file in df server first, then upload to HDFS. If the file is too big, we will not see the file in HDFS. A better way is to start writing to HDFS once the block size is reached. Later, we can merge the file together.

DF Demo df-data-collector can only get updated data when US stock is open. For demo purpose, we also need data available when the market is closed. We'll consider to use spoof data and also consider to add China market as another option.

Need to refactor the server of DF as follows

• Use event bus
• Split code to different sevice verticle
• Redefine/Polish MetaData
• Redefine/Polish communication protocal
• Use HTTPS
• Add authentication

Need to polish the introduction to publish in the main site

This is new feature to batch load file to HDFS.
In this case, the file need to be arrived in DF server first. DF just move/copy the file to HDFS.

We should add options to move the files which are processed to some archive folder so that we know the files are processed.
We also need to support filter files to be process, such as files with leading _

In this case, we can collaborate with stream generator to cosume files smoothly