Demonstrate how to use JBoss Fuse as file transfer hub
For fuse beginner, there is also a fully packaged DEMO.
HIGHLIGHTS
- Integration triggered by trigger file on remote SFTP Server
- Transfer multiple files then create a new trigger file to notify target system that data file are ready
- Using camel-sftp and and local work dir to avoid overhead for big size file
- Using JBoss Fuse BOM in pom.xml, so that easy to migrate new version of fuse
- Divided customer configuration in perperties.xml, so that easy to modify
- Separate definition portion of Endpoint in CamelRoute.xml, so that simplified route definition part
- Camel Route Trigger monitors file “test.trg” is created in “inDir/trigger”. If file exists, read file content and validate that file body is same as System date in format “YYYYMMDD”. Then start another camel route TransferFile to do real transfer of data files.
- Camel Route TransferFile pulls all data files in “inDir/data” and sftp them one by one into “outDir/transfer”. Aggregator will wait all files are transferred finished, then create a new trigger file to notify target system that data has been transferred.
- src/main/resources/OSGI-INF/blueprint
- CamelRoute.xml Implementation of above 2 camel routes.
- processor.xml Declaration of processors used in camel route.(Can be merged into CamelRoute.xml)
- properties.xml Definition of properties used in camel route.(Should be updated to fit customers environment)
- src/main/java/�com.sample
- AwareEndAggregationStrategy.java Strategy class for aggregate multiple data file processing against single trigger file.
- TriggerValidationProcessor.java Processor class to implement business logic like validation the content of trigger file.
To build this project use
mvn install
To setup the sftp connection
NOTE: If your public/private rsa key pair already exists in ~/.ssh/id_rsa and your public rsa key already be copied to sftp Server, you can skip this step.
ssh-keygen -t rsa
ssh-copy-id -i .ssh/id_rsa.pub test@localhost
To run the project you can execute the following Maven goal
mvn camel:run
To deploy the project in OSGi. For example using JBoss Fuse or Apache Karaf. You can run the following command from its shell:
osgi:install -s mvn:com.sample/filetransfer/0.0.1-SNAPSHOT
--- or ---
osgi:install -s file:/target/filetransfer-0.0.1-SNAPSHOT.jar
After that, prepare test data on sftp server like below to kick start
ssh test@localhost
mkdir -p inDir/data
mkdir -p inDir/trigger
echo test1 > inDir/data/file1.dat
echo test2 > inDir/data/file2.dat
echo `date '+%Y%m%d'` > inDir/trigger/test.trg
Confirm the result
ssh test@localhost
tree -a inDir/
tree -a outDir/
Sample result before file transfer
MBP-jfeng:~ test$ tree -a inDir/
inDir/
├── data
│ ├── .done
│ ├── .error
│ ├── file1.dat
│ └── file2.dat
└── trigger
└── test.trg
Sample result after file transfer
MBP-jfeng:~ test$ tree -a inDir/
inDir/
├── data
│ ├── .done
│ │ ├── file1.dat
│ │ └── file2.dat
│ ├── .error
└── trigger
MBP-jfeng:~ test$ tree -a outDir/
outDir/
├── newTrigger
│ └── newTrigger-20160119-134537.552.trg
└── transferedData
├── file1.dat.20160119-134537.227
└── file2.dat.20160119-134537.445
NOTE: If any error occured, the original files will be moved to inDir/data/.error
For more help see the Apache Camel documentation
http://www.jboss.org/products/fuse/overview/