wardlican / filecrush Goto Github PK
View Code? Open in Web Editor NEWThis project forked from edwardcapriolo/filecrush
Remedy small files by combining them into larger ones.
This project forked from edwardcapriolo/filecrush
Remedy small files by combining them into larger ones.
Hadoop filecrusher. Turn many small files into fewer larger ones. Also change from text to sequence and other compression options in one pass. Crush NAME Crush - Crush small files in dfs to fewer, larger files SYNOPSIS Crush [OPTION]... <input dir> <output dir> <timestamp> DESCRIPTION Crush consumes directories containing many small files with the same key and value types and creates fewer, larger files containing the same data. Crush is gives you the control to: * Name the output files * Ignore files that are "big enough" * Limit the size of each output file * Control the output compression codec * Swap smaller files with generated large files in-place * No long-running task problem See the EXAMPLES section ARGUMENTS input dir The root of the directory tree to crush. Directories are found recursively. output dir In non-clone mode, the directory where the output files should be written. In clone mode, the directory where the original files (that were combine into larger files) should be moved. timestamp A 14 digit job timestamp used to uniquely name files. E.g. 20100221175612. Generate in a script with: date +%Y%m%d%H%M%S GLOBAL OPTIONS -?, --help Print this help message. --threshold Percent threshold relative to the dfs block size over which a file becomes eligible for crushing. Must be in the (0, 1]. Default is 0.75, which means files smaller than or equal to 75% of a dfs block will be eligible for crushing. File greater than 75% of a dfs block will be left untouched. --max-file-blocks The maximum number of dfs blocks per output file. Must be a positive integer. Small input files are associated with an output file under the assumption that input and output compression codecs have similar efficiency. Also, a directory containing a lot of data in many small files will be converted into a directory containing a fewer number of large files rather than one super-massive file. With the default value 8, 80 small files, each being 1/10th of a dfs block will be grouped into to a single output file since 8 * 1/10 = 8 dfs blocks. If there are 81 small files, each being 1/10th of a dfs block, two output files will be created. One output file contain the combined contents of 41 files and the second will contain the combined contents of the other 40. A directory of many small files will be converted into fewer number of larger files where each output file is roughly the same size. --compress Fully qualified class name of the compression codec to use when writing data. It is permissible to use "none" and "gzip" to indicate no compression and org.apache.hadoop.io.compress.GzipCodec, respectively. --clone Use clone mode. Useful for external Hive tables. In clone mode, the small files are replaced with the larger files. The small files are moved to a subdirectory of the output dir argument. The subdirectory is same as the original directory rooted at output dir. For example, assume the input dir argument and output dir argument are /user/example/input and /user/example/output, respectively. If a file was originally /user/example/input/my-dir/smallfile, then after the clone, the original file would be located in /user/example/output/user/example/input/my-dir/smallfile. --info Print information to the console about what the crush is doing. --verbose Print even more information to the console about what the crush is doing. DIRECTORY OPTIONS If specified, these options must be appear as a group. When specifying multiple groups of these options, order matters. Defaults for directory options are not used if any are specified. See the EXAMPLES section. --regex Regular expression that matches a directory name. Defaults to .+ if no directory options are specified at all. Empty directories are not required to have a matching regex. Conceptually similar to the first argument of String.replaceAll(). --replacement Replacement string used with corresponding regex to name output files. Defaults to crushed_file-${crush.timestamp}-${crush.task.num}-${crush.file.num} if no directory options are specified at all. The placeholder ${crush.timestamp} refers to the command line argument. ${crush.task.num} refers to the reducer number. ${crush.file.num} is a zero-based count of files producer by a specific reducer. The first file written by a reducer will have ${crush.file.num} = 0, the second = 1, the third = 2, etc. Conceptually similar to the second argument of String.replaceAll(). --input-format Fully qualified class name of the input format for the data in a directory. Can use the "text" and "sequence" shortcuts for org.apache.hadoop.mapred.TextInputFormat and org.apache.hadoop.mapred.SequenceFileInputFormat, respectively. Defaults to sequence if no directory options are specified. --output-format Fully qualified class name of the output format to use when writing the output file for a directory. Can use the "text" and "sequence" shortcuts for org.apache.hadoop.mapred.TextOutputFormat and org.apache.hadoop.mapred.SequenceFileOutputFormat, respectively. Defaults to sequence if no directory options are specified. EXAMPLES Say we have the following files: /user/example/work/input/ small-file1 small-file2 small-file3 small-file4 big-enough-file subdir/ small-file6 small-file7 small-file8 medium-file1 medium-file2 And we invoke the crush like this: Crush /user/example/work/input /user/example/work/output 20100221175612 Since we have not specified any of the directory options, the default regex, replacement, input-format, and output-format are used. We will get: /user/example/work/ input/ small-file1 small-file2 small-file3 small-file4 subdir/ small-file6 small-file7 small-file8 medium-file1 medium-file2 output/ crushed_file-20100221175612-0-0 big-enough-file subdir/ crushed_file-20100221175612-1-0 crushed_file-20100221175612-1-1 Where: crushed_file-20100221175612-0-0 = small-file1 + small-file2 + small-file3 + small-file4 crushed_file-20100221175612-1-0 = medium-file1 + small-file6 + small-file8 crushed_file-20100221175612-1-1 = medium-file2 + small-file7 Notice how big-enough-file was moved to the output directory. The input directory contains only the files that were combined into the larger files. By default, the output file names end with two numbers. The first number is the task number of the reducer that wrote the file. The second number is the zero-based file count of that specific reducer. So a file ending with 0-0 was produced by reducer 0 and was the first file written by that reducer. A file ending 0-1 is the second file written by that reducer. A file ending 1-0 was produced by reducer 1 and was the first file written by that reducer. In the example, notice how the directory subdir was converted into two files. If mapred.reduce.tasks permits, multiple reducers can cooperate to crush a large directory. Now a clone example. Say we invoked the crush like this: Crush --clone /user/example/work/input /user/example/clone 20100221175612 With the clone option. We would end up with: /user/example/ work/input/ crushed_file-20100221175612-0-0 big-enough-file subdir/ crushed_file-20100221175612-1-0 crushed_file-20100221175612-1-1 clone/user/example/input/ small-file1 small-file2 small-file3 small-file4 subdir/ small-file6 small-file7 small-file8 medium-file1 medium-file2 Note how the original directory structure of /user/example/input as it appeared before the crush is reproduced in /user/example/clone. The small files that were combined are moved to the clone directory while the output files and file that were "big enough" are now in the inpu directory. Clone mode is useful for crushing external Hive tables. Just make sure that there are no Hive queries running on the table because they will fail when the small files are moved to the clone directory. Now we try an example using the directory options. Say we invoke the crush like this to control the output file names: Crush \ --regex=.*/(.+) \ --replacement=$1-${crush.timestamp}-${crush.task.num}-${crush.file.num} \ --input=sequence \ --output=sequence \ /user/example/work/input /user/example/work/output 20100221175612 The --regex and --replacement arguments are similar to the arguments passed to String.replaceAll(). The regex argument matches the final part of a directory path. For /user/example/work/input, it will match input. For /user/example/work/input/subdir, it will match subdir. For matching purposes, a directory path does not have a trailing slash. The replacement argument refers to the match group by number to rename the file. The result is: /user/example/work/output/ input-20100221175612-0-0 big-enough-file subdir/ subdir-20100221175612-1-0 subdir-20100221175612-1-1 The regex and replacement options are useful for naming the output files when crushing external Hive tables that are partitioned into directories whose names have business significance. The following invocation fails: Crush \ --regex=.*/input \ --replacement=input-${crush.timestamp}-${crush.task.num}-${crush.file.num} \ --input=sequence \ --output=sequence \ /user/example/work/input /user/example/work/output 20100221175612 Since we have specified some directory options, we must ensure that all directories in hierarchy rooted at the input argument have a matching regex (since the default regex is no longer applicable). In this invocation, there is no regex argument that matches /user/example/work/input/subdir. We must change it to: Crush \ --regex=.*/input \ --replacement=input-${crush.timestamp}-${crush.task.num}-${crush.file.num} \ --input=sequence \ --output=sequence \ --regex=.*/subdir \ --replacement=as-text-${crush.timestamp}-${crush.task.num}-${crush.file.num} \ --input=sequence \ --output=text \ /user/example/work/input /user/example/work/output 20100221175612 This will yield: /user/example/work/output/ input-20100221175612-0-0 big-enough-file subdir/ as-text-20100221175612-1-0 as-text-20100221175612-1-1 Notice subdir has two files whose names differ only by the ${crush.file.num} value. Without the ${crush.file.num}, file names are not guaranteed to be unique. NOTES This program creates a temporary directories in "tmp" of the executing user's home directory in dfs. https://zenodo.org/badge/doi/10.5281/zenodo.11038.png
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.