tsuraan / jerasure Goto Github PK
View Code? Open in Web Editor NEWGithub repo for Jerasure Library - C Implementation of Reed-Solomon coding
Home Page: http://jerasure.org/
License: Other
Github repo for Jerasure Library - C Implementation of Reed-Solomon coding
Home Page: http://jerasure.org/
License: Other
Hi,
I have had problems with the original Makefiles. So, I have fixed them and there is the new ones (I have seperated the headers files in the include directory):
# jerasure/Makefile
PREFIX=/usr/local
LIBDIR=${PREFIX}/lib
INCDIR=${PREFIX}/include
.PHONY: all clean install unistall lib
all: lib
$(MAKE) -C Examples
clean:
$(MAKE) -C src clean
$(MAKE) -C Examples clean
install: lib
@cp -P lib/libJerasure.so* ${LIBDIR}
@mkdir -p ${INCDIR}/jerasure
@cp include/* ${INCDIR}/jerasure
uninstall: lib
@rm ${LIBDIR}/libJerasure.so*
@rmdir ${INCDIR}/jerasure
lib:
$(MAKE) -C src
# jerasure/src/Makefile
CC = gcc
CFLAGS = -g -O3 -fPIC -Wall -Werror -I../include
SRCS = galois.c jerasure.c reed_sol.c cauchy.c liberation.c
OBJS = $(SRCS:.c=.o)
LIB = libJerasure.so
LIBDIR = ../lib
.PHONY: all lib ../lib compile
all: lib
lib: ../lib compile
$(CC) -shared -Wl,-soname,$(LIB) \
-o $(LIBDIR)/$(LIB).0 $(OBJS)
@ln -sf $(LIB).0 $(LIBDIR)/$(LIB)
../lib:
@mkdir -p ../lib
clean:
$(RM) $(RMFLAGS) $(LIBDIR)/$(LIB)* $(OBJS) $(wildcard *.d)
compile: $(OBJS)
%.d: %.c
@$(SHELL) -ec '$(CC) -M $(CFLAGS) $< \
| sed '\''s/\($*\)\.o[ :]*/\1.o $@ : /g'\'' > $@; \
[ -s $@ ] || rm -f $@'
-include $(SRC:.c=.d)
# jerasure/Examples/Makefile
CC = gcc
CFLAGS = -g -Wall -Werror -I../include
LDFLAGS = -L../lib
LDLIBS = -lJerasure
SRCS = $(wildcard *.c)
OBJS = $(SRCS:.c=.o)
TARGETS = $(SRCS:.c=)
all: $(TARGETS)
compile: lib $(OBJS)
clean:
$(RM) $(RMFLAGS) $(OBJS) $(TARGETS)
lib:
$(MAKE) -C ../src/
I have tried to take off all warnings too, but one of them let doubt. In jerasure.c file there is the jerasure_smart_bitmatrix_to_schedule function, and the variable bestrow could be used without be initialized, that was the warning that the compiler gave when I use -Wall compiler's flag. So, I have initialized this variable with zero, but could that variable really pass through that two "for" without be initialized?
Thanks for your attention.
I need a cross-platform reed-solomon library support linux/osx/bsd/windows/android/ios.
Thank you!
Hi. Are you still connected with the jerasure.org
website? Only it seems to have become contaminated with some weird spam referencing “online casinos” rather cleverly insinuated among the actual mentions of the software.
According to the Wayback Machine, this seems to have happened sometime in 2021.
I just discovered this excellent correction code implementation. I just started to explore it, when I discovered what appears to be a general assumption in the code that sizeof(long int)==32.
For example, in jerasure_05.c there’s a loop that fills the data blocks with psudo-random values:
srand48(0);
data = talloc(char *, k);
for (i = 0; i < k; i++) {
data[i] = talloc(char, size);
for(j = 0; j < size; j+=sizeof(long)) {
l = lrand48();
memcpy(data[i] + j, &l, sizeof(long));
}
}
I’m compiling for 64-bit processors, and sizeof(long int) is 64. When stepping through this loop, the first four bytes get random data (because lrand48() returns a value between 0 and 2^31-1) and the next four bytes are zero (because the upper word of l will always be zero). The implicit assumption seems to be that sizeof(long)==32, consist with lrand48().
I can find nothing in the documentation that explicitly mentions the assumed size of a long int, but it seems telling that in most examples w is restricted one of either 8, 16, or 32 and all size parameters are assumed to be multiples of sizeof(long).
Hi, would you be so kind as to update the description of this repository to set the upstream to be http://jerasure.org/ ? The former upstream does not contain useful information anymore.
it would be nice to update the mirror on a regular basis from http://lab.jerasure.org/jerasure/jerasure.git . I would be happy to take care of this if you'd like. When searching for jerasure in google, your repository shows first and it currently gives the impression that the library has not been updated since 2014 :-)
There are global variables in jerasure.c which are accessed without any help of atomic instructions or mutexes:
static double jerasure_total_xor_bytes = 0;
static double jerasure_total_gf_bytes = 0;
static double jerasure_total_memcpy_bytes = 0;
So if someone calls e.g. jerasure_matrix_dotprod() from two different threads in the same time, it's undefined behavior.
Maybe it makes sense to make stats collecting optional by wrapping it in #ifdef-endif?
I am trying to run the following command :
gf_mult f0f0f0f0f0f0f0f01313131313131313 1313131313131313f0f0f0f0f0f0f0f0 128h .
That is for multiplication for two 128 bit number.
But it gives me error -- Bad a
And if I decrease the 'a' and 'b' to 64 bit than it works fine.
So please suggest me what to do.
make
make -C src
make[1]: Entering directory /home/loic/software/ceph/jerasure/tsuraan/src' mkdir -p ../lib gcc -g -O3 -fPIC -c galois.c gcc -g -O3 -fPIC -c jerasure.c gcc -g -O3 -fPIC -c reed_sol.c gcc -g -O3 -fPIC -c cauchy.c gcc -g -O3 -fPIC -c liberation.c gcc -shared -Wl,-soname,libJerasure.so.0 \ -o ../lib/libJerasure.so.0 galois.o jerasure.o reed_sol.o cauchy.o liberation.o ln -sf libJerasure.so.0 ../lib/libJerasure.so mkdir -p "../include/" cp *.h ../include/ make[1]: Leaving directory
/home/loic/software/ceph/jerasure/tsuraan/src'
make -C Examples
make[1]: Entering directory /home/loic/software/ceph/jerasure/tsuraan/Examples' gcc -g -I../include -L../lib -lJerasure -o jerasure_01 jerasure_01.c /tmp/cc0CalcX.o: In function
main':
/home/loic/software/ceph/jerasure/tsuraan/Examples/jerasure_01.c:82: undefined reference to galois_single_multiply' /home/loic/software/ceph/jerasure/tsuraan/Examples/jerasure_01.c:85: undefined reference to
jerasure_print_matrix'
collect2: error: ld returned 1 exit status
make[1]: *** [jerasure_01] Error 1
make[1]: Leaving directory `/home/loic/software/ceph/jerasure/tsuraan/Examples'
make: *** [all] Error 2
Encode using blaum_roth segmentation fault.
k=10, m=4, w=16.
====my patch====
diff -u ../Jerasure-master-ori/Jerasure-master/Examples/decoder.c Examples/decoder.c
--- ../Jerasure-master-ori/Jerasure-master/Examples/decoder.c 2011-11-27 07:24:10.000000000 +0800
+++ Examples/decoder.c 2013-02-04 10:54:34.375865798 +0800
@@ -229,7 +229,7 @@
bitmatrix = liberation_coding_bitmatrix(k, w);
break;
case Blaum_Roth:
bitmatrix = blaum_roth_coding_bitmatrix(k, w);
bitmatrix = blaum_roth_coding_bitmatrix(k, w, m);
break;
case Liber8tion:
bitmatrix = liber8tion_coding_bitmatrix(k);
bitmatrix = blaum_roth_coding_bitmatrix(k, w);
bitmatrix = blaum_roth_coding_bitmatrix(k, w, m);
schedule = jerasure_smart_bitmatrix_to_schedule(k, m, w, bitmatrix);
break;
case Liber8tion:
-int *blaum_roth_coding_bitmatrix(int k, int w)
+int *blaum_roth_coding_bitmatrix(int k, int w, int m0)
{
int *matrix, i, j, index, l, m, p;
if (k > w) return NULL ;
/* Set up identity matrices */
diff -u ../Jerasure-master-ori/Jerasure-master/src/liberation.h src/liberation.h
--- ../Jerasure-master-ori/Jerasure-master/src/liberation.h 2011-11-27 07:24:10.000000000 +0800
+++ src/liberation.h 2013-02-04 10:54:03.603866097 +0800
@@ -51,6 +51,6 @@
extern int *liberation_coding_bitmatrix(int k, int w);
extern int *liber8tion_coding_bitmatrix(int k);
-extern int *blaum_roth_coding_bitmatrix(int k, int w);
+extern int *blaum_roth_coding_bitmatrix(int k, int w, int m0);
#endif
Hi all,
I am using jerasure v 2.0 and gf-complete in a project at my lab.
I have been able to build the libraries.
When I try to link the libraries to my code, I always get an linker error "undefined reference" whenever I try to use any method in the libraries.
My linker command is
g++ -L/media/sathish/6209014a-ae4c-4f58-90be-2085c0162e351/home/sathish/temp/live-proxy/live-core-lib/jerasure/src/.libs/ -L/media/sathish/6209014a-ae4c-4f58-90be-2085c0162e351/home/sathish/temp/live-proxy/live-core-lib/gf-complete/src/.libs/tmp -o "Testing123" ./abcd.o ./timeLogging.o -lgf_complete -lJerasure
Thanks,
Hi,
It would be nice to have a v2 branch, to separate development from production. The idea is to point to this branch as a submodule of the http://github.com/ceph/ceph/ repository instead of copying.
Cheers
Hi,
I appreciate your work. Would it be possible to use Jerasure in the GNU GPL project nubisave? The license in use does not seem to be compatible with GPL, so I would like to ask for your permission first.
Nubisave: (Virtual file system with CRS support and cloud storage integration)
https://github.com/joe42/nubisave/tree/master
www.nubisave.org
I currently use the implementation from jigdfs (10.1109/CICYBS.2009.4925093), but it has some weaknesses. Therefore I should like to substitute it with Jerasure library.
I appreciate your work,
joe42
w = 32 is the largest value supported by Jerasure.
How should I support larger finite fields by cauchy matrix? (such as w = 64)
Use following functions:
int *cauchy original coding matrix(k, m, w);
int *jerasure matrix to bitmatrix(k, m, w, matrix);
void jerasure schedule encode(k, m, w, schedule, data ptrs, coding ptrs, size, packetsize);
int jerasure schedule decode lazy(k, m, w bitmatrix, erasures, data ptrs, coding ptrs, size, packetsize, int
smart);
After cloning Jerasure from Mac and running the makefile supplied with Jerasure, the following linker error is received:
gcc -shared -Wl,-soname,libJerasure.so.0
-o ../lib/libJerasure.so.0 galois.o jerasure.o reed_sol.o cauchy.o liberation.o
ld: unknown option: -soname
collect2: ld returned 1 exit status
make: *** [../lib/libJerasure.so.0] Error 1
My architecture is (uname -a):
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64
Hi,
Shouldn't all the .h files under /include be installed to /usr/local/include? Right now only jerasure.h is been installed, and I think it is because of line 10 in src/Makefile.am. Not completely sure since I don't know much of automake.
Thanks,
Alfonso
after creation of erasure code:
encoder out1.dat 9 2 cauchy_good 9 1024 500000
if you go into /coding directory and delete one of the data file (ie: out1_k1.dat)
the decode actually work fine. (as shown below)
Decoding (MB/sec): 1780.2648556876
De_Total (MB/sec): 302.0309642111
The problem is when you DO NOT delete any data file, basically decode without any missing segment
doing so, it will result in core dump
Core dump happen here, and cdf=0
for (x = 0; x < cdf; x++) {
drive = row_ids[x+ddf+k]-k;
ptr = real_decoding_matrix + k_w_w*(ddf+x);
it should change the ax_ext.m4 file.
Jerasure/m4/ax_ext.m4
- ecx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 3`
- edx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 4`
+ ecx=0
+ edx=0
+ if test "$ax_cv_gcc_x86_cpuid_0x00000001" != "unknown";
+ then
+ ecx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 3`
+ edx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 4`
+ fi
kernel version
$ sudo uname -a
Linux hilton-PC 5.17.3-amd64-desktop #1 SMP PREEMPT Fri Apr 29 15:22:29 CST 2022 x86_64 GNU/Linux
There are global variables in Galois.c, and lazy initialization in multiple functions.
Line 173 in 414c96e
The above problems make multithreaded coding crash.
I know that N+2(2<=N<=16) works well.But I want to know whether Jerasure supports 9+3.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.