Giter VIP home page Giter VIP logo

Comments (22)

abzrg avatar abzrg commented on September 25, 2024 2

LOL. This is crazy. I've just noticed a strange typo. Not sure where it came from.

Ok, basically do this and you'll be fine.

case "$WM_MPLlB" in

Instead of an uppercase I between L and B, I somehow magically replaced it with an l. So basically, it evaluates to an empty string, and in this case it matches the * branch and ends up setting the FOAM_MPI to dummy... :)

Although it seems wrong, it works so far for me. Maybe I tweaked something else somewhere. But let me know If there is any problem. And sorry for your frustration.

Note that I built OpenFOAM before changing these settings, So I guess you don't need to recompile it. Just test it with those changes.

Thanks for testing it @massisenergy.

So I think the issue need to be reopen.

from openfoam-os-x.

abzrg avatar abzrg commented on September 25, 2024

I found the culprit. In etc/config.sh/mpi there is a line:

libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`

It simply returns one of the directories that are to be linked

/opt/homebrew/opt/libevent/lib

whereas there is also another directory that need to be adde:

% mpicc --showme:link
-L/opt/homebrew/Cellar/open-mpi/4.1.2/lib -L/opt/homebrew/opt/libevent/lib -lmpi

Maybe some kind of regex problem (?)...

So I added the /opt/homebrew/Cellar/open-mpi/4.1.2/lib with _foamAddLib:

~~~
libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"
~~~
_foamAddLib     $libDir
_foamAddLib     $libDir2
~~~

Thank you for maintaining this repository. I really appreciate your work.

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

Thanks for the update. Yes, the problem is that on macOS there are two folders in --showme:link, while on Linux it is single folder. Folder lookup code should be corrected.

from openfoam-os-x.

massisenergy avatar massisenergy commented on September 25, 2024

I found the culprit. In etc/config.sh/mpi there is a line:

libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`

It simply returns one of the directories that are to be linked

/opt/homebrew/opt/libevent/lib

whereas there is also another directory that need to be adde:

% mpicc --showme:link
-L/opt/homebrew/Cellar/open-mpi/4.1.2/lib -L/opt/homebrew/opt/libevent/lib -lmpi

Maybe some kind of regex problem (?)...

So I added the /opt/homebrew/Cellar/open-mpi/4.1.2/lib with _foamAddLib:

~~~
libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"
~~~
_foamAddLib     $libDir
_foamAddLib     $libDir2
~~~

Thank you for maintaining this repository. I really appreciate your work.

Thanks for posting this 👍🏽 @reverseila . I think I also have the same issue, so wanted to follow your solution. System: M1Pro with 32Gigs of RAM. Tried OpenFOAM 9 (similar issue) then found this and trying OpenFOAM 8. Part of my etc/config.sh/mpi:

~~~
 45     libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
 46     libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib" #custom addition
 47
 48     # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
 49     # include files and libraries.
 50     _foam_on_macos || export MPI_ARCH_PATH="${libDir%/*}"
 51
 52     _foamAddLib     $libDir
 53     _foamAddLib     $libDir2 #custom addition
 54
~~~

I also got the same output as you:

$ mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'
/opt/homebrew/opt/libevent/lib
$ mpicc --showme:link
-L/opt/homebrew/Cellar/open-mpi/4.1.2/lib -L/opt/homebrew/opt/libevent/lib -lmpi

After the modification, I start the build with Allwmake. But then it returns the same error:

blockMesh
dyld[75517]: symbol not found in flat namespace '_MPI_Abort'
[1]    75517 abort      blockMesh -help

Can you please help me, I think I'm missing something trivial, but can't figure out myself 🙁

from openfoam-os-x.

abzrg avatar abzrg commented on September 25, 2024

taking a look at _foamAddLib function in etc/config.sh/functions

# ~~~
# Prefix to LD_LIBRARY_PATH
_foamAddLib()
{
    while [ $# -ge 1 ]
    do
        export LD_LIBRARY_PATH=$1:$LD_LIBRARY_PATH
        shift
    done
}
# ~~~

Fixing my typo, having WM_MPLIB set to SYSTEMOPENMPI, and checking the value of LD_LIBRARY_PATH after source, I'll see

/platforms/darwin64Clang/gperftools-svn/lib:

and that comes from config.sh/gperftools line 39.

I manually added those two directories to the beginning of LD_LIBRARY_PATH but no cigar.

Another thing I did was checking that whether the script executed the function or not. So I put two echo commands before and after the _foamAddLib execution. Sourcing bashrc didn't print anything into stdout!

It seems that redirection to stdout is thrown away. I redirect echo command to a debug file to inspect. I've noticed that the SYSTEMOPENMPI branch under the case command is fully executed. However, those two directories that we passed to _foamAddLib does not appear in LD_LIBRARY_PATH.

#~~~
    libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
    libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    _foam_on_macos || export MPI_ARCH_PATH="${libDir%/*}"

    echo "before" > ~/debug
    echo "libDir: $libDir" >> ~/debug
    echo "libDir2: $libDir2" >> ~/debug
    echo "LD_LIBRARY_PATH: $LD_LIBRARY_PATH" >> ~/debug
    _foamAddLib     $libDir
    _foamAddLib     $libDir2
    echo "after" >> ~/debug
    echo "LD_LIBRARY_PATH: $LD_LIBRARY_PATH" >> ~/debug
    unset libDir
    unset libDir2
    ;;
#~~~

Now, sourcing etc/bashrc spits following into the ~/debug

before
libDir: /opt/homebrew/opt/libevent/lib
libDir2: /opt/homebrew/Cellar/open-mpi/4.1.2/lib
LD_LIBRARY_PATH: /platforms/darwin64Clang/gperftools-svn/lib:
after
LD_LIBRARY_PATH: /platforms/darwin64Clang/gperftools-svn/lib:

So it seems that _foamAddLib failed to do its job.

Okay, adding an export command after the _foamAddLib should fix it. But it doesn't. So it seems that is not the root of evil. However, I believe something happens in this branch of case command that if it didn't, everything would be fine

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

@reverseila if you take a look at etc/bashrc (line 167):

. $WM_PROJECT_DIR/etc/config.sh/functions

_foam_on_macos && . $WM_PROJECT_DIR/etc/config.sh/mac/functions

So, on macOS additional functions file is sourced (which redefines _foamAddLib function).

LD_LIBRARY_PATH is not used on macOS, variable name is DYLD_LIBRARY_PATH.

These two lines are not touched by patch:

_foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config.sh/ensight`
_foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config.sh/gperftools`

The last does LD_LIBRARY_PATH additions. Yet, as the variable is ignored, I do not worry about it.

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

I look at etc/config.sh/mpi file after fresh clone and patch application, there WM_MPLIB is written correctly with I, not l.

from openfoam-os-x.

abzrg avatar abzrg commented on September 25, 2024

I look at etc/config.sh/mpi file after fresh clone and patch application, there WM_MPLIB is written correctly with I, not l.

@mrklein Yes, as I said, it was a typo on my side.

from openfoam-os-x.

abzrg avatar abzrg commented on September 25, 2024

@mrklein Now I see.

$ type _foamAddLib
_foamAddLib is a shell function from /Users/ali/OpenFOAM/OpenFOAM-8/etc/config.sh/mac/functions

So Now, If I do:

#~~~
    libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`
    libDir2="/opt/homebrew/Cellar/open-mpi/4.1.2/lib"

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    _foam_on_macos || export MPI_ARCH_PATH="${libDir%/*}"

    echo "before" > ~/debug
    echo "libDir: $libDir" >> ~/debug
    echo "libDir2: $libDir2" >> ~/debug
    echo "DYLD_LIBRARY_PATH: $DYLD_LIBRARY_PATH" >> ~/debug
    _foamAddLib     $libDir
    _foamAddLib     $libDir2
    echo "after" >> ~/debug
    echo "DYLD_LIBRARY_PATH: $DYLD_LIBRARY_PATH" >> ~/debug
    unset libDir
    unset libDir2
    ;;
#~~~

I get following:

before
libDir: /opt/homebrew/opt/libevent/lib
libDir2: /opt/homebrew/Cellar/open-mpi/4.1.2/lib
DYLD_LIBRARY_PATH: /Users/ali/OpenFOAM/ali-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/site/8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib/dummy:
after
DYLD_LIBRARY_PATH: /opt/homebrew/Cellar/open-mpi/4.1.2/lib:/opt/homebrew/opt/libevent/lib:/Users/ali/OpenFOAM/ali-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/site/8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib/dummy:

Which is what we actually want. However that does not fix the linking problem. Still I get the following abortion error:

dyld[14516]: symbol not found in flat namespace '_MPI_Abort'
[1]    14516 abort      icoFoam

Now, let's compare it with the case it works (that magical typo). In that scenario, we don't have those _foamAddLib commands, and checking DYLD_LIBRARY_PATH,

/Users/ali/OpenFOAM/ali-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/site/8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib:/Users/ali/OpenFOAM/OpenFOAM-8/platforms/darwin64ClangDPInt32Opt/lib/dummy

We don't have those two directories here anymore. But It fixes that linking issue. So What's wrong?

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

My guess would be: if you remove OpenMPI paths from DYLD_LIBRARY_PATH, OpenFOAM simply does not load MPI libraries. So, errors, caused by MPI libraries, are gone. Yet, you can not execute solvers in parallel.

from openfoam-os-x.

abzrg avatar abzrg commented on September 25, 2024

Yes, you are right. I can confirm that. I cannot decompose mesh using decomposePar utility.

from openfoam-os-x.

kazumamatata avatar kazumamatata commented on September 25, 2024

Is there any update about the issue?
Using OpenFOAM-10 the error still persist..

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

Since, I do not have access to Apple's ARM hardware, I cannot do anything about the issue.

Could you post output of:

  • file /opt/homebrew/openmpi/lib/libmpi.dylib
  • nm /opt/homebrew/openmpi/lib/libmpi.dylib | grep _MPI_Abort

from openfoam-os-x.

kazumamatata avatar kazumamatata commented on September 25, 2024

Here are the output of your suggested commands:

$ file /opt/homebrew/openmpi/lib/libmpi.dylib
/opt/homebrew/openmpi/lib/libmpi.dylib: cannot open `/opt/homebrew/openmpi/lib/libmpi.dylib' (No such file or directory)
$ nm /opt/homebrew/openmpi/lib/libmpi.dylib | grep _MPI_Abort
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: error: /opt/homebrew/openmpi/lib/libmpi.dylib: No such file or directory

from openfoam-os-x.

kazumamatata avatar kazumamatata commented on September 25, 2024

I found the files on a different location:

$ file /opt/homebrew/Cellar/open-mpi/4.1.5/lib/libmpi.dylib
/opt/homebrew/Cellar/open-mpi/4.1.5/lib/libmpi.dylib: Mach-O 64-bit dynamically linked shared library arm64
$ nm /opt/homebrew/Cellar/open-mpi/4.1.5/lib/libmpi.dylib | grep _MPI_Abort
000000000002e298 T _MPI_Abort

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

Thank you for the reply. So, library is there and it has mentioned symbol. Could you post output of the following commands (after setting up OpenFOAM environment):

  • export | grep DYLD
  • nm -m $FOAM_LIBBIN/openmpi-system/libPstream.dylib | grep Abort

from openfoam-os-x.

kazumamatata avatar kazumamatata commented on September 25, 2024

Here are the outputs:

$ export | grep DYLD
DYLD_LIBRARY_PATH=/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system:/opt/homebrew/opt/libevent/lib:/opt/homebrew/Cellar/open-mpi/4.1.4_2/lib:/Users/kunihara/OpenFOAM/kunihara-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/site/10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/dummy
FOAM_DYLD_LIBRARY_PATH=/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system:/opt/homebrew/opt/libevent/lib:/opt/homebrew/Cellar/open-mpi/4.1.4_2/lib:/Users/kunihara/OpenFOAM/kunihara-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/site/10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib:/Users/kunihara/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/dummy
$ nm -m $FOAM_LIBBIN/openmpi-system/libPstream.dylib | grep Abort
                 (undefined) external _MPI_Abort (dynamically looked up)

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

The only difference I see from the output on my Intel macBook is (also you have different versions of open-mpi in this (4.1.4_2) and previous (4.1.5) messages):

nm -m $FOAM_LIBBIN/openmpi-system/libPstream.dylib | grep Abort
                 (undefined) external _MPI_Abort (from libmpi.40)

So, on Intel location of _MPI_Abort symbol is resolved, while on ARM it is not.

Could you post output of:

otool -L $FOAM_LIBBIN/openmpi-system/libPstream.dylib

from openfoam-os-x.

kazumamatata avatar kazumamatata commented on September 25, 2024

Sorry for my delayed answer. This is the output

$ otool -L $FOAM_LIBBIN/openmpi-system/libPstream.dylib
/Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib:
	/Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.23.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

So, basically MPI libraries are not linked in your case. Here is my output:

% otool -L openmpi-system/libPstream.dylib 
openmpi-system/libPstream.dylib:
	/Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/open-mpi/lib/libmpi.40.dylib (compatibility version 71.0.0, current version 71.5.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

Could you post command line used to link Pstream dynamic library? The simples way would be to delete $FOAM_LIBBIN/openmpi-system/libPstream.dylib and run Allwmake script. On my laptop output is:

xcrun c++ -arch x86_64 -std=c++14 -DLIB_NAME=libPstream.dylib -Ddarwin64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wno-overloaded-virtual -Wno-unused-variable -Wno-unused-local-typedef -Wno-invalid-offsetof -Wno-deprecated-register -Wno-undefined-var-template -Wno-unqualified-std-cast-call -O3  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/usr/local/Cellar/open-mpi/4.1.5/include -IlnInclude -I. -I/Volumes/OpenFOAM/OpenFOAM-10/src/OpenFOAM/lnInclude -I/Volumes/OpenFOAM/OpenFOAM-10/src/OSspecific/POSIX/lnInclude   -fPIC -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -Wl,-dylib,-undefined,dynamic_lookup,-w /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UOPwrite.o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UIPread.o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UPstream.o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/PstreamGlobals.o -L/Volumes/OpenFOAM/OpenFOAM-11/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/lib \
	    -L/usr/local/Cellar/open-mpi/4.1.5/lib -L/usr/local/opt/libevent/lib -lmpi  -o /Volumes/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib

from openfoam-os-x.

kazumamatata avatar kazumamatata commented on September 25, 2024

I deleted the file $FOAM_LIBBIN/openmpi-system/libPstream.dylib and run Allwmake as suggested. Here is the output of it.

I noticed that the option -DLIB_NAME=libPstream.dylib is not present in my case..

xcrun c++ -arch x86_64 -std=c++14 -Ddarwin64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wno-overloaded-virtual -Wno-unused-variable -Wno-unused-local-typedef -Wno-invalid-offsetof -Wno-deprecated-register -Wno-undefined-var-template -Wno-deprecated-copy -O3  -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/opt/homebrew/Cellar/open-mpi/4.1.5/include -IlnInclude -I. -I/Users/username/OpenFOAM/OpenFOAM-10/src/OpenFOAM/lnInclude -I/Users/username/OpenFOAM/OpenFOAM-10/src/OSspecific/POSIX/lnInclude   -fPIC -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -Wl,-dylib,-undefined,dynamic_lookup,-w /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UOPwrite.o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UIPread.o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UPstream.o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/PstreamGlobals.o -L/Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/lib \
	    -L/opt/homebrew/Cellar/open-mpi/4.1.5/lib -L/opt/homebrew/opt/libevent/lib -lmpi  -o /Users/username/OpenFOAM/OpenFOAM-10/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib

from openfoam-os-x.

mrklein avatar mrklein commented on September 25, 2024

Thank you for the output. You are mixing architectures and, I think, this causes the problem.

OpenMPI libraries are arm64 binaries (as you posted here: #76 (comment)), while you compile the code for x64_64 architecture (xcrun c++ -arch x86_64 in your last message). Guess, that is why OpenMPI libraries are simply ignored.

Quick solution is to edit $WM_DIR/rules/darwin64Clang/c++ and change the line:

xcrun c++ -arch x86_64 -std=c++14

to

xcrun c++ -arch arm64 -std=c++14

Do the same thing with $WM_DIR/rules/darwin64Clang/c and recompile OpenFOAM.

More complicated solution would be to create separate set of rules in $WM_DIR/rules/darwinArm64Clang and set WM_ARCH variable to darwinArm64. Guess, I will add this to the patches as soon as I get Apple ARM hardware.

from openfoam-os-x.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.