twitter-archive / snowflake Goto Github PK
View Code? Open in Web Editor NEWSnowflake is a network service for generating unique ID numbers at high scale with some simple guarantees.
Home Page: http://twitter.com/
Snowflake is a network service for generating unique ID numbers at high scale with some simple guarantees.
Home Page: http://twitter.com/
Is unique ID guaranteed under concurrent execution on the same machine?
Build Success.
but , how to start server...THX
sorry I created issue already close,i just create new issue.i want get help
this is my product config
new SnowflakeConfig {
serverPort = 7610
datacenterId = 0
//i config the worker id is 0 or 1 both display id duplicated ,you point out here is wrong,please help
// help me,how i do the config workerIdMap
workerIdMap = Map(0 -> InetAddress.getLocalHost.getHostName)
workerIdZkPath = "/snowflake-servers"
skipSanityChecks = true
startupSleepMs = 10000
thriftServerThreads = 10
zookeeperClientConfig = new ZookeeperClientConfig {
hostList = "cmg1.chamago.com:2181,cmg2.chamago.com:2181,cmg3.chamago.com:2181"
}
reporterConfig = new ReporterConfig {
scribeCategory = "snowflake"
scribeHost = "localhost"
scribePort = 1463
scribeSocketTimeout = 5000
flushQueueLimit = 100000
}
admin = new AdminServiceConfig {
httpPort = 9991
}
loggers = new LoggerConfig {
handlers = new FileHandlerConfig {
filename = "snowflake2.log"
level = Logger.TRACE
}
}
}
you replied me:
Your configuration is problematic. The way you construct the worker id map means that all workers will get the same id. There is therefore no guarantee of unique ids.
you point you here is problematic,because my only deploy one worker in one datacenter,,so no all worker i think ,please help me,how i do the configuration about the worker,thanks you.
If the clock moves backwards the sequence is reset to 0 before raising InvalidSystemClock. When the time moves again forward to lastTimestamp then the IdWorker will generate duplicate ids for that millisecond (from 0 up to the old value of the sequence variable, depending on how many calls are made when that millisecond is revisited). The correct way is probably to do the "timestamp < lastTimestamp" check before the "lastTimestamp == timestamp" check.
[warn] module not found: com.twitter#ostrich;2.3.3
[warn] ==== local: tried
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/ivys/ivy.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/jars/ostrich.jar
[warn] ==== local-libs: tried
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /usr/local/xichen/snowflake/libs/ostrich-2.3.3.jar
[warn] ==== localm2: tried
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ivy-2.3.3.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== Scala-Tools Maven2 Repository: tried
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#ostrich;2.3.3: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info]
[warn] :: problems summary ::
[warn] :::: WARNINGS
[warn] module not found: com.twitter#ostrich;2.3.3
[warn] ==== local: tried
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/ivys/ivy.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/jars/ostrich.jar
[warn] ==== local-libs: tried
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /usr/local/xichen/snowflake/libs/ostrich-2.3.3.jar
[warn] ==== localm2: tried
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ivy-2.3.3.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== Scala-Tools Maven2 Repository: tried
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#ostrich;2.3.3: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info]
what should I do?
Hi,
I have downloaded snowflake via github, and installed sbt, however, after running 'sbt update', I get the following error:
unresolved dependency: org.jruby
unresolved dependency: org.markdown
unresolved dependency: org.freemarker
Screenshot is available on http://i40.tinypic.com/2qx9wuw.gif
Is the ID value fixed length?
Is that for any particular reason that one bit is left un-used; just curious?
// 1111111111111111111111111111111111111111111111111111111111111111
// |-------------------41--------------------| |
// | |----10----| |
// | |-----12-----|
// |
// +- 1 bit not used!
Hi,
I've downloaded snowflake and tried to build it. I'm getting the following error:
[ERROR] Failed to execute goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile (thrift-sources) on project snowflake: Execution thrift-sources o
f goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile failed. NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile (thrift-sources)
on project snowflake: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile fa
iled.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: java.lang.NullPointerException
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188)
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
at org.codehaus.plexus.util.FileUtils.copyStreamToFile(FileUtils.java:1059)
at com.twitter.FinagleThriftRunner.getBinary(FinagleThriftRunner.java:43)
at com.twitter.FinagleThriftRunner.compile(FinagleThriftRunner.java:54)
at com.twitter.AbstractFinagleThriftMojo.execute(AbstractFinagleThriftMojo.java:165)
at com.twitter.FinagleThriftCompileMojo.execute(FinagleThriftCompileMojo.java:22)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
... 20 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
C:\Users\zohar\Downloads\snowflake-master>
I found that if I telnet to the snowflake server and send some malform data, the server will crash. I think that's unacceptable to me. Think about this: If one client not works as expected and send an invalid request for an unknown reason, then the server crashes, and all the clients will stop working, that will be a disaster.
Sending a malform request:
[liang@api: snowflake]$ telnet localhost 7609
Trying ::1...
Connected to localhost.
Escape character is '^]'.
this is a bad request
Connection closed by foreign host.
This is the crash message:
2011-05-10T20:29:11.224+0900: 63.620: [GC 63.620: [ParNew: 86017K->908K(716800K), 0.0053158 secs]63.625: [CMS: 0K->899K(102400K), 0.0401349 secs] 86017K->899K(819200K), [CMS Perm : 11118K->11094K(16384K)], 0.0455748 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2011-05-10T20:29:11.270+0900: 63.665: [Full GC 63.665: [CMS[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor3]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2]
: 899K->765K(102400K), 0.0365369 secs] 899K->765K(819200K), [CMS Perm : 11094K->11072K(18492K)], 0.0365862 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
ERROR [Thread-1] (TNonblockingServer.java312) - run() exiting due to uncaught error
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:556)
at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:424)
at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:369)
at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:308)
My machine:
[liang@api: snowflake]$ thrift -version
Thrift version 0.5.0
[liang@api: snowflake]$ scala -version
Scala code runner version 2.8.1.final -- Copyright 2002-2010, LAMP/EPFL
[liang@api: snowflake]$ uname -a
Darwin localhost 10.0.0 Darwin Kernel Version 10.0.0: Mon Oct 12 04:06:05 AST 2009; anappirtrvh:xnu-1456.1.26/BUILD/obj/RELEASE_I386 i386
[liang@api: snowflake]$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) Client VM (build 14.1-b02-90, mixed mode)
its repository can not access.http://artifactory.local.twitter.com/libs-releases-local.
WHAT IS THE THIRD ARGUMENT.
subj)
and a small "how to use: first simple steps"?
Why the class seems empty?
I'm on the updated_deps branch (though I have the same error on master) and I'm getting the following build error when I run 'mvn test'. Any clues as to what I'm doing wrong?
11:35:59-alexsilva~/Documents/snowflake/snowflake (updated_deps)$ mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building snowflake 1.0.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 34.522s
[INFO] Finished at: Fri Feb 28 11:40:13 EST 2014
[INFO] Final Memory: 20M/1963M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to parse plugin descriptor for com.twitter:maven-build-properties-plugin:0.0.2 (/Users/alexsilva/.m2/repository/com/twitter/maven-build-properties-plugin/0.0.2/maven-build-properties-plugin-0.0.2.jar): error in opening zip file -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginDescriptorParsingException
I installed thrift then SBT update successed, but something wrong with compie .
[error] /home/qianniu/Documents/nyu/test1/twitter-snowflake-0047e1f/target/gen-java/com/facebook/fb303/FacebookService.java:138: no interface expected here
[error] public static class Client extends org.apache.thrift.TServiceClient implements Iface {
[error] ^
[error] /home/qianniu/Documents/nyu/test1/twitter-snowflake-0047e1f/target/gen-java/com/facebook/fb303/FacebookService.java:834: cannot find symbol
[error] symbol : class TBaseProcessor
[error] location: package org.apache.thrift
[error] public static class Processor extends org.apache.thrift.TBaseProcessor implements org.apache.thrift.TProcessor {
.
.
.
[error] /home/qianniu/Documents/nyu/test1/twitter-snowflake-0047e1f/target/gen-java/com/facebook/fb303/FacebookService.java:751: cannot find symbol
[error] symbol : variable ___manager
[error] location: class com.facebook.fb303.FacebookService.AsyncClient
[error] ___manager.call(method_call);
[error] ^
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] 100 errors
[info] == compile ==
How can I solve this problem?
thank u for read and sorry for my low English .
Hey. Trying not to sound too much like a noob here but..
I'm going to assume this isn't incremental, from near 0, since its based on time, right? I'm interested in using this to generate IDs I can insert into Amazon DynamoDB for primary keys, and to generate a short URL. Currently my method of generating the short url (The suffix) will expand the number of characters used in the string as the numbers passed in get higher, my worry is since Snowflake is always returning 64 bit ids, will the numbers always be 10765432100123456789
this long, or will that start of smaller?
I appreciate any reply to this, I know this isn't the right place but was hopping someone might be able to explain ;-)
Thanks.
on the same server, it seems that the thrift server can be configured with multiple threads (" serverOpts.workerThreads = thriftServerThreads
")
but inside the worker, sequence is a local var. so theoretically it's possible to get duplicates. no??
Use SBT and update successed . but Failure to compile .
The error informations are as follows:
[error] Error running auto-compile-thrift-java: java.io.IOException: Cannot run program "thrift": java.io.IOException: error=2, No such file or directory
[error] Error running auto-compile-thrift-ruby: java.io.IOException: Cannot run program "thrift": java.io.IOException: error=2, No such file or directory
How to Solve This Problem?
thank u for read
The novice needs to help. Please
It's quite hard to get from github to running snowflake, especially without prior knowledge of scala, sbt and zookeeper.
created test_id error! Error: ER_DUP_ENTRY: Duplicate entry 'f264bYg001' for key 'PRIMARY' the Duplicate id is 49,this is error msg while insert db
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.