twitter-archive / snowflake Goto Github PK

[warn] module not found: com.twitter#ostrich;2.3.3
[warn] ==== local: tried
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/ivys/ivy.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/jars/ostrich.jar
[warn] ==== local-libs: tried
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /usr/local/xichen/snowflake/libs/ostrich-2.3.3.jar
[warn] ==== localm2: tried
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ivy-2.3.3.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== Scala-Tools Maven2 Repository: tried
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#ostrich;2.3.3: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info]
[warn] :: problems summary ::
[warn] :::: WARNINGS
[warn] module not found: com.twitter#ostrich;2.3.3
[warn] ==== local: tried
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/ivys/ivy.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com.twitter/ostrich/2.3.3/jars/ostrich.jar
[warn] ==== local-libs: tried
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /usr/local/xichen/snowflake/libs/ostrich-2.3.3.jar
[warn] ==== localm2: tried
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ivy-2.3.3.xml
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] /home/xichen/.ivy2/local/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://repo1.maven.org/maven2/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ==== Scala-Tools Maven2 Repository: tried
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.pom
[warn] -- artifact com.twitter#ostrich;2.3.3!ostrich.jar:
[warn] http://scala-tools.org/repo-releases/com/twitter/ostrich/2.3.3/ostrich-2.3.3.jar
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#ostrich;2.3.3: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info]

what should I do?

Use SBT and update successed . but Failure to compile .
The error informations are as follows：
[error] Error running auto-compile-thrift-java: java.io.IOException: Cannot run program "thrift": java.io.IOException: error=2, No such file or directory
[error] Error running auto-compile-thrift-ruby: java.io.IOException: Cannot run program "thrift": java.io.IOException: error=2, No such file or directory
How to Solve This Problem?
thank u for read
The novice needs to help. Please

It's quite hard to get from github to running snowflake, especially without prior knowledge of scala, sbt and zookeeper.

on the same server, it seems that the thrift server can be configured with multiple threads (" serverOpts.workerThreads = thriftServerThreads
")

but inside the worker, sequence is a local var. so theoretically it's possible to get duplicates. no??

Build Success.

but , how to start server...THX

subj)
and a small "how to use: first simple steps"?

Is that for any particular reason that one bit is left un-used; just curious?

    //  1111111111111111111111111111111111111111111111111111111111111111
    // |-------------------41--------------------|                      |
    // |                                        |----10----|            |
    // |                                                   |-----12-----|
    //                                                     |
    //                                                     +- 1 bit not used!

Is unique ID guaranteed under concurrent execution on the same machine?

I installed thrift then SBT update successed, but something wrong with compie .

The message is

[error] /home/qianniu/Documents/nyu/test1/twitter-snowflake-0047e1f/target/gen-java/com/facebook/fb303/FacebookService.java:138: no interface expected here
[error] public static class Client extends org.apache.thrift.TServiceClient implements Iface {
[error] ^
[error] /home/qianniu/Documents/nyu/test1/twitter-snowflake-0047e1f/target/gen-java/com/facebook/fb303/FacebookService.java:834: cannot find symbol
[error] symbol : class TBaseProcessor
[error] location: package org.apache.thrift
[error] public static class Processor extends org.apache.thrift.TBaseProcessor implements org.apache.thrift.TProcessor {
.
.
.
[error] /home/qianniu/Documents/nyu/test1/twitter-snowflake-0047e1f/target/gen-java/com/facebook/fb303/FacebookService.java:751: cannot find symbol
[error] symbol : variable ___manager
[error] location: class com.facebook.fb303.FacebookService.AsyncClient
[error] ___manager.call(method_call);
[error] ^
[error] Note: Some input files use unchecked or unsafe operations.
[error] Note: Recompile with -Xlint:unchecked for details.
[error] 100 errors
[info] == compile ==

WHAT IS THE THIRD ARGUMENT.

1. is it user defined string? or not?

its repository can not access.http://artifactory.local.twitter.com/libs-releases-local.

Hi,
I've downloaded snowflake and tried to build it. I'm getting the following error:

[ERROR] Failed to execute goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile (thrift-sources) on project snowflake: Execution thrift-sources o
f goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile failed. NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile (thrift-sources)
on project snowflake: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile failed.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution thrift-sources of goal com.twitter:maven-finagle-thrift-plugin:0.0.2:compile fa
iled.
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: java.lang.NullPointerException
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:188)
at org.codehaus.plexus.util.IOUtil.copy(IOUtil.java:174)
at org.codehaus.plexus.util.FileUtils.copyStreamToFile(FileUtils.java:1059)
at com.twitter.FinagleThriftRunner.getBinary(FinagleThriftRunner.java:43)
at com.twitter.FinagleThriftRunner.compile(FinagleThriftRunner.java:54)
at com.twitter.AbstractFinagleThriftMojo.execute(AbstractFinagleThriftMojo.java:165)
at com.twitter.FinagleThriftCompileMojo.execute(FinagleThriftCompileMojo.java:22)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
... 20 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
C:\Users\zohar\Downloads\snowflake-master>

created test_id error! Error: ER_DUP_ENTRY: Duplicate entry 'f264bYg001' for key 'PRIMARY' the Duplicate id is 49，this is error msg while insert db

If the clock moves backwards the sequence is reset to 0 before raising InvalidSystemClock. When the time moves again forward to lastTimestamp then the IdWorker will generate duplicate ids for that millisecond (from 0 up to the old value of the sequence variable, depending on how many calls are made when that millisecond is revisited). The correct way is probably to do the "timestamp < lastTimestamp" check before the "lastTimestamp == timestamp" check.

sorry I created issue already close,i just create new issue.i want get help

this is my product config
new SnowflakeConfig {
serverPort = 7610
datacenterId = 0
//i config the worker id is 0 or 1 both display id duplicated ,you point out here is wrong,please help
// help me,how i do the config workerIdMap
workerIdMap = Map(0 -> InetAddress.getLocalHost.getHostName)
workerIdZkPath = "/snowflake-servers"
skipSanityChecks = true
startupSleepMs = 10000
thriftServerThreads = 10

zookeeperClientConfig = new ZookeeperClientConfig {
hostList = "cmg1.chamago.com:2181,cmg2.chamago.com:2181,cmg3.chamago.com:2181"
}

reporterConfig = new ReporterConfig {
scribeCategory = "snowflake"
scribeHost = "localhost"
scribePort = 1463
scribeSocketTimeout = 5000
flushQueueLimit = 100000
}

admin = new AdminServiceConfig {
httpPort = 9991
}

loggers = new LoggerConfig {
handlers = new FileHandlerConfig {
filename = "snowflake2.log"
level = Logger.TRACE
}
}

}

you replied me:

Your configuration is problematic. The way you construct the worker id map means that all workers will get the same id. There is therefore no guarantee of unique ids.

you point you here is problematic,because my only deploy one worker in one datacenter,,so no all worker i think ,please help me,how i do the configuration about the worker,thanks you.

Hey. Trying not to sound too much like a noob here but..

I'm going to assume this isn't incremental, from near 0, since its based on time, right? I'm interested in using this to generate IDs I can insert into Amazon DynamoDB for primary keys, and to generate a short URL. Currently my method of generating the short url (The suffix) will expand the number of characters used in the string as the numbers passed in get higher, my worry is since Snowflake is always returning 64 bit ids, will the numbers always be 10765432100123456789 this long, or will that start of smaller?

I appreciate any reply to this, I know this isn't the right place but was hopping someone might be able to explain ;-)

Thanks.

Why the class seems empty?

Is the ID value fixed length？

Hi,

I have downloaded snowflake via github, and installed sbt, however, after running 'sbt update', I get the following error:

unresolved dependency: org.jruby
unresolved dependency: org.markdown
unresolved dependency: org.freemarker

Screenshot is available on http://i40.tinypic.com/2qx9wuw.gif

I found that if I telnet to the snowflake server and send some malform data, the server will crash. I think that's unacceptable to me. Think about this: If one client not works as expected and send an invalid request for an unknown reason, then the server crashes, and all the clients will stop working, that will be a disaster.

Sending a malform request:

[liang@api: snowflake]$ telnet localhost 7609
Trying ::1...
Connected to localhost.
Escape character is '^]'.
this is a bad request

Connection closed by foreign host.

This is the crash message:

2011-05-10T20:29:11.224+0900: 63.620: [GC 63.620: [ParNew: 86017K->908K(716800K), 0.0053158 secs]63.625: [CMS: 0K->899K(102400K), 0.0401349 secs] 86017K->899K(819200K), [CMS Perm : 11118K->11094K(16384K)], 0.0455748 secs] [Times: user=0.05 sys=0.00, real=0.05 secs] 
2011-05-10T20:29:11.270+0900: 63.665: [Full GC 63.665: [CMS[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor3]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2]
: 899K->765K(102400K), 0.0365369 secs] 899K->765K(819200K), [CMS Perm : 11094K->11072K(18492K)], 0.0365862 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 
ERROR [Thread-1] (TNonblockingServer.java312) - run() exiting due to uncaught error
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
    at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:556)
    at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:424)
    at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:369)
    at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:308)

My machine:

[liang@api: snowflake]$ thrift -version
Thrift version 0.5.0
[liang@api: snowflake]$ scala -version
Scala code runner version 2.8.1.final -- Copyright 2002-2010, LAMP/EPFL
[liang@api: snowflake]$ uname -a
Darwin localhost 10.0.0 Darwin Kernel Version 10.0.0: Mon Oct 12 04:06:05 AST 2009; anappirtrvh:xnu-1456.1.26/BUILD/obj/RELEASE_I386 i386
[liang@api: snowflake]$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) Client VM (build 14.1-b02-90, mixed mode)

I'm on the updated_deps branch (though I have the same error on master) and I'm getting the following build error when I run 'mvn test'. Any clues as to what I'm doing wrong?

11:35:59-alexsilva~/Documents/snowflake/snowflake (updated_deps)$ mvn test
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building snowflake 1.0.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 34.522s
[INFO] Finished at: Fri Feb 28 11:40:13 EST 2014
[INFO] Final Memory: 20M/1963M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to parse plugin descriptor for com.twitter:maven-build-properties-plugin:0.0.2 (/Users/alexsilva/.m2/repository/com/twitter/maven-build-properties-plugin/0.0.2/maven-build-properties-plugin-0.0.2.jar): error in opening zip file -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginDescriptorParsingException