finagle / finagle-postgres Goto Github PK

View Code? Open in Web Editor NEW

81.0 9.0 39.0 1.2 MB

PostgreSQL protocol support for Finagle

License: Apache License 2.0

Scala 99.27% Shell 0.73%

finagle-postgres's Introduction

Finagle Postgres

This library provides PostgreSQL database support for Finagle.

Documentation

See the GitHub Pages

Using the Postgres client

Installation

Finagle Postgres is published on Maven Central. Use the following sbt snippet to bring it as a dependency.

libraryDependencies ++= Seq(
  "io.github.finagle" %% "finagle-postgres" % "0.12.0"
)

Connecting to the DB

val client = Postgres.Client()
  .withCredentials("user", Some("password"))
  .database("dbname")
  .withSessionPool.maxSize(1) //optional; default is unbounded
  .withBinaryResults(true)
  .withBinaryParams(true)
  .withTransport.tls("host")
  .newRichClient("host:port")

Selecting with simple query

val f = client.select("select * from users") {row =>
    User(row.getString("email"), row.getString("name"))
}
logger.debug("Responded " + Await.result(f))

Integration Tests

sbt test won't run integration tests by default. In order to make it easier for development, a database environment is provided via Docker. After installing Docker and docker compose, run:

docker-compose up -d

A few environment variables need to be provided as well:

export PG_HOST_PORT=localhost:5432 PG_DBNAME=finagle_postgres_test PG_USER=postgres

With the database and variables in place, all tests will be executed. More details in IntegrationSpec

Changelog

See CHANGELOG.md

Contributors

Mairbek Khadikov (project creator)
The Factory
Vladimir Kostyukov
Jeremy Smith
Leon Maia (current maintainer)

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

finagle-postgres's People

Contributors

Stargazers

Watchers

finagle-postgres's Issues

Unable to set sslmode=require parameter on a Postgres connection

I am trying to connect to a Postgres hosted by Heroku.

https://devcenter.heroku.com/articles/heroku-postgresql#heroku-postgres-ssl

import com.twitter.finagle.Postgres
val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")

returns an error saying ssl is off as expected
com.twitter.finagle.postgres.codec.ServerError: no pg_hba.conf entry for host "82.2.102.2", user "xx", database "xx", SSL off

I am unable to set sslmode=require on the underlying connection. I have tried
val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.conditionally(true, _.withTransport.tlsWithoutValidation)
.withTransport.tls("amazonaws.com")
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")

val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.withTransport.tls("amazonaws.com")
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")

val client = Postgres.Client()
.withCredentials("xx", Some("xx"))
.database("xx")
.withSessionPool.maxSize(1)
.conditionally(true, _.withTransport.tlsWithoutValidation)
.newRichClient("xx.eu-west-1.compute.amazonaws.com:5432")

Cursors and Performance

We are running a PostgreSQL database with Infobright and very large data sets. We are running some rather involved queries against the data, which can return millions of rows, which we need to process as quickly as possible. Using the normal PostgreSQL JDBC driver is way too slow. I tried running our query through this Finagle 'driver' and the performance was even worse.

We get the best performance using the native libpg C library and cursors. It seems to me that we should be able to get pretty close to that in terms of performance from finagle-postgres... but I don't see the path towards that.

Is there "missing implementation" for cursors or streaming results from large queries? I would be willing to help in the implementation here if someone can point me in the right direction.

Tag and publish first release

Would be great to see this on Maven Central :)

Setup Autoreleases

@leonmaia @jeremyrsmith I had a great success with setting up autorelease for Finch-related projects (also finagle-oauth2). I'm happy to volunteer and do the same for finagle-postgress. The end goal is to allow anyone with push access to the repo perform releases with just changing (and committing) a new version in version.sbt.

What do you think about this?

Issues on decoding values of type Long

When using row.get[Long]("id") it returns null. row.get[Int]("id") works as expected though.

Note: I just noticed this behavior, I'll do some tests and update this Issue with my findings later.

Reconnecting database make server hang

I'm currently using finagle-postgres in a project.
I tried running my service, with PG running, do a request and get a response, turn off PG and do the same thing, then it fails fast with a PG Error. If I turn PG back on my service will not process the request, and the connection just hangs infinitely.

Here is a code example from my service

In app.scala

    val pgHost = sys.env.get("PG_HOST").get
    val pgUsername = sys.env.get("PG_USERNAME").get
    val pgPassword = sys.env.get("PG_PASSWORD").getOrElse("")
    val pgDatabase = sys.env.get("PG_DATABASE").get
    val pgClient = Client(s"$pgHost", pgUsername, Some(pgPassword), pgDatabase)

    val puppyService = new PuppyServiceImpl(pgClient)

    val api = makeService(puppyService)

In one of my endpoints

  def getHomeForPuppy(puppyService: PuppyService): Endpoint[PuppyWithHome] =
    get("topics" ? param("country") ? header("Puppy-Id")) { (country: String, puppyId: String) =>
      for {
        puppyHome <- puppyService.getHome(puppyId)
      } yield Ok(PuppyWithHome(puppyHome))
    }

And finally this is my query

  def getHome(puppyId: String): Future[Set[PuppyWithHome]] =
    pgClient.prepareAndQuery("""
        |select number, street, state from puppy_homes where
        |puppy_id = $1
      """.stripMargin, playerId) { row =>
        PuppyWithHome(
          row.get[Integer]("number"),
          row.get[String]("street"),
          row.get[String]("state")
        )
    } flatMap { rows =>
      Future.value(rows.toSet)
    } rescue {
      case err => {
        println(err.toString)
        Future.exception(PGError("puppy_homes"))
      }
    }

Override Postgres.Password stack param's string representation

Finagle clients export their configurations into a global registry if they are running from within a TwitterServer. This means finagle-postgres' password will be exported as is and anyone can query it through a JSON/HTTP endpoint.

To fix this, we'd need to override the show function for a Password stack param.

Something along the lines:

case class Password(password: Option[String]) extends AnyVal
implicit object Password extends Stack.Param[Password] {
  def default: Password = Password(None)
  override def show(p: Password): Seq[(String, () => String)]  = Nil
}

See this Finagle's commit for an inspiration.

Intermittent travis build failures

log.txt
The ValuesSpec json parsing test fails intermittently on travis. Some class of values that can be generated by circe-json breaks things somewhere along the line. Example log attached.

Client doesn't recover from DNS refreshes

Right now finagle-postgres caches the DNS resolution forever when retrieving custom types from Postgres. That causes to the client to be forever dead even if the naming resolution succedes afterward, causing to Finagle stop complaining about Name resolution failed but still causing failures when trying to query anything.

[info] 18:38:46.899 DEBUG [finagle/netty4-4] connection-1 - Sent frontend message of type: com.twitter.finagle.postgres.messages.Sync$
[info] 18:38:46.899 DEBUG [finagle/netty4-4] state machine-1 - Received event com.twitter.finagle.postgres.messages.Sync$ in state com.twitter.finagle.postgres.connection.Connected$
[info] 18:38:46.899 DEBUG [finagle/netty4-4] state machine-1 - Transitioning to state com.twitter.finagle.postgres.connection.Syncing$ and emiting result
[info] 18:38:46.900 DEBUG [finagle/netty3-2] connection-1 - Received backend message of type: com.twitter.finagle.postgres.messages.ReadyForQuery
[info] 18:38:46.900 DEBUG [finagle/netty3-2] state machine-1 - Received event com.twitter.finagle.postgres.messages.ReadyForQuery in state com.twitter.finagle.postgres.connection.Syncing$
[info] 18:38:46.900 DEBUG [finagle/netty3-2] state machine-1 - Transitioning to state com.twitter.finagle.postgres.connection.Connected$ and emiting result
[info] 18:38:46.900 DEBUG [finagle/netty3-2] connection-1 - Emitting result com.twitter.finagle.postgres.messages.ReadyForQueryResponse$
[info] 18:38:46.901 WARN  [finagle/netty3-2] c.t.o.s.m.NoBrokersAvailableExceptionMapper - Name resolution failed with No hosts are available for db, Dtab.base=[], Dtab.local=[]. Aborting.

For now as a workaround we can create the client adding withDefaultTypes() and that will make it work. But only if all the types needed by the application are described in: https://github.com/finagle/finagle-postgres/blob/master/src/main/scala/com/twitter/finagle/postgres/PostgresClient.scala#L84

Insert fails with 'SQLSTATE 22P03: invalid scale in external "numeric" value' for certain BigDecimals

This happens with the following client config:

val dbClient = Postgres
  .Client()
  .withCredentials(user = postgresUser, password = postgresPassword)
  .database(postgresDb)
  .withSessionPool
  .maxSize(1)
  .withBinaryParams(true)
  .withBinaryResults(true)
  .newRichClient(postgresHost)

(If .withBinaryParams/.withBinaryResults is set to false the inserts work)

Reproduction

Database setup:

create table foo (n numeric(6,2));

Scala:

def insertBigDecimal(bd: BigDecimal) =
  sql"""insert into foo (n) values ($bd)"""
    .exec(dbClient)
    .map(rc => s"inserted $rc rows")
    .handle{ case _ => "failed to insert" }

val a = BigDecimal(10)
val b = BigDecimal(10.0)
val c = BigDecimal("1E+1")
val d = BigDecimal("10")

val fa = insertBigDecimal(a)
val fb = insertBigDecimal(b)
val fc = insertBigDecimal(c)
val fd = insertBigDecimal(d)

println(a, Await.result(fa)) // (10,inserted 1 rows)
println(b, Await.result(fb)) // (10.0,inserted 1 rows)
println(c, Await.result(fc)) // (1E+1,failed to insert)
println(d, Await.result(fd)) // (10,inserted 1 rows)

query DSL with null param

select sum(active) as active from mobile_user where date = $date and domain_key = $domainKey and app_key = $appKey

if $domainKey or $appKey is null should I create another query like this ?

select sum(active) as active from mobile_user where date = $date

I think this idea is bed?can someboy help me ?

Arrays support

It would be nice if your great library would support PostgreSQL arrays

Releases aren't being tagged

The last few releases don't have associated git tags. This makes it a lot harder to keep track of what releases there are and what code is in each one.

queryDSL `as` method

Hi:
I meet a problem with use queryDSL spec

val resultFuture = sql"select count(*) as toal from tableA".as[Count].run(postgresClient)

the Count be defined like this below

case class Count(total:Int =0)

and then 

   val resultSeq = Await.result(resultFuture, Duration(10, TimeUnit.SECONDS))
    val total = if (resultSeq.isEmpty) 0 else resultSeq.head.total

but not found in case class Count ,the total is always 0

Release version 0.4.3, please

Requests fail when executed transactionally in parallel

Hi, we're using finagle-postgres for our project, it's great, but we've faced an unpleasant bug.

For example we have table:

CREATE TABLE dummy (
    id INT NOT NULL,
    PRIMARY KEY (id)
);

And we want to insert several values in transactional manner:

  it should "work" in {
    val ids = Seq(1, 2)
    val transaction = dbClient.inTransaction { client =>
      Future.collect {
        ids.map { id =>
          client.prepareAndExecute("INSERT INTO dummy (id) VALUES ($1)", id)
        }
      }
    }
    Await.result(transaction)
  }

But instead we're getting an error:
com.twitter.finagle.postgres.codec.ServerError: SQLSTATE 23505: duplicate key value (id)=(2) violates unique constraint "primary"

Enabling LoggingFilter on the client gives such logs:

Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Query(SELECT * FROM pg_catalog.pg_tables WHERE tableowner = 'root')
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Query(BEGIN)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Parse(,INSERT INTO dummy (id) VALUES ($1),ArraySeq(23))
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Parse(,INSERT INTO dummy (id) VALUES ($1),ArraySeq(23))
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: BIND[0000000131]
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: BIND[0000000132]
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Describe(true,)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Describe(true,)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter log
INFO: Execute(,0)
Jan 21, 2021 1:31:14 PM com.twitter.finagle.filter.LoggingFilter logException
INFO: VERY BAD
com.twitter.finagle.postgres.codec.ServerError: SQLSTATE 23505: duplicate key value (id)=(2) violates unique constraint "primary"

So we can see that sending parameters is different, but somehow it tries to insert the same value twice.

It works fine if we use Future.traverseSequentially instead of Future.collect, but it's not what we want.

Same error pops when we use transactional parallel SELECT requests with client.queryAs.

We use CockroachDB but it behave the same on PostgreSQL. For testing we use "com.dimafeng:testcontainers-scala" but same thing reproduces with instance in docker-compose.
Client config:

  val client: PostgresClient = Postgres.Client()
    .withCredentials(config.user, config.password)
    .database(config.database)
    .filtered(loggingFilter)
    .newRichClient(config.host)

.withBinaryParams doesn't help.

`INSERT ... RETURNING ...` kills the connection

I'm trying to get the simplest query - inserting a row and getting auto-generated primary - to work with no success. Not only it doesn't work, it marks connection as dead. Here's a code to reproduce (Scala 2.11, finagle-postgres 0.4.0)

object Hello {
  def main(args: Array[String]): Unit = {
    val client = Postgres.Client()
      .withCredentials("pgsql", Some("pgsql"))
      .database("app_stats_development")
      .withSessionPool.maxSize(10)
      .newRichClient("localhost:5432")

    val create = client.execute("CREATE TABLE IF NOT EXISTS items (id SERIAL NOT NULL PRIMARY KEY, name VARCHAR(255))")
    Await.result(create)
    println("Table created")
    val insert = client.select("INSERT INTO items (name) VALUES ('item_name') RETURNING id") { row =>
      println(row.get[Int]("id"))
    }
    Await.result(insert)
    println("After insert")
  }
}

And here's log

Nov 26, 2016 4:00:04 AM com.twitter.finagle.Init$$anonfun$4 apply$mcV$sp
INFO: Finagle version 6.39.0 (rev=f49b26aa1a89cbaec5aa7ac956ae7f379b4a5d97) built at 20161011-165646
Table created
Nov 26, 2016 4:00:05 AM com.twitter.finagle.service.FailureAccrualFactory$$anon$2$$anon$1 didMarkDead
INFO: FailureAccrualFactory marking connection to "postgres" as dead. Remote Address: Inet(localhost/127.0.0.1:5432,Map())

Please cut a new release

Hey! I know that the administrative tasks of being an open-source maintainer can be a severe drain on one's personal energy level and that this ticket is probably going to induce at least a slight internal groan but... It would be great to get a new release. The issue in #81 is severe enough for my use-case that I've been running a build from the source of master in order to get the fix but it would be great to be able to use a proper upstream binary release. I do also think that pretty much anyone using finagle-postgres in production ought to get that fix because it can cause very difficult-to-diagnose query failures.

Finagle is going to be dropping finagle-netty3

The Finagle library is finishing up its transition to Netty4, and as such it's going to be dropping Netty3 support in its entirety at some point. However, it looks like finagle-postgres is heavily dependent on the finagle-netty3 package.

There are at least two roads forward.

finagle-postgres can migrate to finagle-netty4.
The finagle-netty3 package could be moved to the finagle org.

These are not mutually exclusive, but either will address the issue at hand. Do you have a preference? There may be enough ecosystem projects to justify the second route regardless but we haven't don't an extensive survey.

Transaction support

Currently this client have limited Transaction support. It would be great if some could start a transaction and then commit it or roll it back, correctly.
I know there were a PR at #7 but somehow it got cancled.

PostgreSQL Date field returning null value

I am trying to retrieve a date field from a postgresql table but I couldn't get the value.

My code:
val ts = row.get[LocalDateTime]("ts")

If I change to:
val ts = row.get[String]("ts")

It had strange characters.
I see the date value in postgres table and the record was inserted with finagle-postgres library.

Support for where conditions with tuples of values

Postgres supports syntax like this: ... WHERE (a, b) in ((1, 2), (3, 4))
It would be nice to pass Seq of Tuple (or even a case class) as a Param in the query, currently it seems to only work with String.

A test for something like this (for sql"" syntax) might look something like this:

"Seq of tuple params for IN operator" in {
  val p1 = Seq(("foo", "bar"), ("three", "four"))
  expectQuery("SELECT * FROM foo WHERE (a, b) IN (($1, $2), ($3, $4))", "foo", "bar", "three", "four") {
    sql"SELECT * FROM foo WHERE a IN ($p1)"
  }
}

java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param;

I meet a problem with finagle-postgres lib
finatra-http:18.10.0
finagle-core: 18.10.0
finagle-postgres:18.10.0

class P2PAnalysisServiceTest extends FeatureTest {

  override val server = new EmbeddedHttpServer(new P2PService)

  val domainKey = "1"
  val appKey = "1"
  val date = "20180103"
  val start = "20180810"
  val end = "20180817"
  val hour = 14

  test("P2PAnalysisService#perform feature") {
    server.httpGet(
      path = s"service/play/count/day/${domainKey}/${appKey}?date=${date}",
      andExpect = Ok
    )
  }
}

system stack

An exception or error caused a run to abort: java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param; 
java.lang.Exception: java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param;
	at com.twitter.inject.server.EmbeddedTwitterServer.$anonfun$runNonExitingMain$1(EmbeddedTwitterServer.scala:434)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at com.twitter.util.Try$.apply(Try.scala:26)
	at com.twitter.util.ExecutorServiceFuturePool$$anon$4.run(FuturePool.scala:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: com.twitter.finagle.client.Transporter$SocksProxy$.param()Lcom/twitter/finagle/Stack$Param;

and this code is throw the stack

object PostgresModule extends TwitterModule with Logging {

  val postgresConn: Flag[String] = flag("postgres",
    "jdbc:postgresql://xxx:xxx/xxxx?stringtype=unspecified",
    "the connection url string for postgres")

  @Singleton
  @Provides
  def postgresClient: PostgresClient = {
    val url = postgresConn().substring(5)
    val uri = URI.create(url)
    val host = uri.getHost
    val port = uri.getPort

    val client = Postgres.Client().withCredentials("postgres", Some("postgres"))
      .database("xxx").withSessionPool.maxSize(5)
      .withBinaryResults(true).withBinaryParams(true).newRichClient(s"$host:$port")

    client
  }

  override def singletonShutdown(injector: Injector): Unit = {
    super.singletonShutdown(injector)
    debug("====singletonShutdown====")
    injector.instance[PostgresClient].close()
  }

  override def singletonStartup(injector: Injector): Unit = {
    debug("=====singletonStartup=====")
    super.singletonStartup(injector)
  }
}

UPDATE...RETURNING causes exception if no results

You must call client.select() in order to get results out of an UPDATE...RETURNING query. But, a CommandComplete tag for Update(), in the absence of a row buffer, leads to an OK() response. Since the select() method treats this as an error condition, you must be very sure that there will be at least one row updated, or the UPDATE...RETURNING query will lead to a semantically meaningless exception being thrown.

select()'s behavior could be changed to silently drop the OK() message, since it is not typically indicative of an error. No information would be lost, because in the case where an OK() message was received by select()'s callback, no rows were updated so the parameter from the Update() tag is always zero. Instead, the OK(_) case could just map an empty row set:

case OK(_) => Seq.empty[Row].map(f)

I understand why the correlating error is thrown from exec(), because in that case rows were received and there is no mapper supplied to map them. That is more easily considered an error condition.

Reproduce:

for {
_ <- client.exec("CREATE TABLE update_returning_test (foo integer)")
r <- client.query("UPDATE update_returning_test SET foo = $1 returning foo", 5)(row => row.get[Int]("foo"))
} yield r

DELETE...RETURNING causes an error

There is no state transition from AggregateRowsWithoutFields when a Delete() tag is received (though it exists for Update()). This makes DELETE...RETURNING queries impossible.

Documented groupId is wrong for 0.2.0

The README has

libraryDependencies ++= Seq(
  "com.github.finagle" %% "finagle-postgres" % "0.2.0"
)

But that's the wrong groupId for version 0.2.0 on Maven Central, which, unlike 0.1.0, has the groupId "io.github.finagle." See http://search.maven.org/#search%7Cga%7C1%7Cfinagle-postgres

(Btw package namespaces starting with io. shadow scala.io => io imports from Predef, which is kind of annoying, but perhaps that's a lost cause, because netty.)

Unknown command complete response tag TRUNCATE TABLE

When issuing a TRUNCATE TABLE command the response from the server is not properly parsed and an exception is thrown:

java.lang.IllegalStateException: Unknown command complete response tag TRUNCATE TABLE
	at com.twitter.finagle.postgres.messages.BackendMessageParser.parseTag(BackendMessageParser.scala:245)
	at com.twitter.finagle.postgres.messages.BackendMessageParser.parseC(BackendMessageParser.scala:218)
	at com.twitter.finagle.postgres.messages.BackendMessageParser.parse(BackendMessageParser.scala:30)

Observed with version 0.10.0

PgCodec problems

Hello, I currently get the following error against PostgreSQL 9.4 while using finangle and finatra

java.lang.AbstractMethodError: com.twitter.finagle.postgres.codec.PgCodec$$anon$1.newClientDispatcher(Lcom/twitter/finagle/transport/Transport;Lcom/twitter/finagle/Stack$Params;)Lcom/twitter/finagle/Service;
    at com.twitter.finagle.builder.CodecClient$Client$3.newDispatcher(ClientBuilder.scala:1121)
    at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1$$anonfun$apply$1.apply(StackClient.scala:343)
    at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1$$anonfun$apply$1.apply(StackClient.scala:343)
    at com.twitter.util.Future$$anonfun$map$1$$anonfun$apply$6.apply(Future.scala:947)
    at com.twitter.util.Try$.apply(Try.scala:13)
    at com.twitter.util.Future$.apply(Future.scala:102)
    at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:947)
    at com.twitter.util.Future$$anonfun$map$1.apply(Future.scala:947)
    at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:893)
    at com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:892)
    at com.twitter.util.Promise$Transformer.liftedTree1$1(Promise.scala:100)
    at com.twitter.util.Promise$Transformer.k(Promise.scala:100)
    at com.twitter.util.Promise$Transformer.apply(Promise.scala:110)
    at com.twitter.util.Promise$Transformer.apply(Promise.scala:91)
    at com.twitter.util.Promise$$anon$3.run(Promise.scala:691)
    at com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:187)
    at com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:158)
    at com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:216)
    at com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:84)
    at com.twitter.util.Promise.continue(Promise.scala:689)
    at com.twitter.util.Promise$Chained.continue(Promise.scala:206)
    at com.twitter.util.Promise$Responder$class.transform(Promise.scala:175)
    at com.twitter.util.Promise$Chained.transform(Promise.scala:182)
    at com.twitter.util.Future.flatMap(Future.scala:892)
    at com.twitter.util.Future.map(Future.scala:947)
    at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1.apply(StackClient.scala:343)
    at com.twitter.finagle.client.StdStackClient$$anon$1$$anonfun$make$1.apply(StackClient.scala:343)
    at com.twitter.finagle.ServiceFactory$$anon$9.apply(Service.scala:206)
    at com.twitter.finagle.postgres.codec.HandleErrorsProxy.apply(PgCodec.scala:80)
    at com.twitter.finagle.postgres.codec.AuthenticationProxy.apply(PgCodec.scala:107)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.builder.CodecClient$$anon$4$$anon$2.apply(ClientBuilder.scala:1071)
    at com.twitter.finagle.service.FailFastFactory.apply(FailFastFactory.scala:183)
    at com.twitter.finagle.pool.WatermarkPool.apply(WatermarkPool.scala:136)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.service.FailureAccrualFactory.apply(FailureAccrualFactory.scala:140)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
    at com.twitter.finagle.loadbalancer.HeapBalancer.apply(HeapBalancer.scala:252)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.factory.StatsFactoryWrapper.apply(StatsFactoryWrapper.scala:43)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.factory.RefcountedFactory.apply(RefcountedFactory.scala:19)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.factory.TimeoutFactory.apply(TimeoutFactory.scala:58)
    at com.twitter.finagle.service.Requeues$$anon$2$$anon$1.com$twitter$finagle$service$Requeues$$anon$$anon$$applySelf(Requeues.scala:50)
    at com.twitter.finagle.service.Requeues$$anon$2$$anon$1.apply(Requeues.scala:74)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.factory.TimeoutFactory.apply(TimeoutFactory.scala:58)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.Filter$$anon$3.apply(Filter.scala:84)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
    at com.twitter.finagle.ProxyServiceFactory$class.apply(Service.scala:214)
    at com.twitter.finagle.ServiceFactoryProxy.apply(Service.scala:239)
    at com.twitter.finagle.ServiceFactory.apply(Service.scala:149)
    at com.twitter.finagle.FactoryToService.apply(Service.scala:313)
    at com.twitter.finagle.postgres.Client.fire(Client.scala:127)
    at com.twitter.finagle.postgres.Client.send(Client.scala:135)
    at com.twitter.finagle.postgres.Client.sendQuery(Client.scala:75)
    at com.twitter.finagle.postgres.Client.executeUpdate(Client.scala:43)
    at de.envisia.hello.HelloWorldServer.warmup(HelloWorldServer.scala:22)
    at com.twitter.inject.app.App$class.main(App.scala:51)
    at de.envisia.hello.HelloWorldServer.com$twitter$inject$server$TwitterServer$$super$main(HelloWorldServer.scala:13)
    at com.twitter.inject.server.TwitterServer$class.main(TwitterServer.scala:33)
    at de.envisia.hello.HelloWorldServer.main(HelloWorldServer.scala:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:167)
    at com.twitter.app.App$$anonfun$nonExitingMain$3.apply(App.scala:166)
    at scala.Option.foreach(Option.scala:257)
    at com.twitter.app.App$class.nonExitingMain(App.scala:166)
    at de.envisia.hello.HelloWorldServer.nonExitingMain(HelloWorldServer.scala:13)
    at com.twitter.app.App$class.main(App.scala:132)
    at de.envisia.hello.HelloWorldServer.main(HelloWorldServer.scala:13)
    at de.envisia.hello.HelloWorldServerMain.main(HelloWorldServer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sbt.Run.invokeMain(Run.scala:67)
    at sbt.Run.run0(Run.scala:61)
    at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
    at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
    at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
    at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
    at sbt.Logger$$anon$4.apply(Logger.scala:85)
    at sbt.TrapExit$App.run(TrapExit.scala:248)
    at java.lang.Thread.run(Thread.java:745)
Exception thrown in main on startup

Exception: sbt.TrapExitSecurityException thrown from the UncaughtExceptionHandler in thread "run-main-0"
java.lang.RuntimeException: Nonzero exit code: 1
    at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 2 s, completed 24.05.2015 20:48:30

Process finished with exit code 1

The error appears to happen ON ALL query types, correct or incorrect ones.

Oh and Integration Tests are running fine, looks like some dependency problems.
But the only different thing is finangle 6.24 and 6.25, however that shouldn't be too incompatible.

Looks like the following pull request will fix the issue: #6

Can't restore after tooManyWaitersException

After TooManyWaitersException FailureAccrual marked connection as dead for 5 seconds. I guess after 5 seconds client tries to retry and again gets error (ChannelException: Remote Info: Not Available) and mark connection as dead for 6..13.. seconds. How could I catch whats happening and restore work? Config of client is default except noFailFast

getAnyOption(0) on SELECT AVG(column1) FROM table1 always return None

in sum, max etc same behavior (((

Implementing Postgres Client in terms of StackClient?

Would this project be interested in a newer PostgresClient that's implemented in terms of the StackClient, al lá finagle-mysql? I ask first instead of a PR because implementing the new PostgresClient would require either breaking changes on com.twitter.finagle.postgres.Client, or duplicating behavior into a newer client and sunsetting / leaving the original com.twitter.finagle.postgres.Client.

Thoughts?

Add a way to prepare update Queries

Hello, currently it is impossible to prepare a update query, this could lead to insecure code.
I think somehow it would be great to have a similar API than Anorm where you could "prepare" a query via SQL"" or SQL"""""" and use string interpolation directly.

Also it doesn't look to hard to have a feature like that.
However the main purpose should be an alternative to executeUpdate(str: String)so that we can INSERT or UPDATE without strings, which are easily attackable to SQL injections.

BinaryParameter(true) with preparedStatement produce weird value for BigDecimal

Hi, I am testing finagle-postgres with CockroachDb, and noticed all negative decimal values inserted into database are strange. After some test I narrowed it down to the binaryParameter.

Here's how I reproduced it :

root@:26257/it_test_db> show create table test;
+-------+-----------------------------------------------+
| Table |                  CreateTable                  |
+-------+-----------------------------------------------+
| test  | CREATE TABLE test (                           |
|       |                                               |
|       |     a INT NOT NULL,                           |
|       |                                               |
|       |     b DECIMAL NULL,                           |
|       |                                               |
|       |     CONSTRAINT "primary" PRIMARY KEY (a ASC), |
|       |                                               |
|       |     FAMILY "primary" (a, b)                   |
|       |                                               |
|       | )                                             |
+-------+-----------------------------------------------+

then create a client withBinaryParams(true) and tried to insert some data:

  val c =Postgres
    .Client()
    .withCredentials("root", None)
    .database("it_test_db")
    .withBinaryParams(true)
    .newRichClient("localhost:26257")

cala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 1, 1.1))
res0: Int = 1

scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 2, -1.1))
res1: Int = 1

scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 3, BigDecimal(1.1)))
res2: Int = 1

scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 4, BigDecimal(-1.1)))
res3: Int = 1

scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 5, BigDecimal(1)))
res4: Int = 1

scala> com.twitter.util.Await.result(c.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 6, BigDecimal(-1)))
res5: Int = 1

but in the database it became :

root@:26257/it_test_db> select * from test;
+---+------------------------+
| a |           b            |
+---+------------------------+
| 1 |                    1.1 |
| 2 |                   -1.1 |
| 3 |                    1.1 |
| 4 | -1844674407370955161.5 |
| 5 |                 100000 |
| 6 |                     -0 |
+---+------------------------+

however if client is created without withBinaryParams(true)

scala>   val c2 =Postgres.Client().withCredentials("root", None).database("it_test_db").newRichClient("localhost:26257")
c2: com.twitter.finagle.postgres.PostgresClientImpl = com.twitter.finagle.postgres.PostgresClientImpl@45682e3e

scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 7, BigDecimal(1.1)))
res6: Int = 1

scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 8, BigDecimal(-1.1)))
res7: Int = 1

scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 9, BigDecimal(1)))
res8: Int = 1

scala> com.twitter.util.Await.result(c2.prepareAndExecute("INSERT INTO test VALUES ($1,$2)", 10, BigDecimal(-1)))
res9: Int = 1

in the database it seems ok:

root@:26257/it_test_db> select * from test;
+----+------------------------+
| a  |           b            |
+----+------------------------+
|  1 |                    1.1 |
|  2 |                   -1.1 |
|  3 |                    1.1 |
|  4 | -1844674407370955161.5 |
|  5 |                 100000 |
|  6 |                     -0 |
|  7 |                    1.1 |
|  8 |                   -1.1 |
|  9 |                      1 |
| 10 |                     -1 |
+----+------------------------+
(10 rows)

Twitter's use of finagle-postgres

Twitter has relatively recently begun work on an internal project that will be using the PostgreSQL wire protocol, and are hoping to use finagle-postgres for that since it's a great project. Obviously, @vkostyukov (and more recently @steveniemitz and I suspect others) from Twitter have been contributing to the project, but we're interested in upping our involvement a bit.

The first question, mostly for @leonmaia but really any interested contributor or user of the project, is whether you're open to us stepping in to help with the maintenance of the project more. And then related to that, we wanted to propose the idea of promoting finagle-postgres to being a core part of the finagle project alongside finagle-mysql, finagle-redis, etc. That would allow the CSL team inside of Twitter to more easily maintain it with the rest of Finagle to keep it in sync with any changes, etc.

Anyway, we really like the project and are excited about using it, and would love to help however we can. 🎉

Exception in thread main

Hello,

Testing a simple connection to a local postgresql instance with Scala 2.11, I got this exception:
Exception in thread "main" java.lang.AbstractMethodError: com.twitter.finagle.stats.MetricsStatsReceiver.com

stacktrace
[error] at com.twitter.finagle.stats.StatsReceiverWithCumulativeGauges$class.$init$(CumulativeGauge.scala:133)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:175)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:179)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:180)
[error] at com.twitter.finagle.stats.MetricsStatsReceiver.(MetricsStatsReceiver.scala:181)
[error] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error] at java.lang.Class.newInstance(Class.java:442)
[error] at com.twitter.app.LoadService$$anonfun$5.apply(LoadService.scala:74)
[error] at com.twitter.app.LoadService$$anonfun$5.apply(LoadService.scala:63)
[error] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
[error] at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
[error] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[error] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
[error] at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
[error] at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
[error] at com.twitter.app.LoadService$.apply(LoadService.scala:63)
[error] at com.twitter.finagle.util.LoadService$.apply(LoadService.scala:14)
[error] at com.twitter.finagle.stats.LoadedStatsReceiver$.(LoadedStatsReceiver.scala:18)
[error] at com.twitter.finagle.stats.LoadedStatsReceiver$.(LoadedStatsReceiver.scala)
[error] at com.twitter.finagle.stats.ClientStatsReceiver$.(LoadedStatsReceiver.scala:47)
[error] at com.twitter.finagle.stats.ClientStatsReceiver$.(LoadedStatsReceiver.scala)
[error] at com.twitter.finagle.client.StackClient$.(StackClient.scala:389)
[error] at com.twitter.finagle.client.StackClient$.(StackClient.scala)
[error] at com.twitter.finagle.Postgres$.com$twitter$finagle$Postgres$$defaultStack(Postgres.scala:69)
[error] at com.twitter.finagle.Postgres$Client$.apply$default$1(Postgres.scala:96)
[error] at com.example.finatrabasics.controllers.testSuite$.main(DbController.scala:63)
[error] at com.example.finatrabasics.controllers.testSuite.main(DbController.scala)

It happens trying the example client connection you have on your quickstart page

val client = Postgres.Client()
        .withCredentials("user", Some("pass"))
        .database("mydb")
        .newRichClient("localhost:5432")

Just creating this issue in case your team would like someone succesfully using this library with newer versions of Scala.

finagle-postgres version for scala 2.10

Are there any plans to create version for scala 2.10? I did not find any available jar in maven repositories.

Support "ApplicationName" connection parameter

With JDBC I can pass the ApplicationName parameter as part of the connection string - jdbc:postgresql://localhost:5435/DBNAME?ApplicationName=MyApp - and have "MyApp" show up in various statistics on the server. It would be useful to be able to do this on finagle-postgres as well.

Regarding openssl issues on master

Seems like there was a breaking change from MD5 to SHA-256 in openssl. The most appropriate solution would be to recreate the encrypted secrets with this newer version.

https://stackoverflow.com/a/39641378/539209

finagle / finagle-postgres Goto Github PK

finagle-postgres's Introduction

Finagle Postgres

Documentation

Using the Postgres client

Installation

Connecting to the DB

Selecting with simple query

Integration Tests

Changelog

Contributors

License

finagle-postgres's People

Contributors

Stargazers

Watchers

Forkers

finagle-postgres's Issues

Reproduction

Recommend Projects

Recommend Topics

Recommend Org