Geo search about amplify-category-api HOT 48 OPEN

Elastic search is a workaround. We'd like native geo features, like Firebase is about to offer. Amplify is meant for mobile apps, and "get all things in this geo box" or "get all things in a radius around this geo point" is a very popular usage for mobiles apps. Please consider making this a high priority

from amplify-category-api.

I looked at the new announcement of ultra warm and hoped this fulfill this need, but unfortunately not. Can we expect anything on this soon?

from amplify-category-api.

@YikSanChan that link is about hacking AppSync to work with ES geo features. Specifically, an object with two Floats is transformed into an ES geo_point. True we can do it, but we are requesting native geo support all the way down to Amplify. Meaning having AppSync create AWSGeoPoint custom type, and graphql transformer/code generator create meaningful search functions when a graphql type has @searchable AND has an AWSGeoPoint variable.

from amplify-category-api.

Is there maybe an update / progress about supporting GEO search / queries in dynamodb. I have it now up and running in a elasticsearch service (via amplify) which works great, but I also notice that it is very expensive while I am just have a few call per months.

from amplify-category-api.

Will this be on the roadmap soon or do we not need to hope for it and accept the current solution?

from amplify-category-api.

Any updates / progress?

from amplify-category-api.

@kaustavghosh06 @undefobj @mikeparisstuff Is there maybe an update about geo support in aws amplify on top of dynamodb or more a kind of serverless elasticsearch implementation where you only pay if you query elastic. This, to decrease the heavy costs of the use of @searchable

from amplify-category-api.

An AppSync dev has confirmed that this feature is in the planning pipeline
https://forums.aws.amazon.com/thread.jspa?threadID=296334&tstart=0

from amplify-category-api.

To add, read https://medium.com/open-graphql/building-a-graphql-api-by-example-restaurant-reviews-part-4-geospatial-search-e0e0d9dc0329 for how to geo search with ES.

from amplify-category-api.

+1

from amplify-category-api.

So correct me if I'm wrong, but there are three necessary pieces for this...

1. To be able to store location in DynamoDB in formats that match elastic search's formats (maybe just automatically recognize them as part of @searchable, maybe with an off switch for backwards compatibility)
2. Stream the data to store it in elastic search in the proper format
3. Create a ModelGeoFilterInput and a SearchableGeoFilterInput so you can perform actions

Maybe starting with the simplest use-case would be a good PR:

• Ability to add single location (lat/lng)
• Ability to index/sort by distance from a given point
• Ability to filter by a bounding box

Some more info:
https://www.elastic.co/blog/geo-location-and-search
https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html

from amplify-category-api.

@rpostulart, if you're using AppSync client, you can do geo-related queries with Elasticsearch as your data source. Might take a couple more steps than your regular DynamoDb setup but it's not too complicated. There's a tutorial here: https://goo.gl/Nbyrri for setting up your resolvers, and Elasticsearch's geo-query documentation is here: https://goo.gl/VHU4aH

from amplify-category-api.

DynamoDB does not support geo queries, which is why that repo is in archive now. AppSync supports both DynamoDB as well as Elasticsearch which you can leverage in a single API and stream the data from DynamoDB to Elasticsearch indexes. That's why @searchable was added: https://aws-amplify.github.io/docs/cli/graphql#searchable

I'm not sure what the feature request is here since Amplify leverages the backend AWS services. Could you see if @searchable in the transformer meets your use case?

from amplify-category-api.

@undefobj, the @searchable directive provides an interface for String, Int, Float, and Boolean searches. If I understand it correctly, I think the feature request here is for geo-related searches which is probably two requests in one when broken down:

1. Support geo-types, i.e a new GraphQLObjectType, like AWSGeo?
2 Enable geo-related queries via @searchable, which is a bit more involved than your typical eq, ne, lt, gt e.t.c

from amplify-category-api.

For new GraphQL scalar types that's best logged as a feature request to the AppSync service. I would recommend logging that here: https://forums.aws.amazon.com/forum.jspa?forumID=280&start=0

If there are enhancements needed to the GraphQL transformer it's best logged as a feature request in the Amplify CLI repo (as this repo is for the library): https://github.com/aws-amplify/amplify-cli

@rpostulart does this match with your request?

from amplify-category-api.

Why do you need ElasticSearch anyway for simple queries such as 'lt' 'ne' etc? DynamoDB supports these by itself. Is this just a away to force users to use and pay for ES even when not necessary?

Anyway, you can now do a Float lat and Float lon and do box queries by havin lat and lat lt and lon gt and lon lt, but come on... Also it fails at poles or if you need large boxes or.... Anyway you shouldn't have to be a spherical geometry expert to do mobiles apps, that's what libs are for (although basic knowledge is still a must IMHO)

from amplify-category-api.

@undefobj
True dynamodb doesn't support geo features natively, so that archived lib simulated geo features by using geohashes and in-memory filtering, similar to what firebase geo libs do. This would be an acceptable solution fornus IMO, or else intergrate @searchable with ES geo queries and types as guys previously said.

from amplify-category-api.

@undefobj I will check asap and let it know here.

from amplify-category-api.

There is this tutorial for geo queries in AppSync. Also, this AppSync tutorial mentions geospatial data/. So it is probably somehow possible, just very convoluted. App-serving frameworks like amplify should be much simpler.

from amplify-category-api.

@doom777 you can use @model to generate the simple queries that you state above, as outlined here: https://aws-amplify.github.io/docs/cli/graphql#generates

However these will just use DynamoDB itself and at scale these filtering techniques may be less performant based on your data access pattern which is why Elasticsearch exists. The @searchable will automatically stream data from DynamoDB to Elasticsearch like in the tutorial that you linked above.

from amplify-category-api.

I checked and agree with @doom777 . ElasticSearch does not suits my use case. I believe that ElasticSearch is too heavily for this use case. Because I expect it will be a highly used mobile app. I don't want to spin up an instance for each request, I prefer a serverless solutions for example with Lambda.

from amplify-category-api.

I may be wrong, but I don't think an ES instance is spun up on every request; I think it stays there permanently, possibly in a cluster.

from amplify-category-api.

Sorry that's what I mean indeed!

from amplify-category-api.

Right, so to be clear @rpostulart as in the link I posted above @model will include some filtering capability out of the box with DynamoDB. For instance any string types will have operations like less than, between, etc:

input ModelStringFilterInput {
  ne: String
  eq: String
  le: String
  lt: String
  ge: String
  gt: String
  contains: String
  notContains: String
  between: [String]
  beginsWith: String
}

This is probably good for prototyping & small apps, which is why we added this capability based on customer feedback. However, if your app takes off and your table gets very large this becomes less scalable and you'll want to leverage a purpose-built search engine like Elasticsearch which is why @searchable exists. The Elasticsearch instance is not "serverless" though as it will always be up whereas DynamoDB is a per-request model. If you do believe your app will be highly used then you should go this route, however if not then you could just start with the base DynamoDB capabilities and later you could always add in Elasticsearch. It's not a one-way door. AWS does not offer a "serverless full text search" service at this time so outside of this not much else we can offer here.

from amplify-category-api.

Ok thanks, then I will focus on that.

from amplify-category-api.

At the very least AWS could offer great documentation on how to implement this common usw casr

from amplify-category-api.

@doom777 what more documentation could we add here? I linked to the transformer documentation above, is more needed there? Or alternatively is something else specifically needed?

from amplify-category-api.

Look, I need to do a "select all entity in this geo box" query. Please show me an AWS tutorial how to do it. This is a common enough case.

from amplify-category-api.

Thanks for the feedback @doom777. As noted at this time you need Elasticsearch with @searchable directive, however we'll look into ways to see if something could be done in the GraphQL transformer in the future for DynamoDB only. I've tagged this as a feature request and added it into the CLI repo.

from amplify-category-api.

@undefobj the issue is less in making it dynamodb focused and dropping ES (although running an ES cluster is not very serverless)

In general there is no support for geo in the Amplify/AppSync stack. No geo types, no query functions.

from amplify-category-api.

@doom777 I like your proposal. Maybe a better way to propose this is to submit an RFC?

from amplify-category-api.

Maybe one day DocumentDB will support geo indexes like normal MongoDB?...

from amplify-category-api.

I have picked up this task again with Amplify and Elasticsearch this time. GEO searching is then still not an out of the box solution, these are just two seperate services and we need to put something in between to do geo searching. It would be still great if AWS can provide something with Amplify to do GEO search on Elasticsearch or Dynamodb. Are there some developments? @doom777 also asked for an update in the AWS forum.

from amplify-category-api.

What about something like this? https://read.acloud.guru/location-based-search-results-with-dynamodb-and-geohash-267727e5d54f

from amplify-category-api.

I know, but I don't want to use a package that hardly is used or maintained. It has open issues older than 2 years :(

from amplify-category-api.

++1

from amplify-category-api.

+1

from amplify-category-api.

+1

from amplify-category-api.

+1

from amplify-category-api.

+1

from amplify-category-api.

+1

from amplify-category-api.

any simple solution for this issue?

from amplify-category-api.

+1

from amplify-category-api.

Can this be closed in favor of Amplify's geo category - https://docs.amplify.aws/cli/geo/search/ ?

from amplify-category-api.

Can this be closed in favor of Amplify's geo category - https://docs.amplify.aws/cli/geo/search/ ?

Geo Search vs geospatial search?

With amplify geo it is Not possible to listPlaces from DynamoDB which Are in the near of me right?

from amplify-category-api.

With amplify geo it is Not possible to listPlaces from DynamoDB which Are in the near of me right?

Not from DynamoDB, but from Amazon Location Service.

from amplify-category-api.

With amplify geo it is Not possible to listPlaces from DynamoDB which Are in the near of me right?

Not from DynamoDB, but from Amazon Location Service.

I guess its not that what here is needed.
it should be a cheaper solution as the @search directive.
So ALS is a forreign api to find places and directions to external data.

from amplify-category-api.

any updates here ?

from amplify-category-api.