Giter VIP home page Giter VIP logo

elasticsearch-net-aws's Introduction

Elasticsearch Net for Amazon AWS

Add-on to elasticsearch-net / NEST for using AWS's elasticsearch service.

Install Package

On Nuget:

# For ElasticSearch.Net >= 5.0.0
Install-Package Elasticsearch.Net.Aws
# or for dotnet core
dotnet add package Elasticsearch.Net.Aws

# For ElasticSearch.Net 2.X
Install-Package bcuff.Elasticsearch.Net.Aws-v2
# or for dotnet core
dotnet add package bcuff.Elasticsearch.Net.Aws-v2

# For ElasticSearch.Net 1.X
Install-Package Elasticsearch.Net.Aws-v1
# or for dotnet core
dotnet add package Elasticsearch.Net.Aws-v1

Setup

Elasticsearch.Net Version >= 2.0.2

Use Package Elasticsearch.Net.Aws.

Typical Setup

// for NEST

// This constructor will look up AWS credentials in the
// same way that the AWSSDK does automatically.
var httpConnection = new AwsHttpConnection();

var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var config = new ConnectionSettings(pool, httpConnection);
var client = new ElasticClient(config);

.NET Core Applications using IConfiguration

IConfiguration config = Configuration;
var options = config.GetAWSOptions();
var httpConnection = new AwsHttpConnection(options);

// same as above

Elasticsearch.Net Version 1.7.1

Use Package Elasticsearch.Net.Aws-v1

Source for this version is maintained on the version-1 branch

// for NEST
var client = new ElasticClient(settings, connection: new AwsHttpConnection(settings, new AwsSettings
{
    AccessKey = "My AWS access key",
    SecretKey = "My AWS secret key",
    Region = "us-east-1",
}));

The AwsHttpConnection class is an implemenation of IConnection that will sign the HTTP requests according to the Version 4 Signing Process.

Serilog Sink Setup

In Code

  const string esUrl = "https://aws-es-thinger.us-west-1.es.amazonaws.com";
  Log.Logger = new LoggerConfiguration()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(esUrl))
                {
                    ModifyConnectionSettings = conn =>
                    {
                        var httpConnection = new AwsHttpConnection("us-east-1");
                        var pool = new SingleNodeConnectionPool(new Uri(esUrl));
                        return new ConnectionConfiguration(pool, httpConnection);
                    }
                })
                .CreateLogger();

In Configuration

{
    "Serilog": {
        "Using": ["Serilog", "Serilog.Exceptions", "Serilog.Sinks.Elasticsearch", "Serilog.Enrichers.Environment", "Serilog.Enrichers.Process"],
        "MinimumLevel": "Debug",
        "WriteTo": [{
                "Name": "Elasticsearch",
                "Args": {
                    "nodeUris": "https://******.us-east-1.es.amazonaws.com",
                    "numberOfShards": 5,
                    "numberOfReplicas": 10,
                    "connection": "Elasticsearch.Net.Aws.AwsHttpConnection, Elasticsearch.Net.Aws"
                }
            }
        ],
        "Enrich": ["FromLogContext", "WithMachineName", "WithExceptionDetails"],
    }
}

elasticsearch-net-aws's People

Contributors

bcipri avatar bcuff avatar chrisoverzero avatar dependabot[bot] avatar janireu avatar jeffashton avatar nanestev avatar odyth avatar raol avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

elasticsearch-net-aws's Issues

AWS4Signer

Hello.
Thanks for useful library. Have you considered integration with AWS4Signer from AWS .Net SDK? Is it possible at least?

Thanks

How do I create a client with the V4 Signing process

Hi, I was wondering if you could help me create a client that is signed with the V4 signing process?

I am using NEST v 6.6.0 and I've installed the Elasticsearch.Net.Aws package but I'm not sure how I create a client with my credentials.

The example for Elasticsearch.Net Version >= 2.0.2 on the README doesn't show how to create the client with any kind of credentials/access key/secret.

Any help would be really appreciated, thanks!

Suggestion

I wanted to leave a suggestion because i wasted FAR too many hours trying to find a way to do it and it was right in front of me. You recommend the following for Serilog

 const string esUrl = "https://aws-es-thinger.us-west-1.es.amazonaws.com";
  Log.Logger = new LoggerConfiguration()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(esUrl))
                {
                    ModifyConnectionSettings = conn =>
                    {
                        var httpConnection = new AwsHttpConnection("us-east-1");
                        var pool = new SingleNodeConnectionPool(new Uri(esUrl));
                        return new ConnectionConfiguration(pool, httpConnection);
                    }
                })
                .CreateLogger();

This works and is fine, but doesn't allow for Configuration-based setup to be used. I would recommend the following which would also allow for people who use appSettings.json as their method for loading

 const string esUrl = "https://aws-es-thinger.us-west-1.es.amazonaws.com";
  Log.Logger = new LoggerConfiguration()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(esUrl))
                {
                    Connection = new AwsHttpConnection()
                })
                .CreateLogger();

In the Serilog configuration you can now do this:

{
    "Serilog": {
        "Using": ["Serilog", "Serilog.Exceptions", "Serilog.Sinks.Elasticsearch", "Serilog.Enrichers.Environment", "Serilog.Enrichers.Process"],
        "MinimumLevel": "Debug",
        "WriteTo": [{
                "Name": "Elasticsearch",
                "Args": {
                    "nodeUris": "https://******.us-east-1.es.amazonaws.com",
                    "numberOfShards": 5,
                    "numberOfReplicas": 10,
                    "connection": "Elasticsearch.Net.Aws.AwsHttpConnection, Elasticsearch.Net.Aws"
                }
            }
        ],
        "Enrich": ["FromLogContext", "WithMachineName", "WithExceptionDetails"],
    }
}

What are the hard-coded IP & URI constants in InstanceProfileService.cs for?

Hey, I was going through the code and found some constants in the code, as copied below:

    private static readonly string[] AliasSeparators = { "<br/>" };
    private const string Server = "http://169.254.169.254";
    private const string RolesPath = "/latest/meta-data/iam/security-credentials/";
    private const string SuccessCode = "Success";

I'm curious as to what the IP address and the URI constants are for?

AWS - The server name or address could not be resolved

Hello,

I am experiencing problems with AWS and Elastic Search using this client.

I have tried with the VPC endpoint and the Kinbana endpoint defined in the AWS Console for the Elastic Search Instance. Could you direct me as to what i am doing wrong?

NEST Version: 6.2
Elastic Search Version: 6.3

Invalid NEST response built from a unsuccessful low level call on HEAD: /_plugin/kibana/
Audit trail of this API call:

  • [1] BadRequest: Node: https://vpc-xxxx-elastic-search-lto7xxxxxxxxxx.us-west-1.es.amazonaws.com/_plugin/kibana/ Took: 00:00:00.4433711
    OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The server name or address could not be resolved
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
    at System.Net.Http.WinHttpHandler.d__105.MoveNext()
    --- End of inner exception stack trace ---

Code

  var httpConnection = new Elasticsearch.Net.Aws.AwsHttpConnection("us-west-1", new Elasticsearch.Net.Aws.StaticCredentialsProvider(new Elasticsearch.Net.Aws.AwsCredentials
            {
                AccessKey = _awsSettings.AwsS3AccessKey,
                SecretKey = _awsSettings.AwsS3AccessSecret,
            }));
            
            var pool = new SingleNodeConnectionPool(new Uri(_esSettings.ElasticSearchURL));

            _esNode = new Uri(_esSettings.ElasticSearchURL);
            _esConfig = new ConnectionSettings(pool, httpConnection)
                        .DefaultIndex(_esSettings.ElasticSearchVideoIndex)
                        .DefaultMappingFor<VideoDocument>(i => i
                            .TypeName(_esSettings.ElasticSearchVideoIndexType)
                        );
            _esConfig.DisableDirectStreaming();
            _esClient = new ElasticClient(_esConfig);
            var pingResponse = _esClient.Ping();

I have tried to use _esConfig.DisableDirectStreaming();

Elasticsearch.Net v7 Support

I'm keen to help out with updating to Elasticsearch.Net v7. Its currently in alpha but it has a particularly useful change elastic/elasticsearch-net@32e2b5c that allows delegating handlers on the request pipeline.

This means we can add in AWS XRay support to get visibility on ES.

There are numerous breaking changes that will need to be updated in the v7 release.

As this project has been stable for some time on v6, this issue is bit of a shoutout to the existing maintainers to see if they need help.

Connection examples don't seem to work.

Hello,

I'm trying to work out how to use this library to connect to Elastic Search running on Amazon's hosted AWS offering. However, it seems like that changes in Elasticsearch.Net have broken this example.

I'm using Elasticsearch.Net.Aws from Git, and I'm using ElasticSearch.Net 2.0.0.0 installed via NuGet.

Here's my code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Elasticsearch.Net;
using Elasticsearch.Net.Aws;

namespace AWSConnectionTests
{
    class Program
    {
        static void Main(string[] args)
        {
            var httpConnection = new AwsHttpConnection(new AwsSettings
            {
                AccessKey = "My AWS access key",
                SecretKey = "My AWS secret key",
                Region = "us-east-1",
            });

            var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
            var config = new ConnectionSettings(pool, httpConnection);
            var client = new ElasticClient(config);
        }
    }
}

I'm getting the following type errors:

  • The type or namespace name 'ConnectionSettings' could not be found (are you missing a using directive or an assembly reference?)
  • The type or namespace name 'ElasticClient' could not be found (are you missing a using directive or an assembly reference?)

Has something changed? Am I tilting and windmills?

Thanks!

-D

Error, try to ping ES connection.

Hello,

I have next error, when I try to ping my connection with elasticSearch.

Unsuccessful low level call on HEAD: /

Audit trail of this API call:

  • [1] BadResponse: Node:

OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute

Request:

<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

Response:

<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

TestCode
private static string Region => TestConfig.AwsSettings.Region;
private static ICredentialsProvider Credentials => new StaticCredentialsProvider(TestConfig.AwsSettings);
[Test]
public void Ping_should_work()
{
var httpConnection = new AwsHttpConnection(Region, Credentials);
var pool = new SingleNodeConnectionPool(new Uri(TestConfig.Endpoint));
var config = new ConnectionConfiguration(pool, httpConnection);
var client = new ElasticLowLevelClient(config);
var response = client.Ping();
Assert.AreEqual(200, response.HttpStatusCode.GetValueOrDefault(-1));

    }

OpenSearch Serverless

Amazon has recently introduced a new serverless option for opensearch. In their infinite wisdom, they decided to change the service name for the purposes of signing HTTP requests from 'es' to 'aoss' for this new service, which means all clients break (fun)...

I forked this project in hopes of fixing the library, but it turns out I was able to replace it with very simple amount of code. You might want to see how much of this can be reused for 4.8.1 and just replace what you have (no more need for reflection, etc.).

I'll push a commit with this version to my fork, but I turned off building for 4.8.1 since I don't need that and don't have time right now to make it work and test.

(depends on the nuget packages AWSCRT and AWSCRT-HTTP, and was based on the code posted here: https://github.com/aws-samples/sigv4a-signing-examples/tree/main/dotnet)

Mostly just posting this here so it helps someone and they can take up the mantle if desired since I don't have the bandwidth myself at the moment and it took a while for me to get this working! Some differences from the link above - we needed a region in the signature, so you need to use the SIGV4 instead of SIGV4A as in the sample code.

    private class AwsConnection : HttpConnection
    {
        private readonly Credentials _credentials;
        private readonly string _serviceName;
        private readonly string _regionName;

        public AwsConnection(string endpointUri)
        {
            var creds = new EnvironmentVariablesAWSCredentials().GetCredentials();
            _credentials = new Credentials(creds.AccessKey, creds.SecretKey, creds.Token);
            var endpointParts = endpointUri.Split('.');
            _serviceName = endpointParts[^3];
            _regionName = endpointParts[^4];
        }

        protected override HttpRequestMessage CreateHttpRequestMessage(RequestData requestData)
        {
            var request = base.CreateHttpRequestMessage(requestData);
            _SignRequest(request);
            return request;
        }

        private void _SignRequest(HttpRequestMessage request)
        {
            if (request.RequestUri == null)
                throw new ArgumentException("Missing request URI on request object");
            var awsRequest = new HttpRequest
            {
                Method = request.Method.ToString(),
                Headers = new[] { new HttpHeader("host", request.RequestUri.Host) },
                Uri = request.RequestUri.PathAndQuery,
            };
            var awsSigningConfig = new AwsSigningConfig
            {
                Service = _serviceName,
                Region = _regionName,
                Algorithm = AwsSigningAlgorithm.SIGV4,
                SignatureType = AwsSignatureType.HTTP_REQUEST_VIA_HEADERS,
                SignedBodyHeader = AwsSignedBodyHeaderType.X_AMZ_CONTENT_SHA256,
                Credentials = _credentials,
            };

            var result = AwsSigner.SignHttpRequest(awsRequest, awsSigningConfig);
            var signingResult = result.Get();
            var signedRequest = signingResult.SignedRequest;
            foreach (var header in signedRequest.Headers)
            {
                if (request.Headers.Contains(header.Name))
                    request.Headers.Remove(header.Name);
                request.Headers.TryAddWithoutValidation(header.Name, header.Value);
            }
        }
    }

Also, FYI - I found out while looking into this that the "right" way to do what you had to do with reflection (your comments in the code) is to use a DelegatingHandler as your base class for your http message handler, and then just create is as a wrapper around the other handler. You create it and set InnerHandler to the inner handler (or use the protected constructor), then in your override of SendAsync you just call base.SendAsync when you want to trigger the inner handler. Not sure if this existed at the time you did the other work, or exists in 4.8.1, but it seemed like a nice way to avoid the reflection.

Use with Serilog Sink

Just looking for solutions at the moment.

There is a plugin for serilog (logging framework) for elasticsearch that uses the elasticsearch package for pumping messages to ES, though I don't have a good idea looking through it's code how i could get it to sign the AWS requests.

I'm wondering if you had any ideas for how to possibly use this with serilog.

(ATM considering forking the current sink they have and dropping in this library and exposing the keys part... unless you've happend to do have done this before or know of a better solution)

Support .NET Standard in AWS-v1

Hi,

I was wondering if you have any plans to convert the AWS-v1 package to NET Standard 1.X as you have for v2? If not, is there anything in particular blocking you?

Thanks

Re-use auth header

Hi Brandon!

That's a great library. I have one question. Here, it looks like a signature is valid for 15 minutes:

"The signed portions (using AWS Signatures) of requests are valid within 15 minutes of the timestamp in the request."

Do you think it would be a good idea to re-use signature for at least ~10 minutes? Or at least make it an option? I believe the overhead would be much more smaller than the current solution.

You can have a signature pool where the key is the http method and the request uri, and the value is the computed signature.

Thanks

Issue with default AWS credentials and roles

Hi,

When creating an instance of AwsHttpConnection it seems to always default to the "default" profile that I have setup in my AWS credentials file.

In my appsettings.json file I have this.

"AWS": {
    "Profile": "localdev",
    "Region": "eu-west-1"
  }

This doesn't seem to get picked up at all which means I can't access the same resources as my other AWSSDK client because I have a role attached to that profile.

Thanks,

Doesn't seem to be compatible with v6.0.0 (ElasticSearch.Net 6.0.0-rc1)

It looks like the internals of the 6.0 release has changed slightly.
I'm getting this exception:

System.MissingMethodException
Method not found: 'Elasticsearch.Net.PostData1<System.Object> Elasticsearch.Net.RequestData.get_PostData()'. at Elasticsearch.Net.Aws.AwsHttpConnection.SignRequest(IRequest request, RequestData requestData) at Elasticsearch.Net.Aws.AwsHttpConnection.CreateHttpWebRequest(RequestData requestData) at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData) in c:\Source\elasticsearch-net-master\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 147 at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) in c:\Source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 439 at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in c:\Source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 68

v6.1.0 Not Working With .NET Standard 2.0

I found using the v6.1.0 version with .NET Standard 2.0 does not seem to sign the request properly causing a 403 error to get returned.

Work Around

I was able to work around the issue by pulling the latest then retargeting the Elasticsearch 6.* packages. The fix appears for it appears to be in the latest version targeting Elasticsearch 7.*.

Strongly Named Bin?

Hey Brandon, Absolutely LOVE that you made this and Open Sourced it. I'm using it for a larger project where I can only use Strongly Named assemblies. Any chance you could sign these before you send them to nuget?

Thanks!

Joey

Dis is awseome!

so I just wanted to say thanks for this...

This really saved my bacon (and an awkward talk with my boss as I thought I would need time to implement this)

๐Ÿ‘

AWS Credentials is null

Hi guys,
I'm not sure if this is an issue or I'm doing something wrong here...

Basically I setup my NEST client following your instructions....

var awsOptions = _configuration.GetAWSOptions();
var httpConnection = new AwsHttpConnection(awsOptions);

The problem seems to be that the GetAWSOptions is not populating the AWSCredentials which then results in an exception...

https://github.com/bcuff/elasticsearch-net-aws/blob/master/src/Elasticsearch.Net.Aws/Elasticsearch.Net.Aws/AwsHttpConnection.cs#L39

I have checked all my AWS configuration and tested it against the AWS .net SDK just querying a S3 bucket and it seems to work without any issues. I had a look in the AWS code and it seems that the credentials do not get set until you actually request a service by calling CreateServiceClient...

var options = _configuration.GetAWSOptions();
var service = options.CreateServiceClient<IAmazonKinesisFirehose>()

Have a look here as well...
https://github.com/aws/aws-sdk-net/blob/master/extensions/src/AWSSDK.Extensions.NETCore.Setup/ClientFactory.cs#L114

Let me know if you need any more information.

Thanks!

AWS Dependency Issue

Hey, I think this is related to #61

AWS just released 3.7.0 and it's exhibiting the same issue :(

Last time a rebuild seemed to fix it, would you mind rebuilding again please.

Doesn't support credentials profile file

I have no AWS credential environment variables, but I have them specified in the [user_folder]/.aws/credentials file instead.

This doesn't pick them up, so I am forced to specify environment variables.

Couldn't able to create AwsElasticSearchInstance

Hi,

Thanks for helping in developing package for authorizing the Elastic search hosted on AWS. we are using "NEST" NPM to index the documents currently. we are planning to move to AWS-Elasticsearch and tried to performed below steps=>

Steps:


I installed nuget package ElasticSearch.Net.Aws on the top of NEST and tried to execute below code snippet:
Node = new Uri("ElasticSearchURL");
IConnectionConfigurationValues settings = new ConnectionConfiguration(Node);
var client = new ElasticClient(settings, connection: new AwsHttpConnection(settings, new AwsSettings
{
AccessKey = "My AWS access key",
SecretKey = "My AWS secret key",
Region = "us-east-1",
}));

As i am getting compilation error that " The base class or interface 'Elasticsearch.Net.Connection.HttpConnection' in assembly 'Elasticsearch.Net, Version=1.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d' referenced by type 'Elasticsearch.Net.Aws.AwsHttpConnection' could not be resolved ".

Can you please help us in this regard.

Regards,
Ravi.

Started receiving 403 for a specific application with message about request signature

Hi, I am using ELasticsearch.Net.Aws version 6.1.0 and AWS Elasticsearch service 6.7.

Using the nuget package was working fine for an EC2 instance I have on AWS but then out of nowhere it just stopped with the below error for all requests.

Status: 403
      {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

I can use VPN to access my Elasticsearch anonymously if I want and when I am on VPN using normal HttpConnection everything works fine. Any ideas?
How can I help with more information?

Authorization Header not passed in Request

I'm trying to use v7.1.0 with Elasticsearch.net v7.7 and Nest v7.14.1. After upgrading from v6.0, I am now getting 403 response from my AWS Elasticsearch endpoint. I found out that with the latest version the Authorization Header is not passed to the request.

Please see sample code below for creating the ElasticClient.

var httpConnection = new AwsHttpConnection(new BasicAWSCredentials(
                    AWSAccessKey,
                    AWSSecretKey
                ), Amazon.RegionEndpoint.USEast1);
                
                ConnectionSettings settings = new ConnectionSettings(pool, httpConnection);

               ElasticClient client = new ElasticClient(settings);

is there any way to avoid the large dependency update in 2.3.x?

I currently have a project that is based on .Net Framework 4.6.2 - when trying to upgrade this nuget from 2.1.4->2.3.3, I get a large list of dependencies which i believe are required for a project depending on .Net Standard but not for .Net Framework - is there any way to avoid injesting this large dependency list in my case? or do I update the nuget and then manually remove the dependencies that aren't required?

screen shot 2017-05-01 at 9 01 56 am

User: anonymous is not authorized to perform: es:ESHttpPost (Only an issue in IIS)

I'm using this nuget package for connecting to an Elasticsearch instance hosted on AWS.

var pool = new SingleNodeConnectionPool(new Uri(Url));
var httpConnection = new AwsHttpConnection(Region);
var config = new ConnectionSettings(pool, httpConnection)
                            .PrettyJson()
                            .DisableDirectStreaming()
                            .DefaultTypeName(TYPE)
                            .DefaultIndex(INDEX);
_client = new ElasticClient(config);

For setting the access key and secret, I have a credentials file stored on my Windows computer here: C:\Users{username}.aws\credential, which has a "default" entry. I've also tried in appsettings.json specifying AppSettings:AWSAccessKey & AppSettings:AWSSecretKey or specifying AWS:Profile (to a profile name that I set up using the AWS SDK Explorer in Visual Studio.) All three of these approaches are working great when I run my ASP.NET Core web application with the Project Properties > Debug > Launch option set to "Project". The "IIS Express" Launch option works as well.

If I check the credentials passed in, it shows the correct Access Key and Secret:

var credentials = FallbackCredentialsFactory.GetCredentials();
var immutableCredentials = credentials.GetCredentials();

Now, if I change my ASP.NET Core app to have the Launch option set to "IIS", all of a sudden my searches stop working.

HResult=0x80131500
Message=Invalid NEST response built from a unsuccessful low level call on POST: /rprsearchresidential/residentialsearch/_search?pretty=true&typed_keys=true

Audit trail of this API call:

  • [1] BadResponse: Node: https://{mynode}.us-east-1.es.amazonaws.com/ Took: 00:00:00.1804828

OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 403 from: POST /rprsearchresidential/residentialsearch/_search?pretty=true&typed_keys=true

Response:

{"Message":"User: anonymous is not authorized to perform: es:ESHttpPost"}

If I attempt to retrieve the immutableCredentials, it throws an exception when calling credentials.GetCredentials();

Message=A socket operation was attempted to an unreachable network
Source=System.Net.Http
StackTrace:
at System.Net.Http.ConnectHelper.d__2.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at ...
at Amazon.Util.AWSSDKUtils.DownloadStringContent(Uri uri, TimeSpan timeout, IWebProxy proxy)
at Amazon.Util.EC2InstanceMetadata.GetItems(String relativeOrAbsolutePath, Int32 tries, Boolean slurp)
at Amazon.Util.EC2InstanceMetadata.get_IAMSecurityCredentials()
at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
at RealtorsPropertyResource.Rpr.SearchesElasticsearch.ESClient.get_Instance() in C:\RPR2\Libraries\SearchesElasticsearch\Entities\ESClient.cs:line 35

It doesn't matter which method I use for storing the AWS Access key and secret, they all error in this manner. Am I missing something required in order to connect through IIS or is this a bug in the nuget package?

Elasticsearch NEST 5.0 compatibility issue

Hi I upgrade nest to 5.0.1 and the net-aws lib begins to throw exception of "Method not found Elasticsearch.Net.IApiCallDetails Elasticsearch.Net.IBodyWithApiCallDetails.get_CallDetails()". Any idea how to fix it or work around the issue?

        var httpConnection = new AwsHttpConnection(ElasticServiceSettings.ElasticAWSRegion, new StaticCredentialsProvider(
        new AwsCredentials
        {
            AccessKey = ElasticServiceSettings.ElasticAWSAccessKey,
            SecretKey = ElasticServiceSettings.ElasticAWSSecretKey
        }));
        var pool = new SingleNodeConnectionPool(ElasticServiceSettings.ElasticAWSUrl);
        var conn = new ConnectionSettings(pool, httpConnection);   
        var client = new ElasticClient(conn);  << failed here

Trouble using this inside VS with AWS SDK

Doesn't seem to want to use my default credentials from AWS. Fails when looking up credentials with error that connection to server could not be made? Why is trying to look up the credentials if they are local to my machine?

asp.net core

Hi, any thoughts on implementing this for asp.net core?

Thanks

7.0.0 - code not being signed

Hi Brendan

I'm trying es 7.0.0 with Elastisearch.net and nest with no success.
Here's the code I'm using. Am I leaving something out? It looks like the request is going through unsigned, indeed untouched.
Result is
{Invalid NEST response built from a unsuccessful (403) low level call on GET: /events/_doc/565540}

            var awsCreds = new Amazon.Runtime.BasicAWSCredentials("myaccess", "mysecret");
            var AWSHttpConnection = new Elasticsearch.Net.Aws.AwsHttpConnection(awsCreds, Amazon.RegionEndpoint.EUWest1);

            var AWSURI = new Uri("my url");
            var pool = new SingleNodeConnectionPool(AWSURI);

            var esAWSConfig = new ConnectionSettings(pool, AWSHttpConnection);

            Nest.ElasticClient client = new ElasticClient(esAWSConfig);
             
            var eventCMS = client.Get<EventCMS>(565540, g => g.Index("events"));

Highlight error with only 2 characters

I search text "abe" and the result is correct for highlight value "<Abe>rGain" but if I used just 2 characters "ab" the highlight result is incorrect because highlight whole word "<AberGain>"

Enabling Request/Response Causes Signature to be Rejected

Everything works fine with connectionSettings.EnableHttpCompression(false);

With connectionSettings.EnableHttpCompression(true);, I get {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

There appears to be a previous issue for this same problem (#36) but it was closed for some reason.

No longer works with Elasticsearch.Net 2.0

Looks like they did some refactoring of the ElasticClient constructor. This is the error im getting for

 this.elasticClient = new ElasticClient(settings, connection: new AwsHttpConnection(settings, new AwsSettings 
{
            AccessKey = Configurator.Get("ElasticsearchAccessKey", "API"),
            SecretKey = Configurator.Get("ElasticsearchSecretKey", "API"),
            Region = "us-east-1"
        }));`

Error CS7069: Reference to type 'IConnectionConfigurationValues' claims it is defined in 'Elasticsearch.Net', but it could not be found.

Examples of using AWS SDK Core to get credentials?

I apologize if this is a silly question but, what's the equivalent in version 6.1.0 for the following code? (which no longer compiles)

var instanceProfileCredentialProvider = new InstanceProfileCredentialProvider();

var httpConnection = new AwsHttpConnection("us-east-1", instanceProfileCredentialProvider);

AWS Dependencies Issue

Hey, there seems to be a disconnect between the nuget package and whats in the repository.

https://www.nuget.org/packages/Elasticsearch.Net.Aws/

This states:

AWSSDK.Extensions.NETCore.Setup (>= 3.3.100.1)

Which is causing problems as AWS just released 3.5 of their libraries.

https://www.nuget.org/packages/AWSSDK.Extensions.NETCore.Setup/

This is on 3.3.101 which supports AWSSDK.Core as AWSSDK.Core (>= 3.3.13.3 && < 3.6.0)

But the nuspec doesn't even specify this dependency at all. So I'm a bit confused.

This is causing build warnings at the moment:

Screenshot_20200821_122609

Do you want a PR to fix this up?

I need a .Net 4.0 version

The current published NuGet package does not support .Net 4.0.
If I get the source code and compile for .Net 4.0 should I expect this to work?
Would you like me to add .Net 4.0 support and submit a pull request?

Thanks,
Martin.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.