Giter VIP home page Giter VIP logo

sample-python-helper-aws-appconfig's Introduction

Sample AWS AppConfig Helper

A sample helper Python library for AWS AppConfig which makes rolling configuration updates out easier.

PyPI - Python Version PyPI version Code style: black

Features

  • Configurable update interval: you can ask the library to update your configuration as often as needed, but it will only call the AWS AppConfig API at the configured interval (in seconds).
  • Handles correct API usage: the library uses the new AppConfig Data API and handles tracking the next configuration token and poll interval for you.
  • Flexible: Can automatically fetch the current configuration on initialisation, every time the configuration is read by your code, or on demand. You can override the caching interval if needed.
  • Handles YAML, JSON and plain text configurations, stored in any supported AppConfig store. Any other content type is returned unprocessed as the Python bytes type.
  • Supports AWS Lambda, Amazon EC2 instances and on-premises use.

Installation

pip install sample-helper-aws-appconfig

Example

from appconfig_helper import AppConfigHelper
from fastapi import FastAPI

appconfig = AppConfigHelper(
    "MyAppConfigApp",
    "MyAppConfigEnvironment",
    "MyAppConfigProfile",
    45  # minimum interval between update checks
)

app = FastAPI()

@app.get("/some-url")
def index():
    if appconfig.update_config():
        print("New configuration received")
    # your configuration is available in the "config" attribute
    return {
        "config_info": appconfig.config
    }

Usage

Please see the AWS AppConfig documentation for details on configuring the service.

Initialising

Start by creating an AppConfigHelper object. You must specify the application name, environment name, and profile (configuration) name. You must also specify the refresh interval, in seconds. AppConfigHelper will not attempt to fetch a new configuration version from the AWS AppConfig service more frequently than this interval. You should set it low enough that your code will receive new configuration promptly, but not so low that it takes too long. The library enforces a minimum interval of 15 seconds.

The configuration is not automatically fetched unless you set fetch_on_init. To have the library fetch the configuration when it is accessed, if it has been more than max_config_age seconds since the last fetch, set fetch_on_read.

If you need to customise the AWS credentials or region, set session to a configured boto3.Session object. Otherwise, the standard boto3 logic for credential/configuration discovery is used.

Reading the configuration

The configuration from AWS AppConfig is available as the config property. Before accessing it, you should call update_config(), unless you specified fetch_on_init or fetch_on_read during initialisation. If you want to force a config fetch, even if the number of seconds specified have not yet passed, call update_config(True).

update_config() returns True if a new version of the configuration was received. If no attempt was made to fetch it, or the configuration received was the same as current one, it returns False. It will raise ValueError if the received configuration data could not be processed (e.g. invalid JSON). If needed, the inner exception for JSON or YAML parsing is available as __context__ on the raised exception.

To read the values in your configuration, access the config property. For JSON and YAML configurations, this will contain the structure of your data. For plain text configurations, this will be a simple string.

The original data received from AppConfig is available in the raw_config property. Accessing this property will not trigger an automatic update even if fetch_on_read is True. The content type field received from AppConfig is available in the content_type property.

For example, with the following JSON in your AppConfig configuration profile:

{
    "hello": "world",
    "data": {
        "is_sample": true
    }
}

you would see the following when using the library:

# appconfig is the instance of the library
>>> appconfig.config["hello"]
"world"
>>> appconfig.config["data"]
{'is_sample': True}

Use in AWS Lambda

AWS AppConfig is best used in Lambda by taking advantage of Lambda Extensions

Security

See CONTRIBUTING for more information.

Licence

This library is licensed under Apache-2.0. See the LICENSE file.

sample-python-helper-aws-appconfig's People

Contributors

amazon-auto avatar dependabot[bot] avatar jamesoff avatar ljmc-github avatar pete-volantautonomy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

sample-python-helper-aws-appconfig's Issues

Need to cut a new version to pip

Hi!

I hit this:
#2

Fresh install:

$ docker run -it --rm python:3 bash
# pip install sample-helper-aws-appconfig
# cat /usr/local/lib/python3.10/site-packages/appconfig_helper/version.py 
VERSION = "2.0.0"

Notably, it doesn't contain 8a2a67e

However, cloning and checking out v2.0.1 does in fact contain 8a2a67e.

Mind pushing a new version to pip?

ValueError: Invalid endpoint: https://appconfigdata..amazonaws.com

Traceback (most recent call last):


File "/usr/local/lib/python3.7/site-packages/appconfig_helper/appconfig_helper.py", line 64, in init
self._client = session.client("appconfigdata")
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 309, in client
config=config,
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 959, in create_client
api_version=api_version,
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 132, in create_client
endpoint_bridge,
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 475, in _get_client_args
endpoint_bridge,
File "/usr/local/lib/python3.7/site-packages/botocore/args.py", line 134, in get_client_args
proxies_config=new_config.proxies_config,
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 402, in create_endpoint
raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://appconfigdata..amazonaws.com

Python Package Version Details
boto3 1.24.46
botocore 1.27.46
sample-helper-aws-appconfig 2.0.3

I have tried with multiple downgraded versions of boto3 and botocore but, this error is constant.

Missing explicit export for mypy --strict

Running mypy --strict when importing this module causes:

demo.py:1: error: Module "appconfig_helper" does not explicitly export attribute "AppConfigHelper"  [attr-defined]

Just need to add __all__ exporting AppConfigHelper at top of file, opening issue before PRing change.

Pip install is not finding a version

pip install sample-helper-aws-appconfig is returning the errors -

ERROR: Could not find a version that satisfies the requirement sample-helper-aws-appconfig (from versions: none)
ERROR: No matching distribution found for sample-helper-aws-appconfig

update_config can cause BadRequestException

The early return in update_config (to deal with empty content when the config hasn't changed)

        if content == b"":
            return False

results in this error

BadRequestException('An error occurred (BadRequestException) when calling the GetLatestConfiguration operation: Request too early')

if the next call to update_config happens soon enough after this call.

The reason the exception is raised by get_latest_configuration is that this ealy return fails to update the variable (self._last_update_time) used to guard against calling before the negotiated minimum poll set via the RequiredMinimumPollIntervalInSeconds arg to start_configuration_session.

Changing the code to

            if content == b"":
                self._last_update_time = time.time()
                return False

would fix this problem.

Can't use with jupyter

It doesn't recognize the module in jupyter notebooks.

!pip install sample-helper-aws-appconfig
Requirement already satisfied: sample-helper-aws-appconfig in ./.venv/lib/python3.8/site-packages (2.0.3)
Requirement already satisfied: PyYAML<7.0,>=6.0 in ./.venv/lib/python3.8/site-packages (from sample-helper-aws-appconfig) (6.0)
Requirement already satisfied: boto3<2.0.0,>=1.20.8 in ./.venv/lib/python3.8/site-packages (from sample-helper-aws-appconfig) (1.26.118)
Requirement already satisfied: botocore<2.0.0,>=1.23.8 in ./.venv/lib/python3.8/site-packages (from sample-helper-aws-appconfig) (1.29.118)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./.venv/lib/python3.8/site-packages (from boto3<2.0.0,>=1.20.8->sample-helper-aws-appconfig) (1.0.1)
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in ./.venv/lib/python3.8/site-packages (from boto3<2.0.0,>=1.20.8->sample-helper-aws-appconfig) (0.6.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./.venv/lib/python3.8/site-packages (from botocore<2.0.0,>=1.23.8->sample-helper-aws-appconfig) (2.8.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in ./.venv/lib/python3.8/site-packages (from botocore<2.0.0,>=1.23.8->sample-helper-aws-appconfig) (1.26.12)
Requirement already satisfied: six>=1.5 in ./.venv/lib/python3.8/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<2.0.0,>=1.23.8->sample-helper-aws-appconfig) (1.16.0)
from appconfig_helper import AppConfigHelper

โŒ ModuleNotFoundError: No module named 'appconfig_helper'

Getting timeouts when running on AppRunner

I'm using the same config locally and on app runner, with the same variables, and yet Gunicorn times out as soon as AppConfig gets called.

E.g. I have this code:

print(f"""Using these variables:
    APPLICATION: {self.APPLICATION}
    ENVIRONMENT: {self.ENVIRONMENT}
    PROFILE: {self.PROFILE}
    UPDATE_CHECK_INTERVAL: {self.UPDATE_CHECK_INTERVAL}
""", flush=True)
self.appconfig: AppConfigHelper = AppConfigHelper(
    self.APPLICATION,
    self.ENVIRONMENT,
    self.PROFILE,
    self.UPDATE_CHECK_INTERVAL,  # minimum interval between update checks (SECONDS)
    fetch_on_init=True,
    fetch_on_read=True
)
print("[BACKEND] Got past app config init... Attempting test", flush=True)

Then i get this in my logs:

02-17-2024 11:34:44 PM Using these variables:
02-17-2024 11:34:44 PM                 APPLICATION: our-app
02-17-2024 11:34:44 PM                 ENVIRONMENT: Staging
02-17-2024 11:34:44 PM                 PROFILE: our-app
02-17-2024 11:34:44 PM                 UPDATE_CHECK_INTERVAL: 45
02-17-2024 11:34:44 PM             
02-17-2024 11:35:15 PM [2024-02-17 23:35:15 +0000] [9] [CRITICAL] WORKER TIMEOUT (pid:11)
02-17-2024 11:35:15 PM [2024-02-17 23:35:15 +0000] [11] [INFO] Worker exiting (pid: 11)
02-17-2024 11:35:16 PM [2024-02-17 23:35:16 +0000] [9] [ERROR] Worker (pid:11) exited with code 1
02-17-2024 11:35:16 PM [2024-02-17 23:35:16 +0000] [9] [ERROR] Worker (pid:11) exited with code 1.
02-17-2024 11:35:16 PM [2024-02-17 23:35:16 +0000] [12] [INFO] Booting worker with pid: 12

And this runs a few times. (This also gets accepted into AppRunner bypassing the "Health Check" which is quite concerning for production, but there you go)

Locally it finishes within a second, so I'm not sure why it hangs on AppRunner.

Any help would be appreciated :)

Start a new session if token expired

The helper as it is right now is not able to recover from expired token (>24h old).

I don't think there is a way to refresh token other than starting a new session and using this initial token.

I would like opinions about possible recovery/prevention methods:

First thought was a recovery method by adding a try/except block around update_config's get_latest_configuration() call to catch invalid tokens and start a fresh session.

        try: 
            response = self._client.get_latest_configuration(
                ConfigurationToken=self._next_config_token
            )
        except BadRequestException:
            self.start_session()  # get a fresh session, might need to close old one not sure
            response = self._client.get_latest_configuration(
                ConfigurationToken=self._next_config_token
            )

Second option would be preventive by adding a time check similar to the if time since update < polling time, but for time since update over 24h.

        if (
            self._next_config_token is None
            or time.time() - self._last_update_time >= 86400
        ):
            self.start_session()

I know this is a rare case, but it happened to me in testing where the service was deployed and testers took over 24h to get to testing, which caused the service to be DOA according to them.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.