cloudbytes-academy / cloudbytes.dev Goto Github PK

Why not simply do sudo cp /usr/bin/python3.10 /usr/bin/python to make python3.10 default?

Discussed in #99

      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/tmp/pip-install-ljjblyio/httptools_182114364e604caa9530ba42c1a43bdc/vendor/llhttp/include -I/tmp/pip-install-ljjblyio/httptools_182114364e604caa9530ba42c1a43bdc/vendor/llhttp/src -I/home/spalfy/vens/backend310/include -I/usr/include/python3.10 -c httptools/parser/parser.c -o build/temp.linux-x86_64-3.10/httptools/parser/parser.o -O2
      httptools/parser/parser.c:20:10: fatal error: Python.h: No such file or directory
         20 | #include "Python.h"
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
```</div>

First off thank you for the great article on how to run selenium in aws lambda. I followed the instructions outlined here and was able to get the lambda to run once.
After the first execution, each subsequent execution of the lambda fails with the error -

errorMessage": "Message: chrome not reachable

If I try again after 20mins or so I'm able to get another successful run which are then followed by unsuccessful runs.

Here is my code -

def handler(event=None, context=None):
    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = "/opt/chrome/chrome"
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-dev-tools")
    chrome_options.add_argument("--no-zygote")
    chrome_options.add_argument("--single-process")
    chrome_options.add_argument("window-size=2560x1440")
    chrome_options.add_argument("--user-data-dir=/tmp/chrome-user-data")
    chrome_options.add_argument("--remote-debugging-port=9222")
    #chrome_options.add_argument("--data-path=/tmp/chrome-user-data")
    #chrome_options.add_argument("--disk-cache-dir=/tmp/chrome-user-data")
    chrome = webdriver.Chrome("/opt/chromedriver", options=chrome_options)

   
    chrome.get("https://www.google.ca/maps/place/Sushi+Maki/@49.2807611,-123.1270736,17z/data=!3m1!5s0x5486717f0a6c35a5:0x2cba7a9c7cdc2aeb!4m7!3m6!1s0x548673d541b27b4d:0x8cdeffd73083c283!8m2!3d49.2807611!4d-123.1248849!9m1!1b1")
    total_reviews = 0
    try:
            time.sleep(1)
            review_count_ele = chrome.find_element(By.XPATH, '//div[@class=\'jANrlb\']')
            total_reviews = int(review_count_ele.text.split('\n')[1].split(' reviews')[0].replace(',', ''))
            print(f'Total reviews to be found: {total_reviews}')

    except Exception as e:
        print(e)
    
    MAX_TIME_OUT = 5
    timeout = 0
    while timeout < MAX_TIME_OUT:
        try:
            menu_bt = chrome.find_element(By.XPATH, '//button[@data-value=\'Sort\']')
            menu_bt.click()
            print('sort menu clicked')
            break
        except Exception as e:
            print('cant find menu button')
            timeout +=1
            time.sleep(1)
    
    timeout = 0
    time.sleep(1)
    SORT_MENU_ITEM_XPATH = '//li[@role=\'menuitemradio\']'
    while timeout < MAX_TIME_OUT:
        try:
            xpath = f'{SORT_MENU_ITEM_XPATH}[2]'
            recent_rating_bt = chrome.find_element(By.XPATH, xpath)
            recent_rating_bt.click()
            print(f'menu item clicked')
            break
        except Exception as e:
            print('cant find menu item')
            timeout +=1
            time.sleep(1)
    
    time.sleep(1)
    reviews_list = chrome.find_elements(By.XPATH, "//div[@data-review-id][@aria-label]")
    reviews_found = len(reviews_list)
    chrome.close()
    chrome.quit()
    return {
        "statusCode": 200,
        "body": json.dumps(
            {
                "message": reviews_found,
            }
        ),
    }

I've tried to increase the memory size to 3008 and that had no impact either.
This works perfectly on my local (sam local invoke).

What's also weird is that it fails at different points on different runs. I suspect chrome is crashing but I have no clue why or how to get around this.

Any suggestions would be greatly appreciated!

Full error message -

{
"errorMessage": "Message: chrome not reachable\n (Session info: headless chrome=103.0.5060.0)\nStacktrace:\n#0 0x563aa11ab759 \n#1 0x563aa1144cf3 \n#2 0x563aa0f23ca7 \n#3 0x563aa0f14b94 \n#4 0x563aa0f156ef \n#5 0x563aa0f17612 \n#6 0x563aa0f0fa78 \n#7 0x563aa0f252e3 \n#8 0x563aa0f8a1ce \n#9 0x563aa0f77ef3 \n#10 0x563aa0f4e27b \n#11 0x563aa0f4f455 \n#12 0x563aa1173870 \n#13 0x563aa11858b0 \n#14 0x563aa11855bc \n#15 0x563aa1185e32 \n#16 0x563aa1174b9b \n#17 0x563aa11860b6 \n#18 0x563aa11664dd \n#19 0x563aa119d888 \n#20 0x563aa119da12 \n#21 0x563aa11b7c2e \n#22 0x7f6436eec44b \n#23 0x7f6435a0356f \n",
"errorType": "WebDriverException",
"requestId": "a7eb93bc-0740-4d96-b665-d720d3cca6bf",
"stackTrace": [
" File "/var/task/app.py", line 93, in handler\n reviews_list = chrome.find_elements(By.XPATH, "//div[@data-review-id][@aria-label]")\n",
" File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 888, in find_elements\n return self.execute(Command.FIND_ELEMENTS, {\n",
" File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in execute\n self.error_handler.check_response(response)\n",
" File "/var/lang/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response\n raise exception_class(message, screen, stacktrace)\n"
]
}

Following this guide results in X11 being uninstalled, among other important components.

After following the instructions, I tried pip install sphinx and got:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pip 22.2.1 (/usr/local/lib/python3.10/dist-packages), Requirement.parse('pip==20.0.2'))

Running a .yml file with a github action in a devcontainer

Hello - I was working through your article on Automate Deployment of Pelican Website to Github pages which uses a vscode devcontainer and everything was going fine until it came to actually running the yaml file, pelican.yml.

I created a .github/workflows/pelican.yml file under the .devcontainer folder and followed the instructions and while the git push command worked OK the instructions in the pelican.yml file never seemed to get triggered. I tried various things such as rerunning the devcontainer from scratch after setting up the file but with no luck.

My github repository with the relevant files is here: and the link to pelican.yml is here.

I'm not so expert at this so might be making some simple error somewhere but if you had any advice it would be much appreciated, and thanks for your article.

Originally posted by @Kujiranoai in #50 (comment)

I was able to install a second Ubuntu using these instructions but then I got messages about it being unsupported. It said to do the upgrades for this version then upgrade to a later version. The upgrade to a later version crashed. I will download a different image and hope I can use that.

I apologize if I made a mistake.

Discussed in #110