C:\Users\caps\.vitualenvs\crawler\Scripts\python.exe main.py --platform dy --lt qrcode 
2023-07-26  22:58:50 MediaCrawler ERROR login dialog box does not pop up automatically, error: Timeout 10000ms exceeded.
=========================== logs ===========================
waiting for locator("xpath=//div[@id='login-pannel']") to be visible
============================================================ 
2023-07-26  22:58:50 MediaCrawler INFO login dialog box does not pop up automatically, we will manually click the login button 
Traceback (most recent call last):
  File "C:\Users\caps\PycharmProjects\MediaCrawler\media_platform\douyin\login.py", line 90, in popup_login_dialog
    await self.context_page.wait_for_selector(dialog_selector, timeout=1000 * 10)
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\async_api\_generated.py", line 8266, in wait_for_selector
    await self._impl_obj.wait_for_selector(
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_page.py", line 368, in wait_for_selector
    return await self._main_frame.wait_for_selector(**locals_to_params(locals()))
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_frame.py", line 322, in wait_for_selector
    await self._channel.send("waitForSelector", locals_to_params(locals()))
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_connection.py", line 461, in wrap_api_call
    return await cb()
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_connection.py", line 96, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 10000ms exceeded.
=========================== logs ===========================
waiting for locator("xpath=//div[@id='login-pannel']") to be visible
============================================================

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\caps\PycharmProjects\MediaCrawler\main.py", line 47, in <module>
    asyncio.run(main())
  File "C:\Users\caps\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\caps\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 649, in run_until_complete
    return future.result()
  File "C:\Users\caps\PycharmProjects\MediaCrawler\main.py", line 42, in main
    await crawler.start()
  File "C:\Users\caps\PycharmProjects\MediaCrawler\media_platform\douyin\core.py", line 62, in start
    await login_obj.begin()
  File "C:\Users\caps\PycharmProjects\MediaCrawler\media_platform\douyin\login.py", line 45, in begin
    await self.popup_login_dialog()
  File "C:\Users\caps\PycharmProjects\MediaCrawler\media_platform\douyin\login.py", line 95, in popup_login_dialog
    await login_button_ele.click()
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\async_api\_generated.py", line 15419, in click
    await self._impl_obj.click(
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_locator.py", line 160, in click
    return await self._frame.click(self._selector, strict=True, **params)
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_frame.py", line 489, in click
    await self._channel.send("click", locals_to_params(locals()))
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_connection.py", line 61, in send
    return await self._connection.wrap_api_call(
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_connection.py", line 461, in wrap_api_call
    return await cb()
  File "C:\Users\caps\.vitualenvs\crawler\lib\site-packages\playwright\_impl\_connection.py", line 96, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded.
=========================== logs ===========================
waiting for locator("xpath=//p[text() = '登录']")
  locator resolved to <p class="lqiPv8cB">登录</p>
attempting click action
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #1
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #2
  waiting 20ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #3
  waiting 100ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #4
  waiting 100ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #5
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #6
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #7
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #8
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #9
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #10
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #11
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #12
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #13
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #14
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #15
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #16
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #17
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #18
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #19
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #20
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #21
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #22
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #23
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #24
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #25
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #26
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #27
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #28
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #29
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #30
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #31
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #32
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #33
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #34
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #35
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #36
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #37
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #38
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #39
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #40
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #41
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #42
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #43
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #44
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #45
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #46
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #47
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #48
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #49
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #50
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #51
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #52
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #53
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #54
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #55
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #56
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #57
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #58
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #59
  waiting 500ms
  waiting for element to be visible, enabled and stable
  element is visible, enabled and stable
  scrolling into view if needed
  done scrolling
  <div id="captcha_container">…</div> intercepts pointer events
retrying click action, attempt #60
  waiting 500ms
============================================================

进程已结束,退出代码1

def show_qrcode(qr_code: str):
    """parse base64 encode qrcode image and show it"""
    qr_code = qr_code.split(",")[1]
    qr_code = base64.b64decode(qr_code)
    image = Image.open(BytesIO(qr_code))

    # Add a square border around the QR code and display it within the border to improve scanning accuracy.
    width, height = image.size
    new_image = Image.new('RGB', (width + 20, height + 20), color=(255, 255, 255))
    new_image.paste(image, (10, 10))
    draw = ImageDraw.Draw(new_image)
    draw.rectangle((0, 0, width + 19, height + 19), outline=(0, 0, 0), width=1)
    new_image.show()

login_flag: bool = await self.check_login_state(no_logged_in_session)
        if not login_flag:
            # wait 2s
            # login_flag: bool = await self.check_login_state(no_logged_in_session)

class Meta:
    abstract = True

class Meta:
    table = "douyin_aweme"
    table_description = "抖音视频"

def __str__(self):
    return f"{self.aweme_id} - {self.title}"

        df_old = pd.read_excel(xls, sheet_name=sheet_name, engine='openpyxl')

        # 使用 pd.concat 替代 append 方法
        df_new = pd.DataFrame([data])
        df_combined = pd.concat([df_old, df_new], ignore_index=True)

        df_combined.to_excel(file_path, sheet_name=sheet_name, index=False, engine='openpyxl')

class Meta:
    table = "douyin_aweme_comment"
    table_description = "抖音视频评论"

def __str__(self):
    return f"{self.comment_id} - {self.content}"

Begin search xiaohongshu keywords:  健身
Traceback (most recent call last):
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_urlparse.py", line 339, in normalize_port
    port_as_int = int(port)
ValueError: invalid literal for int() with base 10: ':1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 35, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "main.py", line 30, in main
    await crawler.start()
  File "/home/MediaCrawler/media_platform/xhs/core.py", line 70, in start
    note_res = await self.search_posts()
  File "/home/MediaCrawler/media_platform/xhs/core.py", line 134, in search_posts
    posts_res = await self.xhs_client.get_note_by_keyword(keyword=self.keywords)
  File "/home/MediaCrawler/media_platform/xhs/client.py", line 110, in get_note_by_keyword
    return await self.post(uri, data)
  File "/home/MediaCrawler/media_platform/xhs/client.py", line 77, in post
    return await self.request(method="POST", url=f"{self._host}{uri}",
  File "/home/MediaCrawler/media_platform/xhs/client.py", line 53, in request
    async with httpx.AsyncClient(proxies=self.proxies) as client:
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_client.py", line 1408, in __init__
    self._mounts: typing.Dict[URLPattern, typing.Optional[AsyncBaseTransport]] = {
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_client.py", line 1409, in <dictcomp>
    URLPattern(key): None
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_utils.py", line 397, in __init__
    url = URL(pattern)
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_urls.py", line 113, in __init__
    self._uri_reference = urlparse(url, **kwargs)
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_urlparse.py", line 246, in urlparse
    parsed_port: typing.Optional[int] = normalize_port(port, scheme)
  File "/home/MediaCrawler/env/lib/python3.8/site-packages/httpx/_urlparse.py", line 341, in normalize_port
    raise InvalidURL("Invalid port")
httpx.InvalidURL: Invalid port

$ python main.py --platform xhs --lt qrcode
Traceback (most recent call last):
  File "/Users/xxx/tp-code/MediaCrawler/main.py", line 8, in <module>
    from media_platform.douyin import DouYinCrawler
  File "/Users/xxx/tp-code/MediaCrawler/media_platform/douyin/__init__.py", line 1, in <module>
    from .core import DouYinCrawler
  File "/Users/xxx/tp-code/MediaCrawler/media_platform/douyin/core.py", line 17, in <module>
    from .login import DouYinLogin
  File "/Users/xxx/tp-code/MediaCrawler/media_platform/douyin/login.py", line 6, in <module>
    import aioredis
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/__init__.py", line 1, in <module>
    from aioredis.client import Redis, StrictRedis
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/client.py", line 32, in <module>
    from aioredis.connection import (
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/connection.py", line 33, in <module>
    from .exceptions import (
  File "/Users/xxx/miniconda3/lib/python3.11/site-packages/aioredis/exceptions.py", line 14, in <module>
    class TimeoutError(asyncio.TimeoutError, builtins.TimeoutError, RedisError):
TypeError: duplicate base class TimeoutError

2023-09-23  22:30:23 MediaCrawler INFO Begin create browser context ... 
2023-09-23  22:30:25 MediaCrawler INFO Begin create xiaohongshu API client ... 
2023-09-23  22:30:25 MediaCrawler INFO Begin to ping xhs... 
2023-09-23  22:30:25 httpx INFO HTTP Request: POST https://edith.xiaohongshu.com/api/sns/web/v1/search/notes "HTTP/1.1 200 OK" 
2023-09-23  22:30:25 MediaCrawler ERROR Ping xhs failed: 您当前登录的账号没有权限访问, and try to login again... 
2023-09-23  22:30:25 MediaCrawler INFO Begin login xiaohongshu ... 
2023-09-23  22:30:25 MediaCrawler INFO Begin login xiaohongshu by qrcode ... 
2023-09-23  22:30:25 MediaCrawler INFO waiting for scan code login, remaining time is 20s 
<PIL.Image.Image image mode=RGB size=175x175 at 0x7F693C062E00>