sachin-sankar / swiftshadow Goto Github PK

View Code? Open in Web Editor NEW

66.0 1.0 4.0 1.25 MB

Free IP Proxy rotator library for python

Home Page: https://sachin-sankar.github.io/swiftshadow/

License: GNU General Public License v3.0

Python 100.00%

free library proxy proxy-checker proxy-list proxy-scraper python python3

swiftshadow's Introduction

Hi, I'm Sachin Sankar 👋

I'm a programming enthusiast from India who loves to write code and build cool projects. I have a passion for programming languages, algorithms, and data structures, and I'm always looking for ways to improve my skills.

Currently, I'm learning data science and dynamic SVG generation, and I'm working on SwiftShadow. I'm also interested in exploring new technologies and experimenting with different programming languages and frameworks.

I also make quality datasets for the open source community and love to scrape the web or go data mining.

Apart from programming, I'm also a space technology enthusiast and an Elon Musk fan.

🔭 I’m currently working on

swiftshadow: Free IP Proxy rotator library for python

🌱 I’m currently learning

Data science
Dynamic SVG generation

💬 Ask me about

Python
Data science
Web scraping
General tech
Space technology

🤝 Open to collaboration

I'm open to collaborating on interesting projects. You can reach me through the links on my Bento page.

🔧 Languages and Tools

swiftshadow's People

Contributors

Stargazers

Watchers

Forkers

saeshu shrirajhegde zirpian markvanjislaaik

swiftshadow's Issues

Lambda on aws unable to proceed due to Read-only file system

Hi! On my local environment works fine, but when I try to run it on my lambda I get this

Function Logs
START RequestId: 5ea8d3a7-8f08-4f9b-b9c0-10a6c3c94cc7 Version: $LATEST
07/05/2023 20:35:43 - [swiftshadow] - ERROR : No cache found. Cache will be created after update
07/05/2023 20:35:43 - [swiftshadow] - INFO : Updating proxies from Proxyscrape
[ERROR] OSError: [Errno 30] Read-only file system: '.swiftshadow.json'
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 13, in lambda_handler
    swift = Proxy()
  File "/opt/python/swiftshadow/swiftshadow.py", line 49, in __init__
    self.update()
  File "/opt/python/swiftshadow/swiftshadow.py", line 100, in update
    with open(".swiftshadow.json", "w") as file:END RequestId: 5ea8d3a7-8f08-4f9b-b9c0-10a6c3c94cc7
REPORT RequestId: 5ea8d3a7-8f08-4f9b-b9c0-10a6c3c94cc7	Duration: 6597.86 ms	Billed Duration: 6598 ms	Memory Size: 128 MB	Max Memory Used: 65 MB	Init Duration: 813.64 ms

According to stackoverflow:

Only /tmp seems to be writable in AWS Lambda.
Is there a way to pass a folder path to your library instance?

Implement a "QuickProxy" class

implement a new class QuickProxy that will get a single proxy as soon as possible

Auto Rotate not working

Following error generated while auto rotate

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

bug: failed instalation using poetry on ubuntu 22

(src-py3.11) vikyw@vikyw-X456UR:~/Desktop/chat-based-financial-gpt/server$ poetry add swiftshadow
Using version ^1.0.0 for swiftshadow

Updating dependencies
Resolving dependencies... (4.4s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing swiftshadow (1.0.0): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
    File "/home/vikyw/.local/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/vikyw/.local/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/vikyw/.local/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmp94df84ec/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmp94df84ec/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
      self.run_setup()
    File "/tmp/tmp94df84ec/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 480, in run_setup
      super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
    File "/tmp/tmp94df84ec/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
      exec(code, locals())
    File "<string>", line 18, in <module>
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named 'requests'
  

  at ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with swiftshadow (1.0.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "swiftshadow (==1.0.0)"'.```

Implement a basic logger

Expiry check invalid

Python 3.11.3 (main, Apr  7 2023, 00:39:07) [Clang 14.0.7 (https://android.googlesource.com/toolchain/llvm-project 4c603efb0 on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from swiftshadow.swiftshadow import Proxy
[ swiftshadow ] ERROR : No cache found. Cache will be created after update
[ swiftshadow ] INFO : Updating proxies from Proxyscrape
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/swiftshadow/swiftshadow.py", line 134, in <module>
    print(a.proxy())
          ^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/swiftshadow/swiftshadow.py", line 122, in proxy
    if cache.checkExpiry(self.expiry):
                         ^^^^^^^^^^^
AttributeError: 'Proxy' object has no attribute 'expiry'
>>>

Proxy update function name inconsistent in main file

Add note about proxy update function

Add a admonition note about how it wil take some time to update proxies.

Caching mechanism is inconsistent

how to use

hi, how to use this with proxychain or oxdork for example

swiftshadow takes over logging

I do not know how but since I added swiftshadow to my codebase my own logging stopped working.
In the file where I import swiftshadow.swiftshadow I do

logging.getLogger().handlers.clear() # <-- first attempt at a workaround
logging.basicConfig(
    filename=script_dir + "/log/mylogfile.log",
    level=debug_level,
    format=(
        "%(asctime)s %(levelname)s " 
        "%(message)s"
    ),
    datefmt="%Y%m%d %H:%M:%S",
)
logger = logging.getLogger("mymainlogger")
logger.info("I am info logging!") # this usually ends up in mylogfile.log

As mentioned this setup worked fine before using swiftshadow.
Now my mylogfile.log stays empty and all I see is swiftshadow output on stdout.

Any idea how I can make my logger work again?
I tried finding your logger with logging.root.manager.loggerDict by to no avail :(

Add "Using with" section in docs

Add documentation on how to use swiftshadow with

Requests
Proxychains

Import "swiftshadow.swiftshadow" could not be resolved

Hi! I'm having an issue where after installing the module it's not found. It's the only with I'have had that issue.
Using Python 3.10 on Windows 11

(venv) PS E:\Github Projects\last-mass> pip install swiftshadow
Collecting swiftshadow
  Downloading swiftshadow-0.2.3-py3-none-any.whl (20 kB)
Requirement already satisfied: requests in e:\github projects\last-mass\venv\lib\site-packages (from swiftshadow) (2.30.0)
Requirement already satisfied: charset-normalizer<4,>=2 in e:\github projects\last-mass\venv\lib\site-packages (from requests->swiftshadow) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in e:\github projects\last-mass\venv\lib\site-packages (from requests->swiftshadow) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in e:\github projects\last-mass\venv\lib\site-packages (from requests->swiftshadow) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in e:\github projects\last-mass\venv\lib\site-packages (from requests->swiftshadow) (2023.5.7)
Installing collected packages: swiftshadow
Successfully installed swiftshadow-0.2.3

Pip show

(venv) PS E:\Github Projects\last-mass> pip list
Package            Version
------------------ --------
beautifulsoup4     4.12.2
boto3              1.26.129
botocore           1.29.129
certifi            2023.5.7
charset-normalizer 3.1.0
idna               3.4
jmespath           1.0.1
pip                22.3.1
python-dateutil    2.8.2
requests           2.30.0
s3transfer         0.6.1
setuptools         65.5.0
six                1.16.0
soupsieve          2.4.1
swiftshadow        0.2.3
urllib3            1.26.15

Implement a Request Handler

Implement a new class that when provided with a url requests the url using a proxy validate the request with a function and if either the check or the request itself fails retries the request with a new proxy.