Giter VIP home page Giter VIP logo

aionyphe's Introduction

aionyphe

Asynchronous API SDK and command line interface written in Python for Onyphe. This software is cross-platform and works on Linux, Mac and Windows.

Using asyncio enables the user to perform concurrent requests without the need of threading or subprocessing. Be careful not to trigger the rate limiting protection though.

Setup

Setup is almost the same for Linux, Darwin and Windows.

# assuming 'python' is python 3 executable
python -m venv venv
source venv/bin/activate
pip install git+https://github.com/koromodako/aionyphe
# next line is for linux and darwin only
pip install uvloop

Testing

There is no automated testing for now. Manual testing was performed using Python 3.10.12 on Ubuntu 22.04. Assume all Python versions above 3.10 are supported.

Documentation

There is no documentation for now but most of the code is documented.

Coding rules

Code is formatted using black and Pylint is used to ensure most of the code is PEP8 compliant and error free.

API Usage

Direct connection

from json import dumps
from asyncio import run
from getpass import getpass
from aionyphe import OnypheAPIClient, client_session

async def main():
    oql = 'category:synscan ip:8.8.8.8'
    api_key = getpass("Enter Onyphe API key: ")
    async with client_session(api_key) as client:
        api_client = OnypheAPIClient(client=client)
        async for _, result in api_client.export(oql):
            print(dumps(result))

if __name__ == '__main__':
    run(main())

Proxy connection

from json import dumps
from asyncio import run
from getpass import getpass
from aionyphe import OnypheAPIClient, OnypheAPIClientProxy, client_session

async def main():
    oql = 'category:synscan ip:8.8.8.8'
    api_key = getpass("Enter Onyphe API key: ")
    proxy = OnypheAPIClientProxy(
        scheme='http', host='squid.domain.tld', port=3128
    )
    async with client_session(api_key) as client:
        api_client = OnypheAPIClient(client=client, proxy=proxy)
        async for _, result in api_client.export(oql):
            print(dumps(result))

if __name__ == '__main__':
    run(main())

Get a specific result page

from json import dumps
from asyncio import run
from getpass import getpass
from aionyphe import OnypheAPIClient, client_session

async def main():
    oql = 'category:datascan domain:google.com'
    api_key = getpass("Enter Onyphe API key: ")
    async with client_session(api_key) as client:
        api_client = OnypheAPIClient(client=client)
        async for _, result in api_client.search(oql, page=2):
            print(dumps(result))

if __name__ == '__main__':
    run(main())

A helper to iterate over pages

from json import dumps
from asyncio import run
from getpass import getpass
from aionyphe import OnypheAPIClient, client_session, iter_pages

async def main():
    oql = 'category:datascan domain:google.com'
    api_key = getpass("Enter Onyphe API key: ")
    async with client_session(api_key) as client:
        api_client = OnypheAPIClient(client=client)
        async for _, result in iter_pages(api_client.search, [oql], 2, 4):
            print(dumps(result))

if __name__ == '__main__':
    run(main())

Command Line Interface

Usage

This client does not support as much features as the original Onyphe client written in Perl but it does allow the user to pipe the output to another tool keeping the JSON-based pipe interface.

# get usage information
aionyphe -h
# get 'export' command usage
aionyphe export -h
# export data for given oql query
aionyphe export 'category:synscan ip:8.8.8.8'
# get your public ip address
aionyphe myip
# get information about your user account
aionyphe user | jq
# show pages 2 to 4 for search query
aionyphe search --first 2 --last 4 'category:datascan domain:google.com'

Configuration File (optional)

aionyphe client can load configuration from a file depending on the operating system being used.

OS Configuration file
Linux /home/{username}/.aionyphe
Darwin /Users/{username}/.aionyphe
Windows C:\Users\{username}\.aionyphe

This configuration file shall contain a JSON object with optional key/value pairs taken from this table :

Key Value
scheme "http" or "https"
host Onyphe API hostname
port Port as an integer in range 0 -> 65535
version Onyphe API version
api_key Onyphe API key (warning: plaintext secret stored on disk!)
proxy_scheme "http" or "https"
proxy_host Proxy hostname
proxy_port Proxy port as an integer in range 0 -> 65535
proxy_username Proxy authentication username
proxy_password Proxy authentication password (warning: plaintext secret stored on disk!)
total Maximal number of seconds for each request made to the API
connect Maximal number of seconds for acquiring a connection from pool
sock_read Maximal number of seconds for reading a portion of data from a peer
sock_connect Maximal number of seconds for connecting to a peer for a new connection

Note: command line arguments override configuration file values.

aionyphe's People

Contributors

koromodako avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar

aionyphe's Issues

Search does not correctly handle OR operator

The OQL OR operator ? is not correctly handled byn aionyphe and need to be manually encoded.

Example with the OQL category:datascan ?hostname:node14.onyphe.io ?hostname:node14main.onyphe.io expecting 3 results.

Using the following code:

from asyncio import run
from aionyphe import OnypheAPIClientSession

async def main():
    oql1 = 'category:datascan ?hostname:node14.onyphe.io ?hostname:node14main.onyphe.io'
    oql2 = 'category:datascan %3Fhostname:node14.onyphe.io %3Fhostname:node14main.onyphe.io'
    kwargs = dict(
        api_key="XXX",
    )
    async with OnypheAPIClientSession(**kwargs) as client:
        async for metadata, _ in client.search(oql1, page=1):
            print("OQL1 :", metadata)
            break
        async for metadata, _ in client.search(oql2, page=1):
            print("OQL2 :", metadata)
            break

if __name__ == '__main__':
    run(main())

Getting the following results:

OQL1 : {'count': 10, 'error': 0, 'max_page': 1000, 'myip': 'XXX', 'page': '1', 'status': 'ok', 'text': 'Success', 'took': 2.088, 'total': 660409285}
OQL2 : {'count': 3, 'error': 0, 'max_page': 1, 'myip': 'XXX', 'page': '1', 'status': 'ok', 'text': 'Success', 'took': 0.021, 'total': 3}

Descriptive message not visible in Onyphe API error

Is it possible to display the descriptive message returned by the API during error 400 ?

Sample message from the API documentation :

{
  "error": 3,
  "text": "Invalid API key format",
  "myip": "<redacted>",
  "status": "nok"
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.