microsoft / msticnb Goto Github PK

Several errors mostly in url_summary, ip.py and host_network_summary.

We have a bunch of systemic pylint errors - we need to update pylint and other linter config to match msticpy.

mypy

msticnb/nb/azsent/host/host_logons_summary.py:499: error: Incompatible return value type (got "Union[str, Any, None]", expected "str")
msticnb/nb/azsent/host/host_logons_summary.py:503: error: Incompatible return value type (got "Union[str, Any, None]", expected "str")
msticnb/nb/azsent/host/host_logons_summary.py:510: error: Incompatible return value type (got "Union[str, Any, None]", expected "str")
msticnb/nb/azsent/url/url_summary.py:13: error: Skipping analyzing "whois": module is installed, but missing library stubs or py.typed marker
msticnb/nb/azsent/url/url_summary.py:13: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
msticnb/nb/azsent/url/url_summary.py:80: error: Incompatible types in assignment (expression has type "None", variable has type "List[Any]")
Found 5 errors in 2 files (checked 39 source files)

prospector

msticnb/nb/azsent/url/url_summary.py
  Line: 57
    pydocstyle: D200 / One-line docstring should fit on one line with quotes (found 4)
  Line: 68
    pydocstyle: D200 / One-line docstring should fit on one line with quotes (found 4)
  Line: 224
    pylint: invalid-name / Variable name "f" doesn't conform to snake_case naming style (col 49)
  Line: 320
    pylint: missing-function-docstring / Missing function or method docstring
    pydocstyle: D103 / Missing docstring in public function
  Line: 321
    pylint: invalid-name / Variable name "s" doesn't conform to snake_case naming style (col 4)
  Line: 325
    pylint: missing-function-docstring / Missing function or method docstring
    pydocstyle: D103 / Missing docstring in public function

msticnb/nblib/ti.py
  Line: 6
    pydocstyle: D400 / First line should end with a period (not 't')
    pydocstyle: D415 / First line should end with a period, question mark, or exclamation point (not 't')
  Line: 24
    pydocstyle: D417 / Missing argument descriptions in the docstring (argument(s) col are missing descriptions in 'get_ti_results' docstring)
  Line: 58
    pydocstyle: D400 / First line should end with a period (not 'e')
    pydocstyle: D415 / First line should end with a period, question mark, or exclamation point (not 'e')

prospector.yml
  Line: 19
    profile-validator: deprecated-tool-code / pep8 tool has been renamed to 'pycodestyle'. Using pep8 to configure the tool will be removed in prospector 2.0+.
  Line: 25
    profile-validator: deprecated-tool-code / pep257 tool has been renamed to 'pydocstyle'. The name pep257 will be removed in prospector 2.0+.

Pylint

    <testsuite>
            <testcase>
			<failure type="failure" message="invalid-name">C0103:Variable name &quot;f&quot; doesn't conform to snake_case naming style
msticnb/nb/azsent/url/url_summary.py:224:49:with open(&quot;screenshot.png&quot;, &quot;wb&quot;) as f:</failure>
			<system-out>msticnb/nb/azsent/url/url_summary.py:224:49:with open(&quot;screenshot.png&quot;, &quot;wb&quot;) as f:</system-out>
			<system-err>C0103:Variable name &quot;f&quot; doesn't conform to snake_case naming style
msticnb/nb/azsent/url/url_summary.py:224:49:with open(&quot;screenshot.png&quot;, &quot;wb&quot;) as f:</system-err>
		</testcase>
		<testcase name="msticnb.nb.azsent.url.url_summary:104:4" classname="pylint" class="refactor" file="msticnb/nb/azsent/url/url_summary.py" line="104">
			<failure type="failure" message="too-many-statements">R0915:Too many statements (85/50)
msticnb/nb/azsent/url/url_summary.py:104:4:def run(  # noqa:MC0001</failure>
			<system-out>msticnb/nb/azsent/url/url_summary.py:104:4:def run(  # noqa:MC0001</system-out>
			<system-err>R0915:Too many statements (85/50)
msticnb/nb/azsent/url/url_summary.py:104:4:def run(  # noqa:MC0001</system-err>
		</testcase>
		<testcase name="msticnb.nb.azsent.url.url_summary:320:0" classname="pylint" class="convention" file="msticnb/nb/azsent/url/url_summary.py" line="320">
			<failure type="failure" message="missing-function-docstring">C0116:Missing function or method docstring
msticnb/nb/azsent/url/url_summary.py:320:0:def entropy(data):</failure>
			<system-out>msticnb/nb/azsent/url/url_summary.py:320:0:def entropy(data):</system-out>
			<system-err>C0116:Missing function or method docstring
msticnb/nb/azsent/url/url_summary.py:320:0:def entropy(data):</system-err>
		</testcase>
		<testcase name="msticnb.nb.azsent.url.url_summary:321:4" classname="pylint" class="convention" file="msticnb/nb/azsent/url/url_summary.py" line="321">
			<failure type="failure" message="invalid-name">C0103:Variable name &quot;s&quot; doesn't conform to snake_case naming style
msticnb/nb/azsent/url/url_summary.py:321:4:s, lens = Counter(data), np.float(len(data))</failure>
			<system-out>msticnb/nb/azsent/url/url_summary.py:321:4:s, lens = Counter(data), np.float(len(data))</system-out>
			<system-err>C0103:Variable name &quot;s&quot; doesn't conform to snake_case naming style
msticnb/nb/azsent/url/url_summary.py:321:4:s, lens = Counter(data), np.float(len(data))</system-err>
		</testcase>
		<testcase name="msticnb.nb.azsent.url.url_summary:325:0" classname="pylint" class="convention" file="msticnb/nb/azsent/url/url_summary.py" line="325">
			<failure type="failure" message="missing-function-docstring">C0116:Missing function or method docstring
msticnb/nb/azsent/url/url_summary.py:325:0:def color_domain_record_cells(val):</failure>
			<system-out>msticnb/nb/azsent/url/url_summary.py:325:0:def color_domain_record_cells(val):</system-out>
			<system-err>C0116:Missing function or method docstring
msticnb/nb/azsent/url/url_summary.py:325:0:def color_domain_record_cells(val):</system-err>
		</testcase>
		<testcase name="msticnb.nb.azsent.url.url_summary:0:0" classname="pylint" file="msticnb/nb/azsent/url/url_summary.py">
			<system-out>All checks passed for: msticnb/nb/azsent/url/url_summary.py</system-out>
		</testcase>
	</testsuite>

IP addresses with Passive DNS results cause and exception in .nblts.azsent.network.IpAddressSummary()

Repro steps:

• Run the notebooklet with an IP address that returns a passive DNS result - above exception caused after running with value = "185.99.3.60"

The YAML metadata files containing the metadata are missing from the PIP package.
Downloaded from https://pypi.org/project/msticnb/#files

To Reproduce
ip_ent_nb = nb.nblts.azsent.network.IpAddressSummary()
ip_ent_nb.default_options()
[]

Expected behavior
ip_ent_nb = nb.nblts.azsent.network.IpAddressSummary()
ip_ent_nb.default_options()
['geoip', 'alerts', 'host_logons', 'related_accounts', 'device_info', 'device_network']

I'm receiving this error when I'm importing the module and execute the init.

import msticnb as nb
nb.init(query_provider="AzureSentinel")

This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.

AttributeError Traceback (most recent call last)
in
1 import msticnb as nb
----> 2 nb.init(query_provider="AzureSentinel")

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticnb\data_providers.py in init(query_provider, providers, **kwargs)
443
444 """
--> 445 d_provs = DataProviders(query_provider, providers, **kwargs)
446 print(f"Loaded providers: {', '.join(d_provs.providers.keys())}")
447 msticnb = sys.modules["msticnb"]

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticnb\data_providers.py in call(self, *args, **kwargs)
57 or self.instance.args != args
58 ):
---> 59 self.instance = self.wrapped_cls(*args, **kwargs)
60 self.instance.kwargs = kwargs
61 self.instance.args = args

c:\Users\dp\Jupyter\MyJupyter\venv\lib\site-packages\msticnb\data_providers.py in init(self, query_provider, providers, **kwargs)
153 for provider in sorted(self.provider_names):
154 try:
--> 155 self.add_provider(provider, **kwargs)
156 except MsticnbDataProviderError as err:
157 print(f"Data provider {provider} could not be added.")

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticnb\data_providers.py in add_provider(self, provider, **kwargs)
220 new_provider = self._query_prov(provider, prov_def, **kwargs)
221 else:
--> 222 new_provider = self._no_connect_prov(provider, prov_def, **kwargs)
223 else:
224 raise MsticnbDataProviderError(f"Provider {provider} not recognized.")

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticnb\data_providers.py in _no_connect_prov(self, provider, provider_defn, **kwargs)
343 prov_args = provider_defn.get_config()
344 # Instatiate the provider
--> 345 return provider_defn.prov_class(prov_args)
346
347 # Helper methods

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticpy\sectools\tilookup.py in init(self, primary_providers, secondary_providers, providers)
84 self.add_provider(prov, primary=False)
85 if not (primary_providers or secondary_providers):
---> 86 self._load_providers()
87
88 self._all_providers = ChainMap(self._secondary_providers, self._providers)

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticpy\sectools\tilookup.py in _load_providers(self)
243 def _load_providers(self):
244 """Load provider classes based on config."""
--> 245 prov_settings = get_provider_settings()
246
247 for provider_entry, settings in prov_settings.items():

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticpy\common\provider_settings.py in get_provider_settings(config_section)
87 name=provider,
88 description=item_settings.get("Description"),
---> 89 args=_get_setting_args(
90 config_section=config_section,
91 provider_name=provider,

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticpy\common\provider_settings.py in _get_setting_args(config_section, provider_name, prov_args)
124 "subscriptionid": "subscription_id",
125 }
--> 126 return _get_settings(
127 config_section=config_section,
128 provider_name=provider_name,

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticpy\common\provider_settings.py in _get_settings(config_section, provider_name, conf_group, name_map)
177 elif isinstance(arg_value, dict):
178 try:
--> 179 setting_dict[target_name] = _fetch_setting(
180 config_section, provider_name, arg_name, arg_value
181 ) # type: ignore

c:\Users\dp\MyJupyter\venv\lib\site-packages\msticpy\common\provider_settings.py in _fetch_setting(config_section, provider_name, arg_name, config_setting)
211 )
212 config_path = [config_section, provider_name, "Args", arg_name]
--> 213 sec_func = _SECRETS_CLIENT.get_secret_accessor( # type:ignore
214 ".".join(config_path)
215 )

AttributeError: 'NoneType' object has no attribute 'get_secret_accessor'

To Reproduce
Steps to reproduce the behavior:

1. Open VScode
2. Create venv
   3 Add Microsoft Python Extension
   4 Create file x.ipynb
   5 Execute on the jupyter cell:
   import msticnb as nb
   nb.init(query_provider="AzureSentinel")
   6 Authenticate against Azure
   7 Error.

Expected behavior
Read the key from msticpyconfig.yaml and continue with init.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• VSCode
  Version: 1.48.0 (user setup)
  Commit: db40434f562994116e5b21c24015a2e40b2504e6
  Date: 2020-08-13T07:50:42.600Z
  Electron: 7.3.2
  Node.js: 12.8.1
  V8: 7.8.279.23-electron.0

• Python 3.8.3

• OS: Windows_NT x64 10.0.19041

• Chrome: 78.0.3904.130

Please let me know if any other info is needed.

Environment details:

• Linux (ubuntu 20.04 - Standard D2s v3 (2 vcpus, 8 GiB memory))
• Python 3.8.10
• Installed all pre-reqs for running the Azure Automated Notebooks from https://github.com/Azure/Azure-Sentinel-Notebooks. I also added/installed msticpy[ml] & gir1.2-secret-1.

Steps to repro:
After I launch AutomatedNotebooks-Manager.ipynb using papermill, the AutomatedNotebooks-IncidentTriage.ipynb is triggered and at some point the msticnb should be initialized. The code looks like this:

# Set up notebooklets
nb.init(qry_prov)
timespan = TimeSpan(start=datetime.now() - timedelta(days=7))

This is where I can the following error:

~/.local/lib/python3.8/site-packages/msticpy/common/provider_settings.py in _get_settings(config_section, provider_name, conf_group, name_map)
    176     if not conf_group:
    177         return ProviderArgs()
--> 178     setting_dict: ProviderArgs = ProviderArgs(conf_group.copy())
    179 
    180     for arg_name, arg_value in conf_group.items():

AttributeError: 'str' object has no attribute 'copy'

NOTE: the error message is longer than this, please let me know if I should paste the entire message.

Several of the notebooks and modules are not being included in the RTD information because either:

• there is an import error in the module
• the document hasn't been included in the ToC/parent documents

Additionally we need to implement the same changes as msticpy to mock packages
such as sklearn and matplotlib

Is your feature request related to a problem? Please describe.
The Network Security Group Flow Logs analysis tool is not very advanced and comfortable. I think it would be very useful to have a functionality to read and analyze NSG Flog logs directly from the Azure Storage Account, being able to carry out the advanced analyzes already existing in the library, as well as the existing data enrichment mechanisms.
https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview

Describe the solution you'd like
I propose an object to research NGS FLow Logs stored in a BlobStorage. This object could have some funtions:

• Azure data summary (read data in blobstorage and create a summary, given the possibility to filter out non-relevant data)
• Download data from blobstorage, having the possibilitie to download all data in all blobs that fit in datetime filter or content filter and creating a result DataFrame.
• Perform data analysis with graphs or statistical analisys.

Describe alternatives you've considered
I have developed a notebook that performs these actions more or less. The code is not very sophisticated (it is only a first approach).

Additional context
https://github.com/lucky-luk3/Infosec_Notebooks/blob/main/NGS_Log_Analysis-Public.ipynb

Notebooklets that expect "data" input fail when used as PivotFunction

To Reproduce
Steps to reproduce the behavior:

1. load notebooklets
2. load pivot
3. try entities.Host.nblt.logon_session_rarity(data=df)
4. See error

Expected behavior
We should distinguish (maybe in the Yaml metadata) which notebooklets expect data as an input and which a single string value.

Use of the ip_summary notebooklet with an external IP address forces you to use XForce. If you don't have XForce set up it fails.
It would be good to make this feature configurable or skipped if XForce not configured.