Giter VIP home page Giter VIP logo

python-cybox's Introduction

python-cybox

A Python library for parsing, manipulating, and generating Cyber Observable eXpression (CybOX™) v2.1.0 content.

Source:https://github.com/CybOXProject/python-cybox
Documentation:https://cybox.readthedocs.io/
Information:https://cyboxproject.github.io/
Download:https://pypi.python.org/pypi/cybox/

Build Status Code Health Version

Overview

A primary goal of the python-cybox library is to remain faithful to both the CybOX standard and to customary Python practices. There are places where these will conflict, and the goal is to make the library intuitive both to those familiar with the XML schemas (but less familiar with Python) and also to experienced Python developers who want to add CybOX support to their programs.

There are currently two levels of APIs for dealing with CybOX content:

  • A low-level API is provided by auto-generated XML Schema - Python class bindings. These bindings were generated using generate_ds. With these, any CybOX content can be parsed from or written to XML, but requires a bit more knowledge of the actual CybOX schemas. These "binding classes" are all located in the cybox.bindings package.
  • A higher-level API consisting of manually designed Python classes. These "native classes" are intended to behave more like Python programmers would expect. As they are designed manually, they currently do not support the entire CybOX standard, but rather those object types we expect are used most frequently. These "native classes" also support exporting their content as Python dictionaries and lists, which can easily be converted to JSON. Importing from JSON is also supported.

Versioning

Releases of the python-cybox library will be given version numbers of the form major.minor.update.revision, where major, minor, and update correspond to the CybOX version being supported. The revision number is used to indicate new versions of the python-cybox library itself.

Installation

The cybox package depends on the following Python libraries:

  • lxml
  • python-dateutil
  • setuptools (only if installing using setup.py)

For Windows installers of the above libraries, we recommend looking here: http://www.lfd.uci.edu/~gohlke/pythonlibs/.

To build lxml on Ubuntu, you will need the following packages from the Ubuntu package repository:

  • python-dev
  • libxml2-dev
  • libxslt1-dev
  • zlib1g-dev

For more information about installing lxml, see http://lxml.de/installation.html.

Layout

The structure of the python-cybox repository is as follows:

  • cybox/ : the root package
  • examples/ : example scripts that leverage the python-cybox library
  • cybox/utils/ : utility modules that are leveraged internally by the python-cybox library
  • cybox/test/ : unit tests
  • cybox/bindings/ : generateDS created xml-to-python bindings (leveraged in parsing and output of CybOX XML content)
  • cybox/core/ : APIs for core CybOX constructs (e.g., Observables)
  • cybox/common/ : APIs for common CybOX constructs (e.g., Measure Source)
  • cybox/object/ : APIs for CybOX objects (e.g., File Object, Address Object)

Please refer to the example scripts for examples of how to use the python-cybox library.

Feedback

Bug reports and feature requests are welcome and encouraged. Pull requests are especially appreciated. Feel free to use the issue tracker on GitHub or send an email directly to [email protected].

python-cybox's People

Contributors

2xyo avatar apsillers avatar chisholm avatar clenk avatar elegantmoose avatar emmanvg avatar gtback avatar ikiril01 avatar imjonsnooow avatar rroberge avatar tirkarthi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-cybox's Issues

Add Observable_Package_Source to Observables api

The Observables class needs an observable_package_source property. The Observable_Package_Source element is of type cyboxCommon:MeasureSourceType, which we have implemented in cybox.common.

Enforce ISODate on export

The DateTimeObjectAttributeType does not enforce that it's value is a properly-formatted ISO date, so we need to ensure that the string passed to it (as well as the JSON-formatted output) are in the valid format. This means that the DateTime Attribute class should probably store a Python datetime object as value and call iso_format() on export. It should also likely use python-dateutil so it can receive dates in a variety of formats.

add XML validation functionality

Create a validate() utility method that verifies the correctness of the output/input document. This will likely need to use the lxml.etree XMLSchema class.

name 'file_binding' is not defined

After the last pulldown I started getting this error:

python-cybox/cybox/objects/win_registry_key_object.py", line 28, in WinRegistryKey
_binding = file_binding
NameError: name 'file_binding' is not defined

AttributeError: 'ImageInfoType' object has no attribute 'Name'

File "python-cybox/cybox/core/observable.py", line 227, in from_obj
obs.add(Observable.from_obj(o))
File "python-cybox/cybox/core/observable.py", line 137, in from_obj
obs.object_ = Object.from_obj(observable_obj.get_Object())
File "python-cybox/cybox/core/object.py", line 130, in from_obj
obj.properties = ObjectProperties.from_obj(object_obj.get_Properties())
File "python-cybox/cybox/common/object_properties.py", line 91, in from_obj
defobj = klass.from_obj(defobj_obj)
File "python-cybox/cybox/common/object_properties.py", line 78, in from_obj
return super(ObjectProperties, cls()).from_obj(defobj_obj)
File "python-cybox/cybox/init.py", line 155, in from_obj
val = field.type_.from_obj(val)
File "python-cybox/cybox/init.py", line 150, in from_obj
val = getattr(cls_obj, field.name)
AttributeError: 'ImageInfoType' object has no attribute 'Name'

incorrect import in extracted_string

I just now pulled down the current version and this patch hasn't yet been applied:

diff --git a/cybox/common/extracted_string.py b/cybox/common/extracted_string.py
index 415688c..57c8d80 100644
--- a/cybox/common/extracted_string.py
+++ b/cybox/common/extracted_string.py
@@ -1,6 +1,7 @@
 import cybox
 import cybox.bindings.cybox_common as common_types_binding
-from cybox.common.properties import String, HexBinary, PositiveInteger, HashList
+from cybox.common.properties import String, HexBinary, PositiveInteger
+from cybox.common.hashes import HashList
 
 class ExtractedString(cybox.Entity):
     def __init__(self, string_value = None):

Thanks for all the other quick fixes :-)

Make a single function to parse CybOX files, and report errors sanely

Perhaps this should be cybox.parse. Eventually, we might add options for explicitly checking the version, etc. but for now there don't need to be any options.

The alternative is to put this function on Observables, so cybox.core.Observables.parse, since it should return an object of type Observables.

Note that you can currently parse a file by using the parseString function on the corresponding binding file, but this is not documented clearly and contradicts our attitude that users should not need to be aware of the bindings unless they really know what they are doing.

hash object constructor has unreliable type_

Code was:

    self.simple_hash_value = hash_value

    if type_ == self.AUTO_TYPE:
        if not hash_value:
            # If not provided or an empty string, don't assign the type
            self.type_ = None
        elif len(hash_value) == 32:
            self.type_ = Hash.TYPE_MD5
        elif len(hash_value) == 40:
            self.type_ = Hash.TYPE_SHA1
        elif len(hash_value) == 64:
            self.type_ = Hash.TYPE_SHA256
        else:
            self.type_ = Hash.TYPE_OTHER

hash_value can come in as either a string or as a SimpleHashValue object, so taking the length of it without qualifying is incorrect. The self.simple_hash_value setter forces a SimpleHashValue object so we can take the length of its value property safely

changed as follows:

    self.simple_hash_value = hash_value

    if type_ == self.AUTO_TYPE:
        if not self.simple_hash_value:
            # If not provided or an empty string, don't assign the type
            self.type_ = None
        elif len(self.simple_hash_value.value) == 32:
            self.type_ = Hash.TYPE_MD5
        elif len(self.simple_hash_value.value) == 40:
            self.type_ = Hash.TYPE_SHA1
        elif len(self.simple_hash_value.value) == 64:
            self.type_ = Hash.TYPE_SHA256
        else:
            self.type_ = Hash.TYPE_OTHER

file_object.to_dict tries to return incorrect variable (from typo)

The filepath_obj in the return is a typo, probably from a cut and paste error:
def to_dict(self):
filepath_dict = String.to_dict(self)
if self.fully_qualified is not None:
filepath_obj['fully_qualified'] = self.fully_qualified
return filepath_obj

It should be:
def to_dict(self):
filepath_dict = String.to_dict(self)
if self.fully_qualified is not None:
filepath_dict['fully_qualified'] = self.fully_qualified
return filepath_dict

binding Export fails due to incorrect object property assumption

Using the function in bindings objects: For the FileObjectType and EmailMessageObjectType, although I suspect it would happen with all objects.
In: cybox/utils.py
Line: 277
object.get_Defined_Object()
AttributeError: 'EmailMessageObjectType' object has no attribute 'get_Defined_Object'

hash object does not set type if value not set on construction

simple_hash_value setter was:

@simple_hash_value.setter
def simple_hash_value(self, value):
    if value and not isinstance(value, SimpleHashValue):
        value = SimpleHashValue(value)
    self._simple_hash_value = value

should use same logic as in constructor,as follows:

@simple_hash_value.setter
def simple_hash_value(self, hash_value):
    if hash_value and not isinstance(hash_value, SimpleHashValue):
        hash_value = SimpleHashValue(hash_value)
    self._simple_hash_value = hash_value
    if not hash_value:
        # If not provided or an empty string, don't assign the type
        self.type_ = None
    elif len(hash_value.value) == 32:
        self.type_ = Hash.TYPE_MD5
    elif len(hash_value.value) == 40:
        self.type_ = Hash.TYPE_SHA1
    elif len(hash_value.value) == 64:
        self.type_ = Hash.TYPE_SHA256
    else:
        self.type_ = Hash.TYPE_OTHER

'NetworkConnectionObjectType' object has no attribute 'destination_tcp_state'

o = Observables.from_obj(cybox_obj)

File "/python-cybox/cybox/core/observable.py", line 227, in from_obj
obs.add(Observable.from_obj(o))
File "/python-cybox/cybox/core/observable.py", line 137, in from_obj
obs.object_ = Object.from_obj(observable_obj.get_Object())
File "/python-cybox/cybox/core/object.py", line 130, in from_obj
obj.properties = ObjectProperties.from_obj(object_obj.get_Properties())
File "/python-cybox/cybox/common/object_properties.py", line 91, in from_obj
defobj = klass.from_obj(defobj_obj)
File "/python-cybox/cybox/common/object_properties.py", line 78, in from_obj
return super(ObjectProperties, cls()).from_obj(defobj_obj)
File "/python-cybox/cybox/init.py", line 150, in from_obj
val = getattr(cls_obj, field.name)
AttributeError: 'NetworkConnectionObjectType' object has no attribute 'destination_tcp_state'

import error for cybox/objects/memory_object.py

I had an issue with the following when trying to create a WinService object:

diff --git a/cybox/objects/memory_object.py b/cybox/objects/memory_object.py
index 655e44c..f625d5a 100644
--- a/cybox/objects/memory_object.py
+++ b/cybox/objects/memory_object.py
@@ -1,6 +1,6 @@
 import cybox
 import cybox.utils as utils
-import cybox.bindings.memory_object_1_2 as memory_binding
+import cybox.bindings.memory_object as memory_binding
 from cybox.common import HashList, ObjectProperties, String, UnsignedLong, HexBinary
 
 class Memory(ObjectProperties):
@@ -73,4 +73,4 @@ class Memory(ObjectProperties):
         memory_.region_size = UnsignedLong.from_obj(memory_obj.get_Region_Size())
         memory_.region_start_address = HexBinary.from_obj(memory_obj.get_Region_Start_Address())
 
-        return memory_
\ No newline at end of file
+        return memory_

Some incorrect imports

As I'm looking at the process_object code, I see that things have probably been moved around since this code was last used.

diff --git a/cybox/common/extracted_string.py b/cybox/common/extracted_string.py
index 415688c..57c8d80 100644
--- a/cybox/common/extracted_string.py
+++ b/cybox/common/extracted_string.py
@@ -1,6 +1,7 @@
 import cybox
 import cybox.bindings.cybox_common as common_types_binding
-from cybox.common.properties import String, HexBinary, PositiveInteger, HashList
+from cybox.common.properties import String, HexBinary, PositiveInteger
+from cybox.common.hashes import HashList
 
 class ExtractedString(cybox.Entity):
     def __init__(self, string_value = None):
diff --git a/cybox/objects/process_object.py b/cybox/objects/process_object.py
index 77feaf4..b287c50 100644
--- a/cybox/objects/process_object.py
+++ b/cybox/objects/process_object.py
@@ -5,7 +5,8 @@ from cybox.objects.address_object import Address
 from cybox.objects.network_connection_object import NetworkConnection
 from cybox.common.extracted_string_list import ExtractedStringList
 from cybox.common.environment_variable_list import EnvironmentVariableList
-from cybox.common.properties import ObjectProperties, String, DateTime, UnsignedInteger, Duration
+from cybox.common.properties import String, DateTime, UnsignedInteger, Duration
+from cybox.common import ObjectProperties

 class Process(ObjectProperties):
     _XSI_NS = "ProcessObj"
@@ -223,4 +224,4 @@ class ImageInfo(cybox.Entity):
         image_info_.current_directory = String.from_dict(image_info_obj.get_Current_Directory())
         image_info_.path = String.from_dict(image_info_obj.get_Path())

-        return image_info_
\ No newline at end of file
+        return image_info_

name 'SIDType' is not defined

Just started getting this error:

python-cybox/cybox/objects/win_process_object.py", line 54, in WinProcess
security_type = cybox.TypedField("Security_Type", SIDType)
NameError: name 'SIDType' is not defined

Create ID Generator class

This will go in cybox.utils and provide functionality similar to create_cybox_id() in api.py.

It will allow for different methods of generating IDs (UUID vs. incrementing integer, as we've discussed) and let the user customize the prefix.

WindowsEventLogObjectType does not have a specified API class

Just got the following backtrace:

return Observables.from_obj(cybox_obj)
File "egg/cybox/core/observable.py", line 227, in from_obj
File "egg/cybox/core/observable.py", line 137, in from_obj
File "egg/cybox/core/object.py", line 130, in from_obj
File "egg/cybox/common/object_properties.py", line 90, in from_obj
File "egg/cybox/utils/init.py", line 14, in get_class_for_object_type

File "egg/cybox/utils/nsparser.py", line 114, in get_class_for_object_type
cybox.utils.nsparser.UnknownObjectType: WindowsEventLogObjectType does not have a specified API class

couple more typos and references to None objects

Some typos and a None object issue in cybox/objects/win_process_object.py:

diff --git a/cybox/objects/win_process_object.py b/cybox/objects/win_process_object.py
index 48aa912..45dd9f1 100644
--- a/cybox/objects/win_process_object.py
+++ b/cybox/objects/win_process_object.py
@@ -85,7 +85,7 @@ class WinProcess(Process):
 
     @staticmethod
     def from_obj(win_process_obj, win_process_cls):
-        if not win_process_dict:
+        if not win_process_obj:
             return None
         if win_process_cls == None:
             winprocess_ = Process.from_obj(win_process_obj, WinProcess())
@@ -93,10 +93,10 @@ class WinProcess(Process):
             winprocess_ = Process.from_obj(win_process_obj, win_process_cls)
 
         winprocess_.aslr_enabled = win_process_obj.get_aslr_enabled()
-        winprocess_.dep_enabled = win_process_dict.get_dep_enabled()
+        winprocess_.dep_enabled = win_process_obj.get_dep_enabled()
         winprocess_.handle_list = WinHandleList.from_obj(win_process_obj.get_Handle_List())
         winprocess_.priority = String.from_obj(win_process_obj.get_Priority())
-        winprocess_.section_list = [Memory.from_obj(x) for x in win_process_obj.get_Section_List().get_Memory_Section()]
+        if win_process_obj.get_Section_List() is not None: winprocess_.section_list = [Memory.from_obj(x) for x in win_process_obj.get_Section_List().get_Memory_Section()]
         winprocess_.security_id = String.from_obj(win_process_obj.get_Security_ID())
         winprocess_.startup_info = StartupInfo.from_obj(win_process_obj.get_Startup_Info())
         winprocess_.security_type = String.from_obj(win_process_obj.get_Security_Type())
@@ -205,4 +205,4 @@ class StartupInfo(cybox.Entity):
         startup_info_.hstdoutput = WinHandle.from_dict(startup_info_dict.get('hstdoutput'))
         startup_info_.hstderror = WinHandle.from_dict(startup_info_dict.get('hstderror'))
 
-        return startup_info_dict
\ No newline at end of file
+        return startup_info_dict

and another None object referenced here in cybox/objects/win_service_object.py:

diff --git a/cybox/objects/win_service_object.py b/cybox/objects/win_service_object.py
index ca28040..3e93400 100644
--- a/cybox/objects/win_service_object.py
+++ b/cybox/objects/win_service_object.py
@@ -111,7 +111,7 @@ class WinService(WinProcess):
         win_service_ = WinProcess.from_obj(win_service_obj, WinService())
         win_service_.service_dll_signature_exists = win_service_obj.get_service_dll_signature_exists()
         win_service_.service_dll_signature_verified = win_service_obj.get_service_dll_signature_verified()
-        win_service_.description_list = [String.from_obj(x) for x in win_service_obj.get_Description_List().get_Description()]
+        if win_service_obj.get_Description_List() is not None: win_service_.description_list = [String.from_obj(x) for x in win_service_obj.get_Description_List().get_Description()]
         win_service_.display_name = String.from_obj(win_service_obj.get_Display_Name())
         win_service_.group_name = String.from_obj(win_service_obj.get_Group_Name())
         win_service_.service_name = String.from_obj(win_service_obj.get_Service_Name())

TypeError in URI object in xml to json conversion

If the DefinedObject does not have a type (not xsi:type) attribute, the NoneType result causes an exception. There should probably be a default type but either way the None should be handled more elegantly as the type attribute is not required.
I used he simple URL example at cybox.mitre.org

typo in cybox/objects/win_service_object.py

to_dict() should return the dictionary that it just created:

diff --git a/cybox/objects/win_service_object.py b/cybox/objects/win_service_object.py
index 90dd4e1..ca28040 100644
--- a/cybox/objects/win_service_object.py
+++ b/cybox/objects/win_service_object.py
@@ -76,7 +76,7 @@ class WinService(WinProcess):
         if self.started_as is not None: win_service_dict['started_as'] = self.started_as.to_dict()
         win_service_dict['xsi:type'] = self._XSI_TYPE
 
-        return win_service_obj
+        return win_service_dict
         
     @staticmethod
     def from_dict(win_service_dict):

WindowsVolumeObjectType does not have a specified API class

o = Observables.from_obj(cybox_obj)

File "/python-cybox/cybox/core/observable.py", line 227, in from_obj
obs.add(Observable.from_obj(o))
File "/python-cybox/cybox/core/observable.py", line 137, in from_obj
obs.object_ = Object.from_obj(observable_obj.get_Object())
File "/python-cybox/cybox/core/object.py", line 130, in from_obj
obj.properties = ObjectProperties.from_obj(object_obj.get_Properties())
File "/python-cybox/cybox/common/object_properties.py", line 90, in from_obj
klass = cybox.utils.get_class_for_object_type(type_value)
File "/python-cybox/cybox/utils/init.py", line 14, in get_class_for_object_type
return META.get_class_for_object_type(object_type)
File "/python-cybox/cybox/utils/nsparser.py", line 114, in get_class_for_object_type
raise UnknownObjectType(err)
cybox.utils.nsparser.UnknownObjectType: WindowsVolumeObjectType does not have a specified API class

Don't double-decode XML entities

When a literal &amp;, &gt; or &lt; is in a String property in CybOX, it is encoded when being output to XML, but then decoded twice during import, resulting in &, >, or <.

This is partially mitigated when wrapped in a block (for instance, if the String also contains a comma). However, this is inconsistent and leads to strange results.

It may be possible to use resolve_entities=False (http://lxml.de/api/lxml.etree.XMLParser-class.html), or remove the second level of decoding in cybox.utils.

There needs to be a comprehensive set of test cases, though, to ensure all combinations are well-covered.

'WindowsServiceObjectType' object has no attribute 'Serivice_DLL_Certificate_Subject'

Just got this:

File "python-cybox/cybox/core/observable.py", line 227, in from_obj
obs.add(Observable.from_obj(o))
File "python-cybox/cybox/core/observable.py", line 137, in from_obj
obs.object_ = Object.from_obj(observable_obj.get_Object())
File "python-cybox/cybox/core/object.py", line 130, in from_obj
obj.properties = ObjectProperties.from_obj(object_obj.get_Properties())
File "python-cybox/cybox/common/object_properties.py", line 91, in from_obj
defobj = klass.from_obj(defobj_obj)
File "python-cybox/cybox/common/object_properties.py", line 78, in from_obj
return super(ObjectProperties, cls()).from_obj(defobj_obj)
File "python-cybox/cybox/init.py", line 150, in from_obj
val = getattr(cls_obj, field.name)
AttributeError: 'WindowsServiceObjectType' object has no attribute 'Serivice_DLL_Certificate_Subject'

The api.py module is possibly mislabeled

The api.py module began as "common_indicators.py" and was meant to ease the creation of a few, commonly shared indicator types: file hashes, email addresses, urls, domain names, and ipv4 addresses.

The api.py module should define methods to ease the creation and binding/linking/relating of observable instance data and/or patterns outside of and including the currently defined set of observable types.

Should we rename this module or simply build it out to support more general use cases?

typo in cybox/objects/process_object.py

In the to_dict() function, it's accidentally printing out "is_hidden" information instead:

diff --git a/cybox/objects/process_object.py b/cybox/objects/process_object.py
index 77feaf4..23694a9 100644
--- a/cybox/objects/process_object.py
+++ b/cybox/objects/process_object.py
@@ -5,7 +5,8 @@ from cybox.objects.address_object import Address
 from cybox.objects.network_connection_object import NetworkConnection
 from cybox.common.extracted_string_list import ExtractedStringList
 from cybox.common.environment_variable_list import EnvironmentVariableList
-from cybox.common.properties import ObjectProperties, String, DateTime, UnsignedInteger, Duration
+from cybox.common.properties import String, DateTime, UnsignedInteger, Duration
+from cybox.common import ObjectProperties
 
 class Process(ObjectProperties):
     _XSI_NS = "ProcessObj"
@@ -78,7 +79,7 @@ class Process(ObjectProperties):
         process_dict = {}
 
         if self.is_hidden is not None: process_dict['is_hidden'] = self.is_hidden
-        if self.pid is not None: process_dict['is_hidden'] = self.pid.to_dict()
+        if self.pid is not None: process_dict['pid'] = self.pid.to_dict()
         if self.name is not None: process_dict['name'] = self.name.to_dict()
         if self.creation_time is not None: process_dict['creation_time'] = self.creation_time.to_dict()
         if self.parent_pid is not None: process_dict['parent_pid'] = self.parent_pid.to_dict()
@@ -223,4 +224,4 @@ class ImageInfo(cybox.Entity):
         image_info_.current_directory = String.from_dict(image_info_obj.get_Current_Directory())
         image_info_.path = String.from_dict(image_info_obj.get_Path())
 
-        return image_info_
\ No newline at end of file
+        return image_info_

Creating objects from real files.

Do you plan to add support for creating objects from real files in order to supply self properties ?

Example:

evidence = File()
evidence.from_file("/path/email.eml")
email = EmailMessage()
email.from_file(evidence)
# OR
email = EmailMessage(file="/path/email.eml")
# OR
email = EmailMessage()
email.from_file("/path/email.eml")

print email.header.from_.to_dict()
>>> {'category': 'e-mail', 'address_value': '[email protected]'}

Unicode decoding error

Hello,

The script email_to_cybox.py raises an UnicodeDecodeError with some UTF-8 emails.

Steps to reproduce the problem:

Result:

$ python email_to_cybox.py  -v email.txt 
** parsing email input file
** parsing headers
** parsing attachments
** parsing urls from email body
** creating domain name object for: cam.ac.uk
** creating uri object for: http://www.cl.cam.ac.uk/~mgk25/
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE doc [<!ENTITY comma '&#44;'>]>

!! error: 'ascii' codec can't decode byte 0xce in position 3637: ordinal not in range(128)
Traceback (most recent call last):
  File "email_to_cybox.py", line 964, in main
    print(observables.to_xml(namespace_dict={'https://github.com/CybOXProject/Tools':'email_to_cybox'}))
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/__init__.py", line 61, in to_xml
    self.to_obj().export(s, 0, namespacedef_=namespace_def)
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/core/observable.py", line 193, in to_obj
    observables_obj.set_Observable([x.to_obj() for x in self.observables])
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/core/observable.py", line 100, in to_obj
    obs_obj.set_Object(self.object_.to_obj())
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/core/object.py", line 72, in to_obj
    obj.set_Properties(self.properties.to_obj())
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/objects/email_message_object.py", line 568, in to_obj
    email_obj.set_Raw_Body(self.raw_body.to_obj())
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/common/properties.py", line 163, in to_obj
    attr_obj.set_valueOf_(normalize_to_xml(self.serialized_value))
  File "/home/yoyo/.virtualenvs/playground-dev/src/python-cybox/cybox/utils/__init__.py", line 34, in normalize_to_xml
    return xml.sax.saxutils.escape(unicode(value), ESCAPE_DICT)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 3637: ordinal not in range(128)
** processing completed

AttributeError: 'HTTPRequestHeaderFieldsType' object has no attribute 'X_WAP_Profile'

Just got the following error with the latest code:

File "/cybox/core/observable.py", line 227, in from_obj
File "/cybox/core/observable.py", line 137, in from_obj
File "/cybox/core/object.py", line 130, in from_obj
File "/cybox/common/object_properties.py", line 91, in from_obj
File "/cybox/common/object_properties.py", line 78, in from_obj
File "/cybox/init.py", line 155, in from_obj
File "/cybox/init.py", line 155, in from_obj
File "/cybox/common/object_properties.py", line 78, in from_obj
File "/cybox/init.py", line 153, in from_obj
File "/cybox/init.py", line 155, in from_obj
File "/cybox/init.py", line 155, in from_obj
File "/cybox/init.py", line 155, in from_obj
File "/cybox/init.py", line 150, in from_obj
AttributeError: 'HTTPRequestHeaderFieldsType' object has no attribute 'X_WAP_Profile'

Add Ability to Disable Pretty Print in to_xml()

The Python bindings support disabling pretty printing of XML output using the pretty_print=False parameter. We should add support for this in the to_xml() method so that users can generate smaller XML files if they don't care about readability.

Typo in cybox/common/properties.py

Noticed a typo:

diff --git a/cybox/common/properties.py b/cybox/common/properties.py
index 226c24e..ce15784 100644
--- a/cybox/common/properties.py
+++ b/cybox/common/properties.py
@@ -332,7 +332,7 @@ class PositiveInteger(BaseProperty):
 
 class UnsignedInteger(BaseProperty):
     def __init__(self, *args, **kwargs):
-        BaseProeprty.__init__(self, *args, **kwargs)
+        BaseProperty.__init__(self, *args, **kwargs)
         self.datatype = "unsignedInt"
 
     def _get_binding_class(self):
@@ -340,7 +340,7 @@ class UnsignedInteger(BaseProperty):
 
 class NonNegativeInteger(BaseProperty):
     def __init__(self, *args, **kwargs):
-        BaseProeprty.__init__(self, *args, **kwargs)
+        BaseProperty.__init__(self, *args, **kwargs)
         self.datatype = "nonNegativeInteger"
 
     def _get_binding_class(self):

issue with "&comma;" when parsing

After creating a process observable that had a comma in the commandline I get an error when trying to parse it. Here's the commandline:

"C:\WINDOWS\system32\rundll32.exe" bthprops.cpl,,BluetoothAuthenticationAgent

The error:

  cybox_obj = core_binding.parse(os.path.abspath(filename))
  File "build/egg/cybox/bindings/cybox_core.py", line 5567, in parse
  File "build/egg/cybox/bindings/cybox_core.py", line 376, in parsexml_
  File "lxml.etree.pyx", line 3197, in lxml.etree.parse (src/lxml/lxml.etree.c:64726)
  File "parser.pxi", line 1571, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:92363)
  File "parser.pxi", line 1600, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:92647)
  File "parser.pxi", line 1500, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:91710)
  File "parser.pxi", line 1047, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:88610)
  File "parser.pxi", line 577, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:84019)
  File "parser.pxi", line 676, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:85122)
  File "parser.pxi", line 616, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:84445)
lxml.etree.XMLSyntaxError: Entity 'comma' not defined, line 31, column 100

CDATA stripped from fields

It seems that some fields (e.g., EmailMessageObject's Raw_Header) that encapsulate data within a CDATA block are having that block stripped either during import or export.

File Object properties are acting like static values

In the File Object [file_object.py]

properties are such:

file_name = cybox.TypedField(String)
file_path = cybox.TypedField(FilePath)
...

The result is the properties become effectively static. That is, the value in any object instance is the value assigned to the last object instance.

I changed it locally as follows

    self._file_name = None

    @property
    def file_name(self):
        return self._file_name

    @file_name.setter
    def file_name(self, value):
        if not isinstance(value, String):
            self._file_name = String(value)
        else:
            self._file_name = value

Artifact.from_obj() makes call to nonexistent function

122    for e in packaging.get_Encryption():
123         artifact.packaging.append(Encryption.from_obj(e))

Line 123 makes a call to Encryption.from_obj(), which does not exist. Currently Encryption is an empty class. Parsing an ArtifactObj with an Encryption child element results in an error.

Remove OBJ_LIST from nsparser.py

Most of the content in OBJ_LIST is duplicated elsewhere:

  • object name/xsi type
  • api class
  • binding
  • namespace
  • dependencies (other objects)

There should be some magic code that imports all of the objects/ directory.

Better Document JSON Output

We should better document our JSON output capability in the simple APIs and mark any issues that exist with it, particularly with regards to where it does not perfectly align with the XML representation of the same data.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.