isa-tools / isa-api Goto Github PK
View Code? Open in Web Editor NEWISA tools API
Home Page: https://isa-tools.org
License: Other
ISA tools API
Home Page: https://isa-tools.org
License: Other
Currently the study filename properties are missing from the tab2json conversions.
Are already aware of this as was due to previous changes, and will need to reintroduce to allow more accurate conversions back to ISAtab from JSON.
Will ISA with JSON serialisation be using the same approach of separate i_* , s_* and a_* files as ISA-TAB, or one JSON file for everything is envisaged?
Sorry if this is already addressed in the documents, could not find a reference.
In tests/test_isatab2cedar.py to test the CEDAR converter, there is a test to use metabolights datasets however these are not present in the repo.
Consider removing the test (as it will always fail without the data) or implement a way to automatically get and use some test data from metabolights or other repo.
When generating JSON using istab2json for BII-I-1, there's several instances where objects appear but have not values in the properties. For example, under protocols, a few components and parameters appear with one component or parameter object but all set to empty strings.
Also for every process listed under processSequence in the assays list.
Converting BII-I-1 is unsuccessful, throwing up the following error when running test_isatab2json.py and elsewhere:
Error
Traceback (most recent call last):
File "/PycharmProjects/isa-api/tests/test_isatab2json.py", line 19, in test_bii_i_1_conversion
isa_json = self.isatab2json.convert(test_data_dir, self.sample_data_dir)
File "/PycharmProjects/isa-api/isatools/convert/isatab2json.py", line 49, in convert
("studies", self.createStudies(isa_tab.studies))
File "//PycharmProjects/isa-api/isatools/convert/isatab2json.py", line 215, in createStudies
("processSequence", self.createProcessSequence(study.process_nodes, source_dict, sample_dict, data_dict)),
File "/PycharmProjects/isa-api/isatools/convert/isatab2json.py", line 270, in createProcessSequence
("executesProtocol", self.createExecuteStudyProtocol(process_node_name, process_nodes[process_node_name])),
File "/PycharmProjects/isa-api/isatools/convert/isatab2json.py", line 305, in createExecuteStudyProtocol
("name", process_node.protocol)
AttributeError: 'ProcessNodeRecord' object has no attribute 'protocol'
Is it possible to extend the examples to include data objects with file names. It's not clear at the moment where one should specify the data file names in the data object.
Also, I've noticed the BII-I-1 JSON examples include pointers to tab delimited files (i_*.txt
, s_*.txt
, a_*.txt
), while all the metadata from these files should already be in the JSON itself.
In BII-S-3, there are sample characteristics with numeric values and 'Term Source REF' and 'Term Accession Number' columns. Is this correct?
e.g.
"Characteristics[geographic location (latitude)]" "Term Source REF" "Term Accession Number"
@proccaserra please confirm this.
This affects current version of the tab2json conversion (#15).
at the moment sources and samples are getting the same sets of characteristics.
Could you clarify, the parameters_schema.json
consists only of ontology annotation (parameterType) and unit, not sure where numeric values should be stored.
fix json-schemas accordingly.
For each Characteristic or Factor Value, there is an additional typing that is not currently present in the schemas.
Example in BII-I-1 tabular:
Characteristics[organism] column is present, but 'organism' part is lost in conversion.
Factor Value[limiting nutrient] column is present, but 'limiting nutrient' part is lost in conversion.
I've already done some work in the isatab package loader to parse these into model objects, so it will probably be straightforward to modify the JSON schemas and tab2json converter to fix this.
As agreed, we are trying to develop a JSON schema, corresponding to the ISA-TAB-Nano material file.
Wondering what would be the appropriate file name, as you already have schema named material_schema.json for the material nodes of the experimental graph.
ideally-canonical.json is not valid-formed JSON, so does not load in the Python JSON parser. Need to take care to run it through a validator before committing.
In order to generate automatically Java classes from the schemas for isa v.1.0 we used jsonschema2pojo CLI application. The generation failed (several exceptions were thrown). The three problems appear to be due to following small typos/errors:
(1) in material_schema.json
"items" : {
"$ref": "material_attribute_schema.json#characteristic"
}
changed to:
"items" : {
"$ref": "material_attribute_schema.json"
}
(2) in process_schema.json
"executesProtocol": {
"$ref": "protocol.json#"
},
changed to:
"executesProtocol": {
"$ref": "protocol_schema.json#"
},
(3) in sample_schema.json
"items" : {
"$ref" : "factor_value.json#"
}
changed to:
"items" : {
"$ref" : "factor_value_schema.json#"
}
The problems (2) and (3) are caused by missing "_schema" in the "$ref" definitions
I am not sure for the first case (1).
and link back to protocol
"#ref" is used instead of "$ref"
Therefore:
"#ref": "ontology_annotation_schema.json#"
should be
"$ref": "ontology_annotation_schema.json#"
use case: retrieve ISA-Tab (as a folder and zip files), ISA-json and configuration files
currently, only to samples
Representing experimental data extracted from published literature is a very common use case, but not straightforward to fit into ISA model.
The samples, the protocols, the assays will be different across papers. As the goal of data gathering exercise is to be able to combine the data from different papers, one investigation per paper is not necessary the right approach. If combining into single investigation, then I am not aware of means to assign the publications to particular assays or samples.
What would be the recommended approach?
ISA-JSON v1 schemas should support comments, as per the ISA-Tab format
relying on json ref
This might already be completed related to the JSON schemas validation. We need to validate against converted ISA-Tab files.
rather than copying converters code here
so that it validates against schemas (see #5)
We need ISA-Tab example datasets
In addition to the schema I am looking for ISA-JSON examples (both v1 and v2).
We've tried locally https://github.com/ISA-tools/isa-rest-api and it converts (some of our) ISA-TAB archives to JSON v1 (some fail). Wondering how complete is the convertor at this moment. For example I can't find the input/output chain in JSON neither in our tests , nor in this BII-I-1 JSON .
The reason to look for examples is it is not clear for me from the schema how the chains will be represented. The input/output nodes of the process chain refer to the same schema (material, data) as for describing sources and samples. Are these going to be repeated in the input/output chain, or referred by some identifier?
I could guess the JSON v1 schema follows very closely the Java object model as used before, but in the Java model the references are just pointing to the same objects, while not sure how this will look in JSON.
Make the API distributable for other developers to use. See https://docs.python.org/2.7//distutils/index.html
JsonToIsatabWriter does not work due to some incompatibilities with Python 3 that removed some string processing functions used, in particular in the common functions package.
There may be more cases of this throughout the codebase and more stringent testing is needed to check. Note that the changes from Python 2 to 3 (and even to later 3.x versions) are not just syntactic, but also have removed some packaged functions entirely.
test_isa_v1_parser.py on feat/reader branch throws up an exception:
Error
Traceback (most recent call last):
File "/Users/dj/PycharmProjects/isa-api/tests/test_isa_v1_parser.py", line 166, in test_jsonToIsatab_writer
mywriter.parsingJson(self._json_dir, output_dir)
File "/Users/dj/PycharmProjects/isa-api/isatools/io/json_to_isatab.py", line 78, in parsingJson
self.writeJsonInvestigationToIsatab(i_filenames, output_dir)
File "/Users/dj/PycharmProjects/isa-api/isatools/io/json_to_isatab.py", line 96, in writeJsonInvestigationToIsatab
investigation_str = self.processInvestigationJson(jsonData)
File "/Users/dj/PycharmProjects/isa-api/isatools/io/json_to_isatab.py", line 129, in processInvestigationJson
my_str = self.writeSectionInvestigation(my_str, "INVESTIGATION CONTACTS", jsonData["investigation"]["investigationContacts"], self._isatab_i_investigation_contacts_sec)
File "/Users/dj/PycharmProjects/isa-api/isatools/io/json_to_isatab.py", line 159, in writeSectionInvestigation
my_str = my_str + "\"" + b[self.commonFunctions.makeAttributeName(i)] + "\"" + "\t"
KeyError: 'investigationPersonAffiliation'
This seems to be caused by the json_to_isatab.py JsonToIsatabWriter() class converter being unable to handle looking up investigationPersonAffiliation in ISA-JSON.
I've noticed that Characteristics[] that appear in the Sources section of study ISAtab files appear in the characteristics of samples after the isatab2json conversion.
I am unsure if this is normal behaviour, as the samples list in the study tab files do not repeat these. Do samples inherit source characteristics?
I am wondering where one could specify numeric factor values.
The schema seems to have support for units, and factors specified by ontology entries, but not for numbers. Could you please clarify?
Start creating individual objects
instead of relying on Protocol Applications:
suggestion from COPO project:
COPO folks were asking about a way to link sources and samples (both ways I suppose, but maybe mainly from samples back to sources).
The current model allows this only through a process (which can be implicit in the ISA-tab files).
Do we want to allow more direct links? I think this would make the model less consistent but we can consider that...
This should consider the last model (JSON schemas). We can leave the previous attempts but only as experimentation.
What would be the correct way to represent an experiment not involving biological material - i.e. what should be the Source
node for physico-chemical measurements of a sample.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.