edinburghgenomics / pyclarity-lims Goto Github PK

update docstring to specify the types of the arguments e.g. inart as string for the artificat id (easily confused with the artifact variable type).

I was actually aware of the below issue but it should probably go in the documenatation for pyclarity_lims.entities.Process - if you want to modify (rather than just view) the UDFs you have to load the step.details as Dan says.

Hi Tim,

Not sure if you were made aware of this or whether you will ever need
to know. Matt had an issue with one of his scripts and turned out
that it was because he was updating a process UDF using the “process
API end point” and not the “step API end point”. It was also related
to the user role assigned to the person logged in (I know it sounds
strange). So if writing scripts to update step UDFs that may ever be
initiated by a user use the step API end point which in pyclarity-
LIMs speak is something like [PROCESS].step.details.put().

Dan

Daniel Forth, PhD
LIMS Manager

I can make sure the tests pass, but I don't have access to an instance of Clarity anymore, so don't have any way of validating new versions IRL. Also not sure if it's still in use at Edinburgh Genomics.

@tbooth - it may be good to review this fork versus the upstream and have a think about what should be done with it

I was needing to update the code for another issue and tried on Python 3.9. It doesn't work because of line 648 of lims.py:

  for node in root.getchildren():

The documentation for ETree says this should be:

  for node in list(root):

As an addendum, I see from issue 58 from last year that this repository may well be dead. Is it time to move to SciLifeLabs original repository?

We should be able to read utf-16-encoded files, e.g. from the Spectramax. requests can return text with any encoding:

>>> request.encoding
'utf-8'
>>> request.text
'some_glitchy_looking_binary_data'
>>> request.encoding = 'utf-16'
>>> request.text
'some_well_formed_data'

We should add this to Lims.get_file_contents. We should also be able to replace CRLFs with \n.

Which one do we want?

see https://github.com/SciLifeLab/genologics/pull/204/files

  File "./clarity_run_id_setter.py", line 68, in main
    Step(lims, id=ep.id).advance()
  File "/lustre/home/pipeline/illuminatus/current/_py3_venv/lib/python3.5/site-packages/pyclarity_lims/entities.py", line 843, in advance
    data=self.lims.tostring(ElementTree.ElementTree(self.root))
  File "/lustre/home/pipeline/illuminatus/current/_py3_venv/lib/python3.5/site-packages/pyclarity_lims/lims.py", line 698, in tostring
    self.write(outfile, etree)
  File "/lustre/home/pipeline/illuminatus/current/_py3_venv/lib/python3.5/site-packages/pyclarity_lims/lims.py", line 703, in write
    etree.write(outfile, encoding='utf-8', xml_declaration=True)
  File "/lustre/software/python/Python-3.5.2/lib/python3.5/xml/etree/ElementTree.py", line 772, in write
    qnames, namespaces = _namespaces(self._root, default_namespace)
  File "/lustre/software/python/Python-3.5.2/lib/python3.5/xml/etree/ElementTree.py", line 875, in _namespaces
    for elem in elem.iter():
AttributeError: 'NoneType' object has no attribute 'iter'

Hi,

I got the stack trace above. I'm not sure if this is something broken in the module or just a failure to catch an error condition and convert it to a meaningful error. I think the step was already completed, and I'm not 100% sure I have permission to close it. Will dig further.

Cheers,

TIM

I am having an issue when using the API to complete one step and move to the next. I followed the example in https://pyclarity-lims.readthedocs.io/en/latest/PracticalExamples.html#start-and-complete-a-new-step-from-submitted-samples until the end where I have an error.

When trying to execute the line
actions = s.actions.next_actions[0]['action'] = 'complete'
I get the error of
The valid actions are: [nextstep, repeat, remove, review, rework, completerepeat]

When I try to use the action nextstep it requires a uri but there isn't documentation of how to assign this. I have tried the line below but that does not work.
actions = s.actions.next_actions[0]['action']['nextsteps'] = 'step_uri_example

Is it possible to create more documentation on how to use this or advise on how to proceed?

add option to open files as binary in lims.get_file_content

When I try to create a queue on version 0.4.2 of this package I am receiving an error. The queue object is created successfully, however when artifacts and queued_artifacts are accessed a stack trace is returned. The stack trace reports "ValueError: unconverted data remains: -0500". I have a feeling this is to due to the queue time for my sample being something like "2018-01-09T15:44:48.571-05:00".

Thank you

Hi,

I am doing the following but it is very slow:

# Get the the values generated from the 'Calculate DNA Mass' macro
for sample in samples:
    for target_file in process.result_files():
        target_file.get(force=True)  # Does not find it unless force a GET
        file_sample = target_file.samples[0].name
        if file_sample == sample.name:
              print target_file.udf['01_O_TS_Region1Mass']

Is there a different logic I should follow to get the UDFs?

Thanks

Daniel

Calling lims.get_batch(files) produces the following error:

  File "/usr/lib/python2.6/site-packages/pyclarity_lims/lims.py", line 660, in put_batch
    uri = self.get_uri(klass._URI, 'batch/update')
  File "/usr/lib/python2.6/site-packages/pyclarity_lims/lims.py", line 78, in get_uri
    url = urljoin(self.baseuri, '/'.join(segments))
TypeError: sequence item 2: expected string, NoneType found

It looks like this would be as simple as adding _URI and _PREFIX to the File class.

class File(Entity):
    """File attached to a project or a sample."""
    _URI = 'files'
    _PREFIX = 'file'

The benefit of supporting this operation is to batch update the is_published property of files.

I'm looking for a way to automate the tracking of reagents using pyclarity-lims. Here I say that a step requires cfDNA_Proteinase_K to be tracked:

However, I cannot see a way to attach a ReagentKit/Lot to the step and therefore I get this error:

Looking at the documentation for step.create I thought I could create a new step with reagent_category=ReagentType.category but I noticed this in the code:

        # TODO: more work needed to understand how the permitted reagent applies here

Is the tracking of reagents possible within pyclarity-lims or should I use udfs, or maybe, step.reagent_label?

This is the step that I am trying to automate:

At the moment Entity creation can only be done via the Entity contructors such as

Sample.create(...)

It would be useful to have leverage the batch/create function endpoints to create multiple samples/containers.
Function should be called: lims.create_batch
Implementation might be complicated as the entities cannot be created before they are passed on to batch.

The documentation for "Process.all_inputs" infers that the output is always a list type variable but the output variable type can be list or dict_values

Process.all_inputs(self, unique=True, resolve=False) produces a list type variable
Process.all_inputs(self, uniqued=True, resolve=True) produces a dict_values

def all_inputs(self, unique=True, resolve=False):
"""Retrieving all input artifacts from input_output_maps.
If unique is true, no duplicates are returned.

    :param unique: boolean specifying if the list of artifacts should be uniqued
    :param resolve: boolean specifying if the artifacts entities should be resolved through a batch query.
    :return: list of input artifacts._

Also modify the Readme to reflect the change of name

Hi,

I am trying to get 'Diluent Volume' out of Clarity LIMs using pyclarity-lims. I can see the UDF if I query the REST API:

<udf:field type="Numeric" name="01_O_QB_Mass">57.84</udf:field>
<udf:field type="Numeric" name="Desired Concentration">2</udf:field>
<udf:field type="Numeric" name="Diluent Volume">2.615</udf:field>
<udf:field type="Numeric" name="Volume to be diluted">1</udf:field>
<udf:field type="Numeric" name="03_I_StockVolume">8</udf:field>
<udf:field type="Numeric" name="02_I_DilutionFactor">7.23</udf:field>
<udf:field type="Numeric" name="01_I_QB_Concentration">7.23</udf:field>

However, I can only return the following UDFs:

{'Desired Concentration': 2.0, '03_I_StockVolume': u'8', 'Volume to be diluted': 1.0, '01_I_QB_Concentration': u'7.23'}

The code I am using looks like this:

for sample in samples:
    for outart in process.result_files():
        file_sample = outart.samples[0].name
        if file_sample == sample.name:
            print outart.udf

Anyone see where I am going wrong? The 'Diluent Volume' is set by running a script with trigger_program().

Thanks

Dan

For class pyclarity_lims.entities.Process for ""result_files()[source]"

The documentation states:
"Retrieve all resultfiles of output-generation-type perInput."

However, it doesn't seem to do this and instead seems to retrieve all artifacts of output_type "ResultFile:

def result_files(self):
"""Retrieve all resultfiles of output-generation-type perInput."""
artifacts = self.all_outputs(unique=True)
return [a for a in artifacts if a.output_type == 'ResultFile']

Hi all, (Edited from my original post, earlier today, because I was using the wrong data structure for StepPools.pooled_inputs)
Not an issue for you, rather for me! I'm trying to pool 2 plates into 2 tubes. First I create the two pools, set up the step and add the Illumina indexes to the tubes, I think I'm all set here, so you can probably ignore this:

    pool1 = Container.create(lims, type=tube_ct)
    pool2 = Container.create(lims, type=tube_ct)
    pools = [pool1, pool2]

    step = Step.create(lims, protocol_step=protocol_step, inputs=artifacts,
                       container_type_name=tube_ct.name)

    process = Process(lims, id=step.actions.step.id)

    # Assign the correct Illumina UDI indexes
<snip>
...
        sample.artifact.reagent_labels = [label]
        sample.artifact.put()

The issue comes when I try and pool the plates. I create a data structure and try and assign this to step.pools.pooled_inputs, thus:

    output_pools = {}
    for idx, plate in enumerate(plates):
        input_arts = []
        plate.get(force=True)
        out_container = pools[idx]
        pool_name = out_container.name
        placement_dict = plate.get_placements()
        for placekey in placement_dict.keys():
            art = placement_dict[placekey]
            print 'Artifact:', art
            input_arts.append(art)
        art_tuple = tuple(input_arts)  # tuble of input arts to add to a tube
        output_pools[pool_name] = tuple((out_container, art_tuple))
        out_container.get(force=True)
        for art in input_arts:
            art.put()
        plate.put()

    step.pools.pooled_inputs = output_pools

    step.advance()

The error I get is:

    step.advance()
  File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/entities.py", line 843, in advance
    data=self.lims.tostring(ElementTree.ElementTree(self.root))
  File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/lims.py", line 172, in post
    return self.parse_response(r, accept_status_codes=[200, 201, 202])
  File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/lims.py", line 214, in parse_response
    self.validate_response(response, accept_status_codes)
  File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/lims.py", line 207, in validate_response
    raise requests.exceptions.HTTPError(message)
requests.exceptions.HTTPError: 400: Unpooled inputs exist

I think I've got the data structure correct for StepPools.pooled_inputs. I'm thinking it must be the caching of the artifacts. Can anyone see where I've gone wrong?

Thanks!

In the Queue Class I feel as if the "queue-time" member variable should exist under artifacts just like location does. However, I have to traverse artifacts._elems[0]._children[0].text to get the queue time. Can we make this so something like artifacts[0].queue_time works? Thanks.

When trying to run integration tests in, e.g, EGCG-Project-Management and trying to import things from within its integration_tests module, Python instead imports the integration_tests module from pyclarity-lims because it's installed to site-packages, and then runs into ImportErrors. We should include 'integration_tests' in setup/packages/exclude

Hi,

I am getting this warning when using Step.available_programs():

/usr/local/lib/python2.7/dist-packages/pyclarity_lims/entities.py:862: FutureWarning: The behavior of this method will change in future versions.  Use specific 'len(ele
m)' or 'elem is not None' test instead.
  if available_programs_et:

Should I change to a different method to get available programs or is this just related to how the Entities object is going to change in the future?

Thanks!

Dan

An empty array is always returned when retrieving a list of entities using the get_* methods of theLims class. For example:

samples = l.get_samples(start_index=500)
# samples == []

The problem is in the _get_instances() method. The response is only parsed when start_index is None.

Calling the UdfConfig.preset.clear() method seems to clear values from the entire UdfConfig object. Even if I set the preset list manually to a new empty list the object is also cleared...Is this expected?

It currently accept a protocolStep which is incorrect

That's definitely too restrictive and causes inconsistencies.
When the UDF does not exist in the dictionary the type is not set which mean no check is performed, setting a string works then.
If the UDF exist then the check is performed and the input value needs to be a date.

Hi,
I'd like to 'Mark protocol as complete' for each sample ONLY IF 'Samples that passed Aggregate_QC...' is true:

I can manually mark as complete using something like:

# TODO: this currently doesn't check the status of the QC flags.
step_actions = step.actions
for artifact in step_actions.next_actions:
    actions = artifact['action'] = 'complete'
step.actions.put()

However, I'd like the code to emulate the check that is performed when hitting 'Apply' in the screenshot. Is it just a matter of checking the QC flag for each sample or is there functionality in pyclarity-lims to do this in one easy step as one would do in the web interface?

Thanks

Dan