Giter VIP home page Giter VIP logo

bjerknesclimatedatacentre / quince Goto Github PK

View Code? Open in Web Editor NEW
7.0 4.0 8.0 46.02 MB

QuinCe is an online tool for processing and quality control of data from scientific instruments, with a primary focus on oceanic data.

Home Page: https://quince.science

License: GNU General Public License v3.0

HTML 4.27% Java 52.08% CSS 1.02% JavaScript 10.57% Shell 0.64% Python 6.16% Jupyter Notebook 25.25%
quality-control data-reduction oceanography co2-measurements near-real-time

quince's Introduction

Status Uptime

Version Activity

build junit codecov

Issues Bugs Pull Requests

License

QuinCe

QuinCe is a system for the collection, processing, quality control and publication of a variety of data streams, mostly focused on marine biogeochemical data but designed to be able to handle almost anything.

See Documentation/setup.md for installation details.

Contact SquareGoldfish for more information.

quince's People

Contributors

cla023 avatar dependabot[bot] avatar glncstll avatar jnegrel avatar jonasfh avatar julienpaul avatar marenkk avatar squaregoldfish avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

quince's Issues

Prevent starting New Instrument in the middle

It's possible to directly type an URL to get to the middle of the New Instrument page sequence without having started it first. Figure out how to stop it.

The general idea would be to check the underlying session bean and see what stage we're at, and go back to the latest stage.

Standards runs - what temperature etc?

The sea water CO2 runs use the sensor data in the data reduction calculations. What happens with the standards runs? What temperatures, pressures etc do they use if any?

Which of the calculated CO2 values do we get from a standards run? We need the corresponding version in the sea water for adjustments.

listToDelimited and delimitedToList do not handle semi-colons in values

listToDelimited and delimitedToList convert between lists of strings and a string of values with semi-colon delimiters. Neither of them yet handle the case where a given value contains a semi-colon - such values will get split.

We simply need to escape these delimiters where they appear within values.

Raw sensor data conversions

Some sensors have their measurements reported in electronic terms (mV, mA etc.) I assume there is an equation that converts these values to the true values. This leads to a number of questions:

  1. What is the definition of this equation, and is it a linear/polynomial equation?
  2. Is this equation fixed for specific sensor models? Is it worth (in future) adding a list of sensors and their equations to the system somehow?
  3. Is calibration applied at the electronic level or the parameter level (i.e. are we calibrating the temperature or the electrical signal?)

Job thread pool initialisation

The JobThreadPool is currently initialised in Pools.java, using a fixed number of threads.
This should be configurable.

Keep track of running threads

The Job Manager doesn't track running threads, so if all the threads mysteriously disappear, the job thread pool will never have anything in. Set up something to recreate threads if we lose track of them.

Disable Settings menu

The Settings menu must be disabled at certain points, e.g. when adding a new instrument.

Pointer on user menu

The pointer on the user menu is a text selector. Should be a link clicky one.

Also check that the text can't be selected.

Multiple sensors - handle one faulty.

When there's multiple sensors, it's possible that one will have faults while the other is fine. We need a way to allow the user to remove faulty values or get the system to ignore the sensor for a given file.

In each case this would cause the QC to be run again.

Allow concatenation of files?

On the Benguela Stream, a single voyage might be split across two data files. Allow two files to be uploaded and concatenated?

Rewrite the job thread pool

The current job thread pool is broken. In the short term, rewrite it so that threads are created as needed, tracked (to make sure we're not running too many) and then destroyed at the end.

Tidy up JobManager exceptions

The JobManager can throw an awful lot of exceptions (InvalidClass, InvalidConstructor etc.). These all have to be caught outside, but the outside world doesn't need to know these details.

Make all the underlying classes inherit an overall InvalidJobException or similar, so only that one has to be handled.

Coding the stat bars

On the Data Files page, there will be stat bards and pie chart views of some of the data.

The data should be in a Javascript data structure at the top of the file. When the page loads, a script will run to generate the stat bars. Pie charts will be generated on the fly when they are invoked.

Build sensor inputs by loop?

It may be possible to build the sensor inputs in the names.xhtml page using a loop. This depends on whether the value elements (named after bean properties) can be generated programatically.

Track used job threads

The job thread pool does not currently track threads that are running. This can cause problems if threads die unexpectedly, since they are expected to return themselves to the job pool.

The thread pool should keep a list of all running pools and periodically (how?) check that those threads are either still running or dead, and return them to the pool accordingly.

Move configuration file path

The configuration file path is currently in web.xml. Move it to context.xml with appropriate JNDI lookup for consistency.

Setting processing limits

A function to specify the start and end times for a specific file (i.e. ignore all before/after this point) would be handy.

Verify parameters when adding job?

Currently, running addJob doesn't verify the parameters - it leaves that until the job is actually run.

Instant checking of this might be useful, but it might not - the user won't be able to do anything about it anyway, and the job monitor (when it's written) is probably the place to get this stuff checked out.

Handle regex-y separator characters

The String.split method uses a regex. So when we're extracting file data, regex characters need to be recognised and escaped as appropriate.

The odds of anyone using these characters are virtually nil, but even so...

Needs doing for New Instrument and general file uploads.

Locking

Files should be locked when someone is editing them. How?

First login fails

When the application is first started, the first attempt to log in doesn't work - it goes straight back to the login page. After that it's fine.

This may or may not be intermittent.

Add stat bar to top of data file page

Somewhere on the QC page (in the title bar, most likely) show a live count of at least the number of Needs Flag records, and maybe counts of Good, Questionable and Bad records too.

Handling failed jobs

If a job fails, the system tries to write the error to the database. If that fails, it does nothing. It should be put in a log file.

Re-running QC - when should it happen?

When a user has edited flags and suchlike, the QC and/or data reduction should be run again. How will the system know when to do this?

Possible options:

  • User manually triggers it (undesirable)
  • Any change triggers the appropriate action. Have a delay so it doesn't run instantly for every change, or wait until the user leaves the page.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.