das-developers / das2py-server Goto Github PK
View Code? Open in Web Editor NEWA CGI based data stream reduction and caching service for das2 readers
License: GNU General Public License v3.0
A CGI based data stream reduction and caching service for das2 readers
License: GNU General Public License v3.0
In many situations it would be handy if the core pyserver could find out how recently a sub-set of data had been modified. To stick with the overall pyserver mentality, the interface for asking this question should be handled by a sub-program. My proposed way to handle this is to add a new processes type:
This is as opposed to read process type which directly provides data. Though a full set of metadata items need not be defined here, the most immediately useful one is newest_mod_time
. This metadata column should contain the most recent modification timestamps for all input files needed by a reader over the given coordinate range, or coordinate range set.
For a more concrete example, a given meta reader might have an interface as follows:
waves_lrs_meta begin end interval metadata_item1 metadata_item2 metadata_item3
So given the call:
waves_lrs_meta 2022-01-01 2022-01-05 86400 newest_mod_time
The following response might be generated (using 2.3/basic-xml
stream format in this case):
<stream version="2.3/basic-xml" >
<packet id="1">
<x pdim="time">
<array type="isotime" use="min">
</x>
</y pdim="newest_mod_time">
<array type="isotime" use="max">
</y>
</packet>
<d id="1">2022-01-01 2022-01-15T14:47:32.345</d>
<d id="1">2022-01-02 2022-01-15T14:47:33.664</d>
<d id="1">2022-01-03 2022-01-15T14:47:33.976</d>
<d id="1">2022-01-04 2022-01-15T18:03:18.768</d>
...
</stream>
This new capability could be utilized by the das2-pyserver cache system to determine which cache blocks are out of date, and thus avoid blind refreshes which are inefficient.
There are other uses for metadata information (as noted in the comments below) so this basic capability might affect external interfaces as well.
The configuration instructions should include information how to configure SELinux to allow cgi scripts to be placed in an alternate directory. I don't know how to do this after spending an hour on the problem. SELinux fails silently, not just at the console but also in it's log file /var/log/audit/audit.log
as well.
I've activated HAPI on a Python3 installation. The HAPI
landing page works, but the sub-pages (capabilities
, catalog
...) are broken.
The error is
Error loading module catalog from das2server.h_api: No module named 'error'
For reference, the util
module is working, so there should be a way to make h_api
work.
The setup.py script should not overwrite the following...
... if they already exist in the install area.
Furthermore there should be a setup.py option to skip example data sources installation.
Darrelle and I were trying to get a script to work to serve Jade data, and while the script would work when called directly from Autoplot, the Das2Server would clip the data at the year boundary. The mistake was that validRange was set, so the server would clip the call to the Autoplot source.
It would be helpful if the error message were more clear, that the requested interval was outside the bounds of validRange. https://github.com/das-developers/das2-pyserver/blob/5c2d4f75e0a5ad82303ef69554b4128294aee9a9/das2server/defhandlers/dsdfDataset.py#L117 is where the error is thrown, and perhaps a message could indicate if the requested interval was before the validRange or if it was after the validRange.
I'm thinking about the fact that we develop code for both python2.x
and python3.x
. There are a few issues here, directely linked to this (#3, #4, #5...). The extra work required for having a single code for both version is probably not adequate.
Since Python 2.7 will not be maintained any more in less than 40 days, I think we should take a hard turn. I would propose to have a dedicated branch for the python3.x version. Let's also decide also which version of python 3.x we support. I'd advocate personally for Python 3.7.
What do you think ?
The user's guide needs a fair bit of work. It is not complete and may need some reorganization.
Notetable problems include:
A top-level script, say das2_srvcgi_admin should be created that can be used for basic
maintenance task such as kicking off pre-cache builds, cleaning up log files and seeing which datasets get the most use.
The task queue programs das2_svr_arbiter and das2_srv_todo are broken under python 3.
CDF based das2 readers relying on spacepy.pycdf expect to write to their home directory, which fails for web-services. Maybe providing a standalone pycdf as part of das2py is a solution.
We need a way to have at least one worker automatically start at system boot.
The most common ways to handle this are systemd unit files and systemV init scripts. Maybe examples of each should be placed in PREFIX/etc by the install command.
Furthermore the worker program must be tested as a properly behaved daemon. Reliance on the python-daemon package may be the way to go.
When attempting this action:
curl -i "https://jupiter.physics.uiowa.edu/das/server?dataset=Juno/WAV/Jovicentric&server=dsdf"
the server returns a 500 (internal server error), when it should return 404 Not Found
instead.
The error (see below) looks like a missing .decode()
to convert <bytes>
to <str>
.
dataset request was
https://das2server.obs-nancay.fr/das2_new/server?server=dataset&start_time=2019-11-20T11%3A00%3A00.000Z&end_time=2019-11-20T11%3A00%3A10.000Z&resolution=0.017761989342806393&dataset=Nancay%2FNenuFAR%2FSpectra%2F20191120_095800_20191120_125900_SUN_TRACKING%2FStokes_I
Technical Contact: [email protected]
error:
Exception in handler: das2server.defhandlers.dsdfDataset
Traceback (most recent call last):
File '/var/www/html/das2_new/server', line 455, in main
nRet = H.handleReq(U, sReqType, dConf, fLog, form, sPathInfo)
File '/usr/local/das2server_new/lib/python3/das2server/defhandlers/dsdfDataset.py', line 138, in handleReq
nRet = U.auth.authorize(dConf, fLog, form, sDsdf, dsdf['readAccess'])
File '/usr/local/das2server_new/lib/python3/das2server/util/auth.py', line 334, in authorize
nRet = checkGroupAccess(dConf, fLog, form, sResource, sValue)
File '/usr/local/das2server_new/lib/python3/das2server/util/auth.py', line 225, in checkGroupAccess
(sUser, sPasswd) = _getUserPasswd(fLog)
File '/usr/local/das2server_new/lib/python3/das2server/util/auth.py', line 74, in _getUserPasswd
lAuth = sAuthPlain.split(':')
TypeError: a bytes-like object is required, not 'str'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.