dyalog / jarvis Goto Github PK
View Code? Open in Web Editor NEWAPL-based web service framework supporting JSON or REST
Home Page: https://dyalog.github.io/Jarvis/
License: MIT License
APL-based web service framework supporting JSON or REST
Home Page: https://dyalog.github.io/Jarvis/
License: MIT License
The function HtmlPage needs a UTF-8 conversion of the result, else the result has type 160 and cannot be returned via Conga.
Allow a setting such that the client can be assigned a session without first calling StartSessionEndpoint.
Such a request could also include basic HTTP authentication credentials.
It appears that the check below is intended to prevent the standard JSON Server header checking in cases when we are trying to access the HTML interface. However, it appears that checking is also disabled when the HTMLInterface is set to 0. If this is the case, then it seems that we will never check the headers for correctness, even though it seems that we should?
Line 528 in ccd05ff
If the user does a request.Fail in the endpoint code, Jarvis should not do any further processing on the response.
Since Jarvis.EndPoints now looks at the ⎕AT of possible endpoints and this is executed within the CodeLocation hierarchy, and CodeLocation may have non-default settings for ⎕ML and ⎕IO, EndPoints needs to be ⎕ML and ⎕IO insensitive.
And if empty, any requests that would return the defaultPage should return a 404. (minimal processing to avoid too much CPU load through "invalid activity")
I am getting no response when I run the Jarvis Demo:
srv.CodeLocation←'..\Samples\JSON\'
srv.Start
2020/02/12 @ 21:53:35 - Jarvis started in "JSON" mode on port 8080
2020/02/12 @ 21:53:35 - Serving code in #.CodeLocation (populated with code from "C:/Users/arcfi/OneDrive/Documents/Jarvis/
Samples/JSON/")
2020/02/12 @ 21:53:35 - Click http://localhost:8080 to access web interface
0
]load HTTPCommand
#.HttpCommand
cmd←⎕NEW HttpCommand
cmd.(Command URL)←'POST' 'localhost:8080/GetSign'
cmd.Headers⍪←'content-type' 'application/json'
cmd.Params←'[10,31]'
q←cmd.Run
q.rc
0
q.Data
q
[rc: 0 | msg: "" | HTTP Status: 200 "OK" | ⍴Data: 0]
You’ll notice that I set up the interface for the HandleCORSRequest
function so that it will signal whether there was a CORS request in its result. The hope here is that by doing this, the same function might be able to be used for the REST handler as well, just that the REST handler will be able to “pass along” instead of returning right away as the JSON handler does. The question remains then, how to appropriately set the Methods allowed header for the REST defaults.
As an initial default, so that there are no API changes required, it might be good to just use the RESTMethods
defaults. The HandleCORSRequest
handler could be modified to take an optional left argument that contains the methods that will be supported. This might allow the handler to be more flexibly deployed on demand at different locations once the support methods are known.
My current vote for "fast to initial working" solution is to add an optional left argument to HandleCORSRequest
that will set the methods, and then pass RESTMethods
as that left argument when we call HandleCORSRequest
from the HandleRESTRequest
handler. [That's a lot of hands.]
Is there a test suite for Jarvis, and if so, how do I run it?
When a request's headers exceed BufferSize we currently handle it as a DRC.Wait error and close the connection. I believe the more proper way to handle it is to issue a HTTP 431. This will give the client more information on how to correct the request.
431 Request Header Fields Too Large
The server is unwilling to process the request because its header fields are too large. The request may be resubmitted after reducing the size of the request header fields.
The following line will only be checked if the method is a 'get' method.
Line 561 in d2c5e53
There are many more methods that might be passed besides 'get' and 'post' and the current handler assumes that if the method is not a 'get' method, then it is a 'post' method.
Currently, if Jarvis is loaded using the Jarvis.dws workspace, relative file paths are determined from the workspace location. This has some implications for the dyalog/jarvis container because it uses that workspace. I believe a more flexible (and user friendly) solution would be to use the location of the JarvisConfig file, if it exists, as the root folder from which relative folders are found. This makes Jarvis apps more portable.
Line 1723 in 4c23d76
sint←{⎕IO←0 ⋄ 83=⎕DR ⍵:⍵ ⋄ ((⍳,-+⍳)128)[utf8 ⍵]}
same goes for HttpCommand:
https://github.com/Dyalog/HttpCommand/blob/2659ea89357f85cbe10d38797500c629c1f2cb8b/source/HttpCommand.dyalog#L958
Am I correct in understanding that the web interface does not currently work? When visiting the root of a server I get the following:
LAUNCH
2020/02/12 @ 21:20:02 - Jarvis started in "JSON" mode on port 8080
2020/02/12 @ 21:20:02 - Serving code in #.server
2020/02/12 @ 21:20:02 - Click http://localhost:8080 to access web interface
0
2:VALUE ERROR: Undefined name: n
ScriptFollows[2] r←{⍵/⍨'⍝'≠⊃¨⍵}{1↓¨⍵/⍨∧\'⍝'=⊃¨⍵}{⍵{((∨\⍵)∧⌽∨\⌽⍵)/⍺}' '≠⍵}¨(1+n⊃⎕LC)↓↓(180⌶)2⊃⎕XSI
If the client connection is closed while an endpoint (requested by the connection) is executing, provide a switch to tell Jarvis to kill the request-handling thread.
I happened to define an ambivalent AuthenticateFn where a monadic one was expected.
The session output (after starting APL with jarvis.dws and a configfile-argument) was:
⍎Unable to start server - "#.CITA_Server. 0
Server←AutoStart
∧
If the default is DEBUG, then the container will exit immediately unless -it or RIDE_INIT are set.
As our policy is to support the 3 most recent releases of Dyalog, that would at present be 18.2, 18.0, and 17.0. As such we need to use 819⌶ for case conversion (at least until Dyalog 19.0 is released and we can use ⎕C).
The Jarvis and JarvisService workspaces contained an earlier version of Jarvis that inadvertently used ⎕C. This was corrected in the Jarvis source code, but the workspaces had not been updated.
The KillSession test here will fail at some point because when Power Op is called with a function instead of a value, it will always run at least once. This means that KillSession will always run at least once. This will also result in potential errors depending on the datatypes used for sessions and a number of other undesirable behaviors.
Line 548 in d2c5e53
If you are attempting to access a route and something unexpected happens, such as a GET request instead of a POST request for the JSON Server, the server does not respond with any data. I believe this is due to a resource leak located around here:
Line 529 in ccd05ff
This line and the two/three after it appear to return from the HandleRequest
function having updated the response object with errors, but never actually sending the response. As a result, I believe that HandleRequest
returns a 0 code at the end and does not expunge the object, but then does nothing with it. This essentially leaves the object in la la land for an indeterminant amount of time and the result is an unresponsive system on the front end.
When a new, valid request comes in, it appears to trigger something that pushes out some clean up and results in an error code 1105 for these responses:
2020/02/15 @ 20:59:20 - RunServer: DRC.Wait reported error 1105 ERR_RECV Could not receive data on SRV00000000.CON00000000 (IP Address [::1]:53760)
2020/02/15 @ 20:59:20 - RunServer: DRC.Wait reported error 1105 ERR_RECV Could not receive data on SRV00000000.CON00000001 (IP Address [::1]:53761)
Please update the container to use v19.
Each HTTPHeader, HTTPBody, HTTPChunk and HTTPTrailer event is passed to HandleRequest
in a new thread.
HandleRequest
is wrapped in a :hold on the connection name.
I've recently discovered or rediscovered that :hold is not a fair mutex. So, if a few events on the same connection queue up on the :hold, I think they will not get access to the critical section in order.
This probably is not a big issue for un-chunked requests, as there are only two events that can happen (Header then Body), so there are never multiple threads waiting on the mutex (This could change if pipelining were implemented).
But I'm concerned about chunked messages. Is it not possible that chunks could queue up waiting for the mutex and then be serviced out of order?
It would be really useful.
Thanks Michael
Maybe prefix with JARVIS_
to avoid clashes?
The problem:
Jarvis has a "ConnectionTimeout" setting intended to clean up connections that have gone dormant. The default value is 30 seconds. Currently, we may inadvertently close a connect for a request to endpoint that runs for longer than ConnectionTimeout.
The suggestion:
Add a "in-progress" flag to the connection namespace that the connection reaper will respect and not close connections for requests that are being serviced.
With the merge of the KillSession
adjustments, they use a r
under the assumption that it was the return value.
Line 547 in 03ae42e
With HandleRequest
being changed to not return a value, this r
is no longer localized.
When using Jarvis in REST mode, req.Fail
can put error information from ⎕EM
or ⎕SI
(depending on ErrorInfoLevel
) into the HTTP response header as the "reason phrase". However, HTTP clients can ignore the reason phrase, proxies can change it and it was removed in the HTTP/2 spec.
I think the error information should be put as the response body/payload when req.Fail
is invoked.
It appears to me that expunging the obj
is handled in the call to HandleRequest
when it returns a 1. Is there a reason that we use an r=0 and then manually expunge the object at the following line?
Line 539 in ccd05ff
Swagger UI is a free library used for API documentation.
The use of it is so wide spread it can be considered industry standard.
In C# the library is available as a nuget package that can be installed and configured for each individual project.
https://learn.microsoft.com/en-us/aspnet/core/tutorials/web-api-help-pages-using-swagger?view=aspnetcore-7.0
It would be nice if this was available in Jarvis as well, preferably by default.
The current Jarvis doesn't appear to handle CORS situations when browsers access the data. Is there a clean and minimal way to do this in Jarvis itself that doesn't get too "over-engineered?"
Definitely not an issue: more a curiosity. I am intrigued by this line:
Line 479 in 8c53cea
Why the #.? I have very similar code in one of my projects (unsurprisingly since I copied it from JsonServer) but I don't think I went for the "#.". Is there a good reason to create that particular namespace as a child of root?
Thank you!
I am very new to APL, but I have been working on a proof-of-concept web application to demonstrate the potential use of Dyalog / Jarvis to others. However, I keep getting these seemingly random crashes - sometimes it only takes a minute, sometimes it takes half an hour, but Jarvis always seems to exit with a cryptic "OS / Undefined Name" error. I could be wrong, but I don't think this is related to the code I am running. I've tried hitting my endpoints with good data and error cases, but I can't find a pattern.
I've attached the error I've been receiving along with the code I am running under Jarvis. I am using the Jarvis docker container, and have included the command invoking it. Is this an issue with Jarvis itself, or just a problem with the code I'm running?
For what its worth, I've tried this both under WSL and Xubuntu and it seems to behave the same in both cases.
Let me know if you have any questions :)
My session reports:
2024-06-09 @ 04.25.13.962 - Local Conga v4.3 reference is #.Rumba.[LIB]
when the version is v3.4.
A cursory inspection of the code indicates this appears to perhaps be more than a cosmetic issue as the private field CongaVersion
that is referenced is also used in other places and tested against numerically.
The culprit appears to be the uncommon locution:
0.1⊥2↑LDRC.Version
which, with a well placed rotate
might work in some cases, but would still fail in others.
CreateSecureParms has a logic error when not using ServerCertSKI
As suggested at Dyalog'22 SA2 workshop
It seems in the following line, that the URL is decoded after splitting on '?' but before splitting on &
and =
in the query string. This would seem to be incorrect, as this would treat encoded &
and =
values as query separators, which they are not.
Line 874 in 048bf3b
I created simple functions for Get, Post, Put, ... like so:
r←Get req
r←req.Endpoint
Then I issued a request from Insomnia.
For Get everything works fine.
For all other methods, I need something in my request body to make it work.
Otherwise jarvis ignores it.
Especially for Delete this makes no sense to me.
Help needed,
Thx,
Karsten
│Jarvis│1.16.3│2024-01-12│
IDE:
Version: 4.4.3687
Platform: MacIntel
Date: 2022-02-23 13:44:34 +0100
Git commit: a3d8123cfb690ead493e261ccced59407db04c64
Preferences:{
"colourScheme":"Dracula",
"confirmations":"{\"SaveFileOptionsExtension:.aplf\":100,\"SaveFileOptionsExtension:.apln\":100}",
"kbdLocale":"en_US_Mac",
"lbarOrder":"← +-×÷*⍟⌹○!? |⌈⌊⊥⊤⊣⊢ =≠≤<>≥≡≢ ∨∧⍲⍱ ↑↓⊂⊃⊆⌷⍋⍒ ⍳⍸∊⍷∪∩~ /\\⌿⍀ ,⍪⍴⌽⊖⍉ ¨⍨.⍣∘⍤⍥@ ⍞⎕⍠⌸⌺⌶⍎⍕ ⋄⍝→⍵⍺∇& ¯⍬ ",
"menu":"# see below for syntax\n\nDyalog {mac}\n About Dyalog =ABT\n -\n Preferences... =PRF\n - \n &Quit =QIT\n&File {!browser}\n &Open... =OWS {local}\n &New Session =NEW\n &Connect... =CNC\n - {!mac}\n &Quit =QIT {!mac}\n&Edit\n Undo =UND {!browser}\n Redo =RDO {!browser}\n - {!browser}\n Cut =CT {!browser}\n Copy =CP {!browser}\n Paste =PT {!browser}\n Select All =SA {mac}\n - {mac||!browser}\n &Find... =SC\n Find and &Replace... =RP\n - {!mac}\n Preferences... =PRF {!mac}\n&View\n Show Language Bar =LBR\n Show Status Bar =SBR\n Show Workspace Explorer =WSE\n Show Debug =DBG\n Line Wrapping in Session =WRP\n - {!browser}\n Show Status Window =SSW\n Auto Status =ASW\n - {!browser}\n Stops =TVB\n Line Numbers =LN\n Outline =TVO\n - {!browser}\n Increase Font Size =ZMI {!browser}\n Decrease Font Size =ZMO {!browser}\n Reset Font Size =ZMR {!browser}\n - {!browser}\n Toggle Full Screen {!browser}\n&Window\n Close All Windows =CAW\n&Action\n Edit =ED\n Trace =TC\n -\n Clear all trace/stop/monitor =CAM\n Weak Interrupt =WI\n Strong Interrupt =SI\n&Threads {rp21}\n Pause on Error =POE\n Pause all Threads =PAT\n Unpause all threads =UAT\n Continue all threads =MA\n&My\n Dyalog\n Mastering APL =https://mastering.dyalog.com/README.html\n -\n APL Cart =https://aplcart.info\n APL Wiki =https://aplwiki.com\n&Help\n Getting &Started =https://dyalog.com/introduction.htm\n -\n Dyalog &Help =DHI\n &Language Elements =LEL\n &Documentation Centre =DOX\n -\n Dyalog &Website =https://dyalog.com/\n &Email Dyalog =EMD\n -\n Latest &Enhancements =ENH\n Read &Me =RME\n &Third Party Licences =TPL\n &About =ABT {!mac}\n\n# Syntax:\n# &x access key, alt+x\n# =CMD command code; some are special:\n# LBR WRP WSE render as checkboxes\n# =http://example.com/ open a URL\n# {} conditional display, a boolean expression\n# operators: && || ! ( )\n# variables: local browser mac win\n# - separator (when alone)\n# # comment\n\n# The =PRF (\"Preferences...\") menu item must be present.",
"otherExe":"/Applications/Dyalog-18.2.app/Contents/Resources/app/../Dyalog/mapl",
"wse":"1",
"zoom":"3"
}
Interpreter:
Version: 18.2.45505
Platform: Mac-64
Edition: Unicode/64
Date: Apr 7 2022 at 00:12:12
I'm guessing pipelining is not supported by Jarvis?
If I send multiple requests off from a client on the same keep-alive connection, without waiting for a response from each one, I get:
23:VALUE ERROR: Undefined name: Req
HandleRequest[21] ns.Req.Thread←⎕TID
I'm not sure if there is much reason to support pipelining, but if Jarvis does not, it's not exactly clear to me what it should do. One option is to service the first message, and then simply close the connection, ignoring all the other incoming requests on the connection. I don't think the server can send back an error response.
Add code to allow starting as root to allocate a port <1024 and then switch to non-root userid
Add HTTP compression to Jarvis
Add a setting to limit the maximum payload size for a request. Default ¯1 meaning "no limit".
When a request fails (HTTP Status ≠ 2XX) enable the user to define a custom handler
I suspect 819⌶
(case convert) needs to be replaced with ⎕C
or wrapped with an error-guarding dfn
DOMAIN ERROR: I-Beam function 819 has been withdrawn; use ⎕C instead
nocase[0] nocase←{(lc ⍺)⍺⍺ lc ⍵} ⍝ case insensitive operator
∧
Currently, only code files (.dyalog, .apl[fonc]) are loaded. .apla files could also be loaded.
One complication is that the APL array notation has evolved over time and we need to take that into account.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.