Comments (4)
@allanw Can you answer these 2 questions?
Q1. Can I confirm that you are not trying to write anything to aXcelerate via tap-rest-api? Just reading the data out?
Q2. Can you point me to the document that describes the API endpoint you are trying to invoke?
I'm asking this because it's unusual for an API to use POST method while it is not creating or modifying the record.
If you are trying to use tap-rest-api for modifying the data on aXcelerate, tap-rest-api is not a tool to do so.
from tap-rest-api.
Hi @daigotanaka,
- This is just reading data out - no data is being written in this case
- The particular endpoint in question is called
/report/run
(https://admin.axcelerate.com.au/apidocs/) All it does is runs a report which I have saved and returns the data as JSON
I realise that this is an unusual scenario - I'm not actually sure why their API requires this to be a POST method.
from tap-rest-api.
Hi @allanw
Thanks for the clarification and reference.
I'm not quite sure if this fits to tap-rest-api's intended cases for these 2 reasons (I'm not familiar with aXcelerate's, so I may be wrong):
- From reading the doc, it seems
/report/run
actually triggers aXcelerate's report running process. It would update aXcelerate's stored values instead of just reading out. (Therefore, the POST method instead of GET) - I looked at the example response object. This does not look like what singer.io typically handles. The record is usually a stream of JSON records either with newline separated format or a list (array) within a JSON object that wraps around it. Each record should have a See timestamp_key, datetime_key, index_key section.
For #2, The example I see on doc is:
{
"DATA": [
{
"Qualification Name": "Certificate I in Business",
"Organisation": "VM Learning / aXcelerate",
"Class Descriptor": "# W OUT DATES",
"Trainer Full Name": "",
"Actual Completion Date": "",
"Full Name": "Nathan Gordon",
"Commencement Date": "2015-07-17",
"USI": "",
"Qualification Code": "BSB10112",
"Enrolment Date": "2015-07-17",
"Expected Completion Date": "2015-07-17"
}
],
"ERRORMSG": "",
"REPORTNAME": "Qualification Enrolments and Outcomes",
"PATH": "",
"SUCCESS": true,
"COUNT": 1,
"CONTENTTYPE": "application/json",
"FILTERS": [
{
"VALUE2": "",
"NAME": "enrolments.enrolstatus",
"VALUE": "comp"
},
{
"VALUE2": "",
"NAME": "enrolments.dateenrolled",
"VALUE": "01/01/2015"
},
{
"VALUE2": "",
"NAME": "contacts.givenname",
"VALUE": "Nathan"
},
{
"VALUE2": "",
"NAME": "contacts.surname",
"VALUE": "Gordon"
}
],
"DESCRIPTION": "JSON/Collection"
}
...which seems like a one-page report than a list of records.
It is technically possible to modify tap-rest-api to call this endpoint with POST method and pull the data out. But I'm hesitant in supporting this in this tap because POST call typically requires data as payload. Providing logic to prepare the payload depends on the applications and the use cases. There isn't a good way of authoring that other than developing custom code.
It seems your objective is to call this API periodically to pull the report and load it on a destination data store. Wouldn't it be simpler to just call it with Python's requests library and store the result?
I'm curious to know why you think singer.io or tap-rest-api is suited for this use case. Let me know what you think. More than happy to help what you need to achieve whether you use tap-rest-api or not.
from tap-rest-api.
Hi @daigotanaka, thanks for the detailed reply!
- I see what you mean here and you're right, this API endpoint actually runs a report. However, I'm not sure if it does actually update any stored values. If it does, then a POST would make sense, but I'm also new to this API so I could be wrong too.
- In that example, there is only one result in the
DATA
array. In my case, this will in fact return an array with lots of objects in it, one for each row of data in the report. This is the only thing I care about retrieving from the API, and I can get this working by settingrecord_list_level: "DATA"
in the config file.
To get this working in my test environment, I did actually modify tap-rest_api
code a little bit to be able to do a POST request.
You make a good point that this could be done using the requests library and then storing the result, but I was keen to stay within the Singer tap/target ecosystem as this is how I'm extracting other sources of data on a regular schedule, and ideally I'm trying to avoid writing too much bespoke code.
Interested to hear any more feedback you have. I appreciate your help and understand why you'd be hesitant about adding POST
as an additional method without a good use case for it.
from tap-rest-api.
Related Issues (16)
- OAuth refresh token support
- end_datetime is not honored when timestamp_key is used
- missing variable for max_page logging
- Infer schema mode produces null record that causes "CRITICAL list index out of range" error HOT 1
- Fully support multiple streams
- Prevent auto-conversion from null to false, 0, 0.0, '' and etc
- Invalid type on json schema causes crash HOT 6
- OpenAPI specification input HOT 2
- Migrate to SDK HOT 1
- UnboundLocalError: local variable 'streams' referenced before assignment HOT 2
- `No such file or directory` found when running discovery HOT 2
- Use digest for dup check instead of raw record HOT 1
- Wrong params for singer.utils.backoff decorator
- Pagination by unique keys
- Support record_level & record_list_level at arbitrary levels
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tap-rest-api.