beapi-io / spring-boot-starter-beapi Goto Github PK
View Code? Open in Web Editor NEWSpringboot 'convention over config' starter for API Automation
License: Other
Springboot 'convention over config' starter for API Automation
License: Other
adding in functionality to allow services to be declared for business logic as well as controllers.
Literally would just have to add a TYPE to IO state [CONTROLLER,SERVICE] and dynamically call service and route in BeapiController (much the same way I do with controller)
This provides alternative way of building, flexibility and way to build internal static calls (which I need) for the tool.
This way I can also add localizd IO state files to the project without adding controllers.
need a functional test to test params via:
so rather than having the endUser have to put their public uri's in 'apiProperties.reservedUris', lets just assume that ALL uri's implemented with '@RequestMapping' (or variations thereof) are public... or at least not handled by the starter (thus we assume they are public).
We can check this by checking:
Localized public apis will have no reason to create IO state files.
PLUS... if they are annotated with @controller, we can create a 'publicApis' field in the cache when we are creating the mappings as they will not map to a state file.
This will have to be tested.
apidocs need to send mockdata so we can build out the 'api call example' and potentially add back in sandbox.
Right now the 'api call' example is incorrect
need to have a boolean toggle for BATCH/HOOKS in apidocs that tells whether this functionality is available to this user based on their roles in this networkGrp
since I can't (and don't want to) use a DB in starter, we can store in cache and backup to disk the webhooks. This reduces need for developers to implement controllers/services and everything can be internalized without DB.
Just have to implement into existing cache functionality.
So after much head banging against desk, realizing I have to create a map of cached values that can be passed in properties/cache(more likely cache) to implementing app so they can build a configurationSource for cors
@Override
protected void configure(HttpSecurity http) throws Exception {
http.csrf().disable().cors().configurationSource(configurationSource()).anyRequest().requiresSecure();
...
}
private CorsConfigurationSource configurationSource() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();
config.addAllowedOrigin("*");
config.setAllowCredentials(true);
config.addAllowedHeader("X-Requested-With");
config.addAllowedHeader("Content-Type");
config.addAllowedMethod(HttpMethod.POST);
config.addAllowedMethod(HttpMethod.GET);
config.addAllowedMethod(HttpMethod.PUT);
config.addAllowedMethod(HttpMethod.DELETE);
source.registerCorsConfiguration("/**", config);
return source;
}
}
Just have to send the map of networkgrp allowed origins per 'controller' and create the rules via antMatcher. This right here just became a WHOLE SEPARATE project... UGHG!
Need to check request details to make sure token hijacking isn't occurring (yes, this CAN be spoofed but this provides an additional layer that they have to spoof as well when doing a MITM; this allows us to log out user and pre-emptively warn at the very least)
String ip = request. getRemoteAddr()
String userAgent = request.getHeader("User-Agent");
NOTE: will have to have associated table ('a given User has one /many RequestDetail and RequestDetail has one User')
Found this on Stackoverflow. This is sloppy and should be converted to compiled regex...
String browserDetails = request.getHeader("User-Agent");
String userAgent = browserDetails;
String user = userAgent.toLowerCase();
String os = "";
String browser = "";
log.info("User Agent for the request is===>"+browserDetails);
//=================OS=======================
if (userAgent.toLowerCase().indexOf("windows") >= 0 )
{
os = "Windows";
} else if(userAgent.toLowerCase().indexOf("mac") >= 0)
{
os = "Mac";
} else if(userAgent.toLowerCase().indexOf("x11") >= 0)
{
os = "Unix";
} else if(userAgent.toLowerCase().indexOf("android") >= 0)
{
os = "Android";
} else if(userAgent.toLowerCase().indexOf("iphone") >= 0)
{
os = "IPhone";
}else{
os = "UnKnown, More-Info: "+userAgent;
}
//===============Browser===========================
if (user.contains("msie"))
{
String substring=userAgent.substring(userAgent.indexOf("MSIE")).split(";")[0];
browser=substring.split(" ")[0].replace("MSIE", "IE")+"-"+substring.split(" ")[1];
} else if (user.contains("safari") && user.contains("version"))
{
browser=(userAgent.substring(userAgent.indexOf("Safari")).split(" ")[0]).split("/")[0]+"-"+(userAgent.substring(userAgent.indexOf("Version")).split(" ")[0]).split("/")[1];
} else if ( user.contains("opr") || user.contains("opera"))
{
if(user.contains("opera"))
browser=(userAgent.substring(userAgent.indexOf("Opera")).split(" ")[0]).split("/")[0]+"-"+(userAgent.substring(userAgent.indexOf("Version")).split(" ")[0]).split("/")[1];
else if(user.contains("opr"))
browser=((userAgent.substring(userAgent.indexOf("OPR")).split(" ")[0]).replace("/", "-")).replace("OPR", "Opera");
} else if (user.contains("chrome"))
{
browser=(userAgent.substring(userAgent.indexOf("Chrome")).split(" ")[0]).replace("/", "-");
} else if ((user.indexOf("mozilla/7.0") > -1) || (user.indexOf("netscape6") != -1) || (user.indexOf("mozilla/4.7") != -1) || (user.indexOf("mozilla/4.78") != -1) || (user.indexOf("mozilla/4.08") != -1) || (user.indexOf("mozilla/3") != -1) )
{
//browser=(userAgent.substring(userAgent.indexOf("MSIE")).split(" ")[0]).replace("/", "-");
browser = "Netscape-?";
} else if (user.contains("firefox"))
{
browser=(userAgent.substring(userAgent.indexOf("Firefox")).split(" ")[0]).replace("/", "-");
} else if(user.contains("rv"))
{
browser="IE-" + user.substring(user.indexOf("rv") + 3, user.indexOf(")"));
} else
{
browser = "UnKnown, More-Info: "+userAgent;
}
log.info("Operating System======>"+os);
log.info("Browser Name==========>"+browser);
Since I am not seeing feedback for the plugin, I'm going to integrate an automatic issue creation tool in case anyone has problems with build. This will mitigate 'shyness' and give me the reports I want
Need to investigate a few options and see best way to do this but since most of GITHUB is api driven, should be very simple.
Unlike HATEOAS that AUTOMATICALLY sends back a ton of additional data that you may not want, I wanted to make this optional upon REQUEST.
You can send a HEADER ('X-LINK-REQUEST') for the related links and it wouyld provide a 'concatenated'response:
"response": {
...
},
"links": {
"/accounts/12345/deposit":{
"POST",
"REQUEST":{"id"},
"RESPONSE":{"id","amount","acctNo"}
}
}
Naturally like everything else, this would be related to the requesting users ROLES in their token.
This makes us more feature compliant with HATEOS while not being as bloated as HATEOS
Look into using a custom servlet rather than using filter; should be faster and more functional and reduce collisions with othger filters.
All in all just better solution if I can do it this way.
https://www.baeldung.com/context-servlet-initialization-param
ChainExchangeService is not properly changing the PRECHAIN request method to GET for followup API calls in chain and thus it throws ERROR for all PRECHAIN calls (as it should).
This is easily fixed by doing a detect and changing method to GET for all following calls in a prechain call
BatchFunctionalTest can install ROLES that do not get properly rolled back causing errors upon next run.
This is an easy fix if you know to just uninstall the authorities from the database but you shouldn't have to do this.
We need to have transaction and rollback in our services
need to test changing of cache variables on the fly (ie diskExpiryThreadIntervalSeconds)
Since these variables can be changed on the fly without a restart, would be invaluable to make this part of caching functionality (and also very hazardous if people get security incorrect)
These will never be an issue for caching because only GET calls are cached.
the more I look at encrypting/hashing the PKEY/FKEY/IDX, the more I ponder encryption as an option and replacing hashing altogether.
Streaming cyphers are faster and perform similarly (at least for our purposes) and using encryption would make it easier for testing and rotating hash (or encryption in this case).
The excuse previosuly has been that it would be easier for testing but didn't want to go down this route because it opens privacy concerns.
This is why we would need to make this a VERY OPEN and WELL DOCUMENTED set of processes to maintain trust if we go down this route.
Providing security means providing TRUST.
networkGrp has a rateLimit which concatenates with ROLE rateLimit and even USER rateLimit
In beapi_api.yml
rateLimit: {'ROLE_USER':1000}
dataLimit: {'ROLE_USER':1000000}
if ROLE rate limit does not exist, defaults to rateLimit for networkGrp. If networkGrp rate limit does not exist, no rateLimit is applied (UNLIMITED)
NOTE: would need to add two new fields to user table to accomodate for this.
the open apis (ones that do not require sign-in or token) will not have these checks applied and therefore will be unlimited.
This promotes the ability to SELL additional access to api both for ROLE(company wide) and for user(individual)
CachedResult resides in Request initialization filter and does not properly take into consideration results for ROLES. Needs to be adjusted to consider requesting ROLE.
Also should consider automating Controller automation for cache an return data off a common object and role:
LinkedHashMap data = generateReturnData(Object object, String Role)
return data
Just realized I can simplify a few things by loading properties files via a CDN. This should be added to the SDK
All will download same file from CDN at runtime but PARENT can reload and push (via webhook) to CHILD servers
Need to build tests for ThrottleCacheSrvice to make sure rate limiting is working properly
I spit on */* as an ACCEPT type!!! If you don't know WHAT THE HELL you are sending me, then I don't know if I'm accepting it! Standards need to lock this shit down and stop creating engineering docs for 'elbows that turn 360 degrees'!
DEFAULT THIS SHIT to the content-type (if thats what we are sending back, then thats what they should be sending in 90% of cases) else JSON. UGH!
To better secure users, we need to record 'device' used with token and to compare upon in initial REQUEST in order to see if token was 'hijacked'
https://github.com/ahand/mobileesp/blob/master/Java/UAgentInfo.java
empty/null responses should set statusCode to 204 (not 200)
Hi,
I'm not sure if this is the correct place to post issues regarding documentation, but it is currently deployed on gh-pages from this repo so I thought I'd post it here.
The instructions for configuring the demo project reference a state of the config repo that contained beapi_api.yaml in the root of the repo, but this no longer seems to be the case. I assumed I needed to copy the config folder for the dev enviornment.
Similarly, I noticed that boot.sql was deleted recently (in this commit, but the current installation/setup instructions don't reflect this change.
Going to need to separate out all properties based on role:
etc
Need to have list of available SDK server endpoints and what they are for.
in spring-boot-starter-beapi-config repo, we need the config stored in separate branches for version like we do all code.
Lets start this startying in 0.7
As part of this process, lets go through config and determine what is deprecated and start cleaning things up and consolidating.
Creation of consolidated script for all scaffolding
beapi scaffold-controller full.class.name
beapi scaffold-domain full.class.name
beapi scaffold-connector full.class.name
BRILLIANT FEATURE!!!
give dev ability to run with logging value of 'devnotes' (much like INFO/WARN/DEBUG) so you can get 'developer notes on why a HTTP STATUS code may have occurred'
Better yet... rather than logging, can be emailed letting them know of error and possible solutions
Needs to be internationalized but this can avoid TONS of question and further automate platform and make easier to use.
Only reading in one file right now when we need to read in everything from dir.
see 'parseResource' in IoStateService
we are using getIOSet in RequestInitializationFolter to BUILD the requestList/responseList with each endpoint call
This should be built at runtime as part of ApiDescriptor as a finished list able to be reference by:
Need to have a property in each IO State URI to allow a toggle for 'CacheOn'
This would allow developers/maintainers to set whether we 'cache' return sets or not.
The reasoning is that some endpoints may ALWAYS want LIVE DATA.
should consider automating Controller automation for cache and return data off a common object and role:
LinkedHashMap data = ApiObjectService.generateReturnData(Object object, String Role)
return data
This simplifies ALL controller generation and aids in automation.
We can more easily template off this and autoGernerate controller as well
right now have default values for caches but as people scale, these need to be overwritten.
Can come up with an 'autocalc' but who are we kidding... these are Java devs, we need a separate properties file. :)
Since we are using BOTH RequestMappingHandlerMapping and SimpleUrlHandlerMapping, we can get parameters from the SpringcontextholderAwareRequestWrapper via request.getParameterMap()
This can greatly reduce our processing
APIdocs are autogenerated based on ROLE so USER should NOT see ADMIN endpoint, etc
Need to build tests to make sure this is working properly
Need to add a HookExchangeService for handling CRUD functionality for all endpoints.
Have already added routing so just have to make sure an exchange is there to process it.
badly formatted URI's are falling to RequestMappingHandlerMapping where (if not found) throw a long error:
java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
we need to build functionality to at LEAST catch this error so logs don't fill due to badly formatted URI's
implementors may want to use uList and thus we need to convert to the UriObject we already have built; have no clue why we used uList (probably just a placeholder to test prior to using object)
The following classes are affected:
Fix in THAT ORDER!
so while RESTful advocates like /controller/? to map to CRUD rest calls, it is very limiting.
And to date I have just thrown an error with behaviour like this.
But what if I were to return the apidoc data for the entire controller if no action is sent?
Useful and better than nothing. Plus can help with create your api calls and api chaining.
Bootstrapped user should have a limited role mainly for testing. This role can also be used for automated testing.
Bootstrap user with ROLE_TEST instead of ROLE_USER and edit IOSTATE/connectors to have this role as a limited baseline (or default to permitALL)
most everyone is calling these 'connectors' now so we should probably rename these.
Everything should work the same; just need to rename directory and change where directory points (plus change documentation reference).
adding in api docs as a service to autogenerate api docs based on role in token.
Thus (unlike SWAGGER), docs will only show endpoints token has access to.
This is in compliance with OWASP API security
Need DAILY quartz on MASTER that will clean EXPIRED tokens in DB
filter is calling interceptor twice but second call is ignored. Still this is major issue and needs to be addressed
Need to test if request/response variable TYPE is PKEY/FKEY and HASH (do not hash an 'INDEX' - if this isn't in there may need to add)
That way we can return PKEY/FKEY in response and have them sent in a way we can compare to original.
Also need to create a way to change/rotate SALT every 24 hrs
NOTE : randSalt should be implemented as webHook that pushes value to all services which in turn writes it to local properties.
Need to have a config that checks that all values have been changed in :
and any other settings files
This will throw appropriate error (prior to hitting any tests) and exit build.
Literally, this SHOULD only need to be checked after a new install but its good to have something like this handy when you screw up :)
NOTE : Going to hate having to test this because this means having to screw up my environment... ALOT!
Trace needs to have tests built to see if it is working properly
Nexus repo must have reverted to old revision in last day or two as now artifact is missing. Have to reload.
For all API's, we are automating and autopopulating with SimpleUrlHandlerMapping but Spring is defaulting to RequestMappingHandlerMapping with hardcodes values with annotations rather than DYNAMICALLY POPULATING FROM A PROPERTIES FILE.
We need to write our own CORS filter and inject in Security Filter Chjain in place of their CorsFilter and allow it to :
Right now we are creating config for ADMIN/TEST users and then manually bootstrapping from properties
Proper way to do this would be with a service.
May be able to create separate UserBootstrap class that can be added to Application. Need to investigate this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.