Comments (8)
We could probably get the best of both worlds (free shared hardware + short runs) by normalizing the run with a fixed control. For example, say that our control takes 5 seconds to run on ideal hardware. Suppose we then run the benchmark on shared hardware alongside the control and it takes 10 seconds. We can then normalize the benchmark by multiplying by current_execution_time / execution_time_on_ideal_hardware
or 10 seconds / 5 seconds
. What do you guys think?
from sanic.
I'm fully agree for this idea, but, the main issue is travis-ci: Based on my experience to validate techempower framework benchmarks, I never have two times the same timing between each run, even if I change nothing, because the tests run in a container, executed in a VM: You can't predict the load average of others containers and VMs.
To have a predictable test, you need to use hardware without virtualization and follow the good practices from @Haypo with perf: http://perf.readthedocs.io/en/latest/
Moreover, you need to test always on the same hardware, to enable you to compare between two executions.
Is it a travis-ci like service proposes a service like that ?
from sanic.
Agreed, we need a standard way of doing this. My tests on AWS are not stable and involve me running them over a long period of time until I get consistent results. Even the m4 instances which aren't supposed to burst fluctuate in performance about 1-3k req/sec.
It seems to me like we have 2 options:
- Dedicated hardware - Currently AWS charges $2/hr + fees for dedicated hardware. Unless we can come up with a cheaper solution, I think we'd have to resort to running it less often. Maybe once a week or before releases.
- Shared hardware + long-running tests - Since we're just looking for relative difference in speed between 2 branches, we could just run the tests and compare averages/best run/top percentile results (whichever is most telling). My guess is we'd have to run tests for about 5 minutes to get some consistent results. This may not be that bad, but would definitely be annoying to wait for if the committer is relying on travis to check if if their pull request passed.
I'm inclined to go with option #2, unless there's a solution that lets us spin up dedicated hardware cheaply. What do you guys think?
from sanic.
option 2 seems to be more realistic.
Nevertheless, we might have a new business model just here to launch a new company ;-)
More seriously, it might be a question I could raise to PSF, because they have dedicated hardware to test and benchmark CPython. I'll try to take the temperature.
from sanic.
@channelcat you can use Scaleway.com for cheap dedicated hardware; they have €3/month ARM dedicated servers (basically a hosted RPi alternative), or the cheap dedicated servers offered by Kimsufi.com / Online.net.
from sanic.
profile (and cProfile) might be useful. Could look at changes in number of calls or percentage of time in sanic files relative to overall.
from sanic.
@brian-bates I definitely think that, that idea maybe mixed with @jpiasetz idea could really be helpful for completing this.
from sanic.
Closing due to no follow-up in 2 years. There are discussions on benchmarking in the current community forums (https://community.sanicframework.org)
from sanic.
Related Issues (20)
- 一个求助 HOT 1
- "body not consumed" error on GET request with content-length=0 HOT 1
- mypy error type for async get function in HTTPMethodView HOT 2
- Unexpected behaviour with REQUEST_MAX_HEADER_SIZE HOT 1
- Unexpected behavior when using blueprint groups and app.url_for HOT 1
- How to avoid abort when the code throw an Exception HOT 1
- Sanic drops part of HTTP response data HOT 8
- REQUEST_TIMEOUT & RESPONSE_TIMEOUT not working
- Documentation Hidden by Menu HOT 1
- Native Gzip & Brotli compression HOT 3
- AttributeError: 'Sanic' object has no attribute 'multiplexer' HOT 3
- help pls, when func A and func B are in defference Sanic-Server, same global asyncio.Event() var in them cant sync HOT 1
- TypeError: listener() got an unexpected keyword argument 'priority' HOT 2
- HTTPMethodView subclass not supports generics
- A question about request.json HOT 1
- Add sanic to crowdin
- How can a routing function not pass the request parameter
- Throws an exception when I run with pycharm debug HOT 4
- sanic command line to start the app, developer mode -dev model error HOT 1
- Cors related traceback on `run_async.py`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sanic.