Laravel tests not realistic. about php-framework-benchmark HOT 37 CLOSED

I think this discussion basically proves how worthless these types of
"benchmarks" are.

On Mon, Mar 30, 2015 at 4:13 AM, neomerx [email protected] wrote:

@samdark https://github.com/samdark Really not. You will have IoC, HTTP
kernel pipeline, controllers, support for DB/cookies/cache/auth/bus/ and
etc. Not all of them are really needed for the test but cannot be switched
off completely without side effects.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from php-framework-benchmark.

After performing the optimizations on Laravel 5 (like you would in the real world), it clocks in slightly faster than Cake. Please update your tests or remove Laravel as it's not even remotely accurate since it is not installed properly.

from php-framework-benchmark.

Thank you for your feedback! I will update my benchmarks.

from php-framework-benchmark.

@kenjis Nice! Have you updated the README as well?

from php-framework-benchmark.

I ran my benchmark, but current results is like below:

Laravel5 results improved slightly, but not faster than Cake. I don't know why @taylorotwell 's laravel is so fast. If you have any clue, would you please teach me?

from php-framework-benchmark.

@taylorotwell @kenjis the result is realistic BUT
Laravel out of the box comes with middleware such as 'maintenance check', 'cookie encryption', 'session', etc which is not really needed for this particular 'Hello world' test. When it switched off and reduced number of empty/not used providers I got

• from 390 rps (L5) vs 3984 rps (codeigniter)
• to 1514 rps (L5)

If Taylor add some optimizations to piping who knows L5 might get even more 👍 (session and cookie provider that could be really switched off could add even more)

from php-framework-benchmark.

That would leave you w/o framework, wouldn't it? :) Abstraction always costs extra performance, it's not a big surprise.

from php-framework-benchmark.

@samdark Really not. You will have IoC, HTTP kernel pipeline, controllers, support for DB/cookies/cache/auth/bus/ and etc and even console artisan. Not all of them are needed for the test but cannot be switched off completely without side effects.

from php-framework-benchmark.

Not really. These benchmarks show how bare framework performs i.e. what's the overhead. While it matters mostly in cases when application code is perfectly optimal, it still matters. At least sometimes.

from php-framework-benchmark.

I've got to agree with @taylorotwell on this one - it's a useless benchmark. You're not even close to comparing apples to apples here.

from php-framework-benchmark.

@ConnorVG why? Isn't it comparing typical middleware request cycle overhead?

from php-framework-benchmark.

I've made a research (link below) and come to the following conclusions

• Kernel stack, TCP stack, PHP and web server should be optimized to unlock the application potential.
• Requests per second alone cannot be reliable measure for checking application performance. Specialized tools should be used instead.

Way more detailed explanation TL;DR :)
https://github.com/neomerx/rhw-l5#speed-up-real-hello-world-application-laravel-5

from php-framework-benchmark.

Great post. Love how detailed it is and the conclusion.

Still, I think that the benchmark itself is OK since I doubt there are alike significant default template changes @kenjis made to any of frameworks tested. At least I can say so about Yii.

from php-framework-benchmark.

@kenjis Don't get discouraged, these kind of benchmarks are far, far away from useless. I am going to state two, in my opinion, most important:

• Lower bound for resources: these kind of benchmarks provide us with lower bound for resources requirements. With these kind of data, we can project in best case scenario which hardware we need for specific project depending on technology we use and do we need to scale in advance. It is not an ideal, but we have some kind of a picture in front of us (in comparison to nothing).
• Motivation to improve: of course you are going to make some people angry - and that is cool, as long as they will be motivated to improve. Competition is always good and for us as end consumers of frameworks - we do not see anything wrong in making framework even more optimised. Per example, I am SF2 developer, SF2 is much better for development then horrific CI, but CI is much faster (of course it is, it has nothing in it). Speed is not the reason for me to switch to CI, god forbid, but why SF2 should not be as fast as CI? I would surely like to be so :)

Please, keep up with good work! And thank you for this data, this is great contribution to community!

from php-framework-benchmark.

@TheCelavi you might need to know more about framework internals, their differences and got some experience with benchmarking.
This test compares bicycles to tractors where some go down the hill and some go up.
However I agree on motivation.

from php-framework-benchmark.

@neomerx after all this years I still get surprised how people can stoop so low - you have never met me, you have never worked with me, you have no idea who I am, and nevertheless, you entitled yourself with a right to speculate about my education, knowledge and working experience. Let me return you the curtesy of speculating about what do you do - you are Laravel developer, right?

from php-framework-benchmark.

@TheCelavi your statements were unjustified. You basically said 'Guys, listen, I want you to know my unjustified thoughts'. If you have anything meaningful to say welcome https://github.com/neomerx/rhw-l5#speed-up-real-hello-world-application-laravel-5

from php-framework-benchmark.

@TheCelavi why supporting such tests is a bad idea #5

from php-framework-benchmark.

@neomerx I am sorry? What I said was not addressed to "some guys", nor to you, it was mentioned to @kenjis. I said thanks because I see my personal benefit from this, now I have some lover bound for SF2 (which I commonly use), some XY req/sec that can be max achieved - so I can do some rough estimations and projections what do I need, hardware wise, for some project release before I even start to work on it. It is not precise info, but it is a start, and I am ok with that. I am sorry that that kind of info does not have value for you - but it has a value for me. And that data is provided free of charge, so thanks @kenjis again. I would like to buy you a beer - and I am very serious about it.

Regarding second statement - well, I am sorry, from now on, when I recognise something that is good (by my personal opinion, of course) I will try not to step on your toe and publicly express my appreciation and gratitude, I will from now on send a personal note and/or email. This is my first time, so I really hope you are going to forgive me this one.

Regarding "bad idea to support this kind of tests" - if these kinds of tests are unrealistic and/or irrelevant and/or does not contain any value -> why would you waste your time here, why do you care what some stupid test say? It is stupid, right?

And after all - may I remind you that this is a free world, I can support whatever I want. I choose to support this. Go benchmark! Go! And?

from php-framework-benchmark.

@TheCelavi Thank you!

I show you all how to benchmark, so you can freely interpret the results. And you can benchmark on your own environment and tweek them as you like or think worthy.

Regarding second statement - well, I am sorry, from now on, when I recognise something that is good (by my personal opinion, of course) I will try not to step on your toe and publicly express my appreciation and gratitude, I will from now on send a personal note and/or email. This is my first time, so I really hope you are going to forgive me this one.

What is "second statement"? The paragrah is for me or not? Sorry I'm not good at English.

By the way, I updated my benchmarks. Now 13 frameworks.

from php-framework-benchmark.

I applied all artisan command:
https://github.com/kenjis/php-framework-benchmark/blob/master/setup.sh#L63-65

But my Laravel is not as fast as Cake. That is my question.

from php-framework-benchmark.

Nice to see you added Lumen already, good job @kenjis

I tried the benchmarks on my own server (php5.4 + apc, centos, apache), which gave similar results as the results you posted (relatively speaking of course) and puts Lumen around the same rps as Yii 2 and Silex 1.2 (But not 90% fasteras Silex, as stated on the Lumen website, more like 20%).

But of course it all depends on the server you run the benchmarks on..

@taylorotwell Does Lumen need some more optimizing, or is it correctly configured as is?

from php-framework-benchmark.

@kenjis It was response addressed to neomerx which disapproves my saying "thanks" and "I appreciate your efforts" to you in public manner.

from php-framework-benchmark.

Ok these Lumen benchmarks just confirm my suspicions that these benchmarks
are totally screwed up. Lumen is almost twice as fast as Silex and I can
post reproducible bash scripts to demonstrate that.

On Wednesday, April 15, 2015, Nikola Svitlica [email protected]
wrote:

@kenjis https://github.com/kenjis It was response addressed to neomerx
which disapproves my saying "thanks" and "I appreciate your efforts" to you
in public matter.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from php-framework-benchmark.

Please do, it's interesting.

from php-framework-benchmark.

@taylorotwell I myself did one, but not sure Lumen is that fast as mentioned. I am also interested to see the same.

from php-framework-benchmark.

I will literally post a video of my benchmarking these frameworks today to
put this to rest. Including blog.

On Wednesday, April 15, 2015, Hari K T [email protected] wrote:

@taylorotwell https://github.com/taylorotwell I myself did one, but not
sure Lumen is that fast as mentioned. I am also interested to see the same.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from php-framework-benchmark.

@taylorotwell Why not the code as well so we can reproduce your tests on our local machines?

from php-framework-benchmark.

👍

from php-framework-benchmark.

Yes I will post the code because this is the most weird and misleading
benchmarks I've ever seen and I'm sick of hearing about it.

On Wednesday, April 15, 2015, Alexander Makarov [email protected]
wrote:

[image: 👍]

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

from php-framework-benchmark.

from php-framework-benchmark.

http://taylorotwell.com/how-lumen-is-benchmarked/

from php-framework-benchmark.

@taylorotwell why your browser shows port 8000 however you test ab with default port 80 (2:36)?
Is it a homestead feature?

from php-framework-benchmark.

The benchmarks are probably running inside Homestead, on the default port 80. From outside, the port 8000 is mapped to 80 in homestead.

from php-framework-benchmark.

@barryvdh Yep most likely. It just wasn't clear. Anyway even after L5 kicks I think the idea of comparing very different out-of-the-box functionality of frameworks and used methodology are less than optimal.

from php-framework-benchmark.

@taylorotwell Thank you. But unfortunately it seems there is no secret recipe.

So I still don't know why my Laravel is not as fast as Cake.

I close this issue, because I accepted all your suggestions and discussion goes to Lumen.
So I created new issue: #13

My question will be solved if I know why my Lumen is so much slower than yours.
If you guys have some clue or opinion, please post. I appreciate your posts.

from php-framework-benchmark.

I commend the initiative of trying to get benchmarks like these out. Being able to have some comparison of PHP frameworks is really useful when one has a performance critical application. That being said, I think most people who find themselves with a demand that approaches the capacity of these frameworks, probably have the resources to just write straight, application specific PHP code, which will out-perform any framework. Even a "slow" 81 RPS is 6 998 400 unique hits per day, and that's just the ones that would be getting through your cache. And by that point, other things in the request chain will probably be playing a much bigger role in your slowest requests. I feel a lot of these benchmarks neglect to give you a feel for what "reasonable" is with these numbers.

However, given the obvious philosophical differences, with frameworks, I think its dangerous for people to walk away with the conclusion that it provides some baseline or "lower bound". Realistically, all a benchmark like this can (and should) ever provide is what you can expect out of a framework on day one, before you put in any of your own code. I think your primary concern at this point is getting your application started and all you actually need is maybe 4-5 RPS to support development. Some frameworks opt to give you everything you might need so that you can get going quickly and then trim things later when performance is more of a concern. Others give you almost nothing and have you build up what you need as you go. Either approach has pros and cons but taking the results of these benchmarks at face value without normalizing for this factor can only lead to questionable conclusions.

from php-framework-benchmark.