Comments (28)
SF 16:
L1: 1536, nn-5af11540bbfe.nnue
nps: 1823913
SF 16.1:
L1: 2560, nn-b1a57edbea57.nnue
nps: 1210743
(with smaller network)
SF 16.1:
L1: 1536, nn-5af11540bbfe.nnue
nps: 1472458
master: 1096393
SF 16, still used a mixed of classical and main net, where the classical part is probably giving the other additional speedup.
SF 16 bench was a mix of positions with NNUE and classical, I think 50/50 split.
SF 16.1 is 100% NNUE in bench.
SF 16.1 uses the main net and the small net, with classical completely removed...
from stockfish.
Yes the comment is outdated but a table not being used anymore and a useless prefetch not being present doesn't make the engine slower, the change was non functional.
The newer versions are slower because the net arch changed, the layers are bigger and the inference has to do more work, hence it's slower (multiplications aren't free), this issue is just a duplicate of the previous one.
from stockfish.
We should probably simplify materialKey away though.
[Edit] Never mind. Looks like we still use it for syzygy.
from stockfish.
Before removing classical eval:Simplify PvNode reduction Jul 11,2023
6a8767a
Nodes/second : 1502286
With removing classical eval: Jul 11, 2023
https://github.com/official-stockfish/Stockfish/commits/master/?before=0716b845fdef8a20102b07eaec074b8da8162523+315
Nodes/second : 885742
Same net
I don't understand why.
from stockfish.
As Disservin already said, it is because of 50% of the bench positions used only the classical evaluation, which was much faster
To compare only the "real" Stockfish, use bench 16 1 13 default depth NNUE
for commits before the removal of classical
from stockfish.
Done
bench 16 1 13 default depth NNUE
before
Nodes/second : 924580
after
Nodes/second : 769656
I'm very sorry, but yet I don't understand.
from stockfish.
I will answer you, though I'm already regretting it (interaction with you is sickening).
Classical Eval role, was mianly about delevering not so good eval in compansation of speed, so even with using NNUE as default you still use Classical Eval in winning positions.
from stockfish.
Lines 1056 to 1059 in 6a8767a
from stockfish.
command: ./stockfish bench 16 1 13 default depth NNUE;
commit : 6a8767a0d5d9502e6d4de1bef97468b5d6fab80a (Simplify PvNode reduction)
changes: with classical part removed in evaluate
nps : 1111266
command: ./stockfish bench 16 1 13 default depth NNUE;
commit : 6a8767a0d5d9502e6d4de1bef97468b5d6fab80a (Simplify PvNode reduction)
nps : 1180150
command: ./stockfish bench 16 1 13 default depth NNUE;
commit : af110e02ec96cdb46cf84c68252a1da15a902395 (Remove classical evaluation)
nps : 1112121
from stockfish.
OK: so, for two elo points, do you degrade performance so fearfully?
from stockfish.
What
from stockfish.
from stockfish.
how is this relevant to your problem about the benchmark nps?
from stockfish.
hce wasn't removed to gain 2 Elo, it was removed even if removing it lost 2 Elo, because with long enough tc the impact was 0.
A 0 impact on performance is not performance degradation, the fact the engine reports a slower bench speed because hce isn't there to inflate the metric doesn't mean anything.
from stockfish.
In general arguing about hce removal now is useless, it's a done deal, new and better solutions now fill the void the hce removal left, devs are working on sf 17 and stuff that was definitive in sf 16 is definitely not relevant now.
from stockfish.
For me, it is crazy and I don't agree at all, but clearly do as you wish.
from stockfish.
you don't understand anything you talk about, that's all there is to it, as usual.
Ofc switching over to hce inflates the speed, you are avoiding the nnue slowdown by not using the nnue at all, switching back to fully hce eval would make the engine faster too, but so what?
speed is just a metric, you can sacrifice speed to gain elo, it's what the hce to nnue switch did, moreover removing hce isn't that much of a speed drop, bench shouldn't have been mixed in the first place and the metrics it reports are inflated by the positions that are > 2048 cp in value.
Moreover part 2 current sf dev has a solution to speed up the engine in won positions, it's a smaller net that takes the place of what was hce before (without the need to keep all that hce code around).
For once you should try to understand what you are talking about before second guessing tens of devs that all know far more than you.
from stockfish.
You are consistently as confident as you are wrong, and that's a really hard feat, stop second guessing devs when you don't understand how a nnue arch works or what a pv node is, it's pathetic.
from stockfish.
For me, it is crazy and I don't agree at all, but clearly do as you wish.
I don't even get your point... on my benchmarks #5157 (comment),
it shows that the actual true bench speed of stockfish was 1180150
and after removing HCE it was 1112121
.
That difference is marginal... for some reason people hate it when sf has less nps because it is not using hce anymore, but it is fine for you to have less speed because of a marginally better larger network and some even use this network without any clear evidence that it is stronger even before Stockfish master has it! (looking at you talkchess)
HCE was removed with some loss at STC, the higher marginal higher NPS which Stockfish had before were countervailed with the stronger evaluation.
from stockfish.
I develop a fork of stockfish with the goal of making it understandable/useful to the OTB player, not necessarily, therefore, stronger, and I wondered why this frightening degradation of performance. That's all.
from stockfish.
how is a 2 Elo impact at bullet time controls a "frightening degradation of performance"?, why are we acting like the basically 0 Elo impact (at the very very very long time controls you otb players love) wasn't made even less impactful by the fact smallnet exists? the truth of the matter is that sf is now stronger than when hce was still present and it's stronger than it would be if hce was still present.
You are wondering wrong things based off of wrong assumptions and either outdated or wrong "knowledge" (or lack thereof).
from stockfish.
I develop a fork of stockfish with the goal of making it understandable/useful to the OTB player, not necessarily, therefore, stronger, and I wondered why this frightening degradation of performance. That's all.
NPS doesn't mean anything, if you want high nps go back to pure alpha beta but lose accuracy of the evaluation. Your speed comparison that you posted earlier were wrong since it used a mixed bench, which is not even what stockfish did during regular play. In my two tests, the difference was like 5%, which is not "frightening degradation of performance" imo.
What we aim for (and everyone else in the community who wrote their own engine which competes at a high level) is elo gain. Over the time we have merged many new networks which made STC a bit weaker (reduced nps) but overall scaled rather well to high time controls... which is also what you are after...
from stockfish.
@amchess You don’t need to prove yourself to anyone. You don’t need to spend your energy trying to earn other’s acceptance or waste your time trying to explain yourself. You don’t need to justify your ideas to others who won’t understand. Spend your time doing your thing. Spend your energy learning and growing. Pour yourself into your ideas and dreams and let go of what others are saying…. Because in the end, it just doesn’t matter what they think — it only matters what you think.
from stockfish.
It doesn't matter what anyone thinks, this is about numbers, nps, elo and time controls. This is equivalent to math, it doesn't matter if i think that 3 + 2 is 7 or if my dream is for 12 * 10 to be 97, this is factual information that can only be correctly or incorrectly interpreted, there's no room for opinions here.
That being said it's pretty evident that you are a troll so there's no use engaging with you and i won't do so any further.
from stockfish.
Commits on Jan 14, 2024
Remove the dependency on a Worker from evaluate
before refactor
Nodes/second : 826439
after refactor
Nodes/second : 822816
Same net and bench
The refactoring isn't no functional changes.
Just about facts, numbers...
from stockfish.
that's a supposedly 0.4% speed drop, it's too small of a drop to be measured with a single bench run or even 3 or 5, it's easily explained away by noise and error bars in the bench measurement.
If you want an accurate way to measure speed use the script that is customarily used to measure for speed gains/losses. you'll notice that with enough runs the change is just neutral.
The refactoring is definitely non functional
from stockfish.
Commits on Jan 14, 2024 Remove the dependency on a Worker from evaluate
before refactor Nodes/second : 826439
after refactor Nodes/second : 822816
Same net and bench The refactoring isn't no functional changes. Just about facts, numbers...
A single bench run isn't very reliable. You could probably redo a single bench at some point and get reversed results. You should be using one of the many speedup scripts to better determine the effect of a patch on the overall performance.
from stockfish.
Here you can find all the info on how to properly measure the speed of stockfish: https://github.com/official-stockfish/Stockfish/wiki/Advanced-topics#measure-the-speed-of-stockfish, coincidentally it also says A speedup of 0.3% could be meaningless (i.e. within the measurement noise)
which is exactly what you are measuring.
from stockfish.
Related Issues (20)
- Stockfish release 16.1 for vnni256 not working. HOT 2
- How to compile Stockfish 14.1 on Apple M1 MAX macOS for ipmanchess ranking list? HOT 13
- Update from arm_version: '8' to arm_version: '9.2' is needed HOT 2
- Lots of issue about `position fen` and `go` command HOT 5
- export_net only exports the bignet HOT 1
- The windows VNNI binary does not work. HOT 10
- Stockfish > 16 ignores setoption after go HOT 18
- False evaluations for positions with multiple knights HOT 2
- Issue with the benchmark HOT 8
- wrong mated in scores HOT 1
- Stockfish needlessly hangs the queen HOT 14
- Winning evaluation with tablebases in cursed win HOT 19
- issue when running the .exe of version 16.1 on Windows. HOT 1
- expect not properly spawning Stockfish in github actions.
- eval command breaks after AccumulatorRefreshTable merged
- Faulty analysis - too big change of position value HOT 2
- Disappointed with the project HOT 1
- test
- Suboptimal speed on multi-socket / numa systems HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stockfish.