All valid points. But regarding the measurements, neither LTT or GN testing methodology are realy scientific. Those are youtubers with limited understanding of experimental design and analysis. I have never seen them do simple significance tests or try to explain the variability in the data. But I havent watched all the videos so 乁( •_• )ㄏ
For the variability point, they do tests in as a controlled environment as they could, and do the tests until they get consistent data. But what do you mean by significance test and how can they do it?
I agree they’re not the gold standard but they’re the best we got in terms of independent third party testers, and I would assume they’re more than good enough for tech stuff.
Lets say you run a single test and collect 10 samples at steady-state for temperature and power. This data will have some degree of variability depending on many factors such as airflow at that exact moment, CPU utilization and also inherent noise in the measurement device itself. Additionally, if you repeat the same test multiple times on different days with different testers, you will not get the exact same results.
So if you then compare a system A to system B you might see that system B is 12% “better” (depending on your metric), then you must answer the question–> is this observed difference due to system B actually being better or can this difference be explained by the normal variability in your test setup. Most of the time there are so many external factors influencing your measurement that even if you see a difference in your data, this difference is not significant but due to chance. You should always present your data in a way that its clear to the reader how much variability was in your test setup.
Gamers Nexus has entire videos dedicated to their methodology for different tests. No, there are not p values on their charts; however GN does a great job discussing their experimental design and how it mitigates confounding factors to the extent that GN can control for them.
Can you provide a link to that video? And also a link to a good example how they present the data and the different confounding factors? Do they pubish the data somewhere other than in the videos? Thanks 👍
All valid points. But regarding the measurements, neither LTT or GN testing methodology are realy scientific. Those are youtubers with limited understanding of experimental design and analysis. I have never seen them do simple significance tests or try to explain the variability in the data. But I havent watched all the videos so 乁( •_• )ㄏ
For the variability point, they do tests in as a controlled environment as they could, and do the tests until they get consistent data. But what do you mean by significance test and how can they do it?
I agree they’re not the gold standard but they’re the best we got in terms of independent third party testers, and I would assume they’re more than good enough for tech stuff.
Lets say you run a single test and collect 10 samples at steady-state for temperature and power. This data will have some degree of variability depending on many factors such as airflow at that exact moment, CPU utilization and also inherent noise in the measurement device itself. Additionally, if you repeat the same test multiple times on different days with different testers, you will not get the exact same results.
So if you then compare a system A to system B you might see that system B is 12% “better” (depending on your metric), then you must answer the question–> is this observed difference due to system B actually being better or can this difference be explained by the normal variability in your test setup. Most of the time there are so many external factors influencing your measurement that even if you see a difference in your data, this difference is not significant but due to chance. You should always present your data in a way that its clear to the reader how much variability was in your test setup.
Gamers Nexus has entire videos dedicated to their methodology for different tests. No, there are not p values on their charts; however GN does a great job discussing their experimental design and how it mitigates confounding factors to the extent that GN can control for them.
Can you provide a link to that video? And also a link to a good example how they present the data and the different confounding factors? Do they pubish the data somewhere other than in the videos? Thanks 👍
Idk why they don’t just run the test 3 times and average the result.
It would fix a lot of these issues… If you are testing a cooler, remove, clean CPU, reapply thermal compound and reseat cooler.
It’s really not that hard to make their testing, maybe not perfect, but a lot better
Hardware unboxed does this if I remember correctly
I’m not really sure if you ever watched a GN video, but he throws facts at you at a rate which no one else i know does.
He knows his stuff and is always open to learn new things.