No experiment, no proof. But, taken with a grain of salt a good survey can be better than pure speculation where experiment is impossible or unethical. On the other hand experiments can prove something, but depending on how reduced or artificial the context they may not be proving as much as you hope, either. Science is just difficult in general.
Exactly. Luckily I’m in a field where true experiments are possible, but I have many colleagues who can’t ethically run true experiments. It’s surveys or nothing for the most part. They have very advanced statistics to account for the lack of control in their research.
And even if you can carry out a proper experiment, it might be useful to see of there’s already a survey on the same topic. If there is, you can use that data to design your experiment, and hopefully you’ll be able to take important variables into account.
where true experiments are possible, but I have many colleagues who can’t ethically run true experiments. It’s surveys or nothing for the most part. They have very advanced statistics to account for the lack of control in their research.
There is a whole discipline on causal inference with observational data that is more than a hundred years old (e.g. John Snow doing a diff-in-diff-strategy). Usually, it boils down to not having to control for every detail but to get plausibly exogeneous variation in your treatment either due to a policy only implemented in one group(state), a regulatory threshold, or other “natural experiments”. Social scientists typically need to rely on such replacements for true experiments. Having a good survey is only the first step before you even think at how you could potentially get at the effects of interest. Looking at some correlations in a survey is usually only some first descriptive to find interesting patterns. Survey design itself is a whole different problem. There you also have a experiments and try to find how non-response and wrong answers work. For example, there are surveys in scandinavia, the netherlands, france in Germany that can easily be linked to social security (or even individual credit card data in the danish case) to validate answers or directly use high-quality administrative data.
No experiment, no proof. But, taken with a grain of salt a good survey can be better than pure speculation where experiment is impossible or unethical. On the other hand experiments can prove something, but depending on how reduced or artificial the context they may not be proving as much as you hope, either. Science is just difficult in general.
Exactly. Luckily I’m in a field where true experiments are possible, but I have many colleagues who can’t ethically run true experiments. It’s surveys or nothing for the most part. They have very advanced statistics to account for the lack of control in their research.
And even if you can carry out a proper experiment, it might be useful to see of there’s already a survey on the same topic. If there is, you can use that data to design your experiment, and hopefully you’ll be able to take important variables into account.
There is a whole discipline on causal inference with observational data that is more than a hundred years old (e.g. John Snow doing a diff-in-diff-strategy). Usually, it boils down to not having to control for every detail but to get plausibly exogeneous variation in your treatment either due to a policy only implemented in one group(state), a regulatory threshold, or other “natural experiments”. Social scientists typically need to rely on such replacements for true experiments. Having a good survey is only the first step before you even think at how you could potentially get at the effects of interest. Looking at some correlations in a survey is usually only some first descriptive to find interesting patterns. Survey design itself is a whole different problem. There you also have a experiments and try to find how non-response and wrong answers work. For example, there are surveys in scandinavia, the netherlands, france in Germany that can easily be linked to social security (or even individual credit card data in the danish case) to validate answers or directly use high-quality administrative data.