Is there some formal way(s) of quantifying potential flaws, or risk, and ensuring there’s sufficient spread of tests to cover them? Perhaps using some kind of complexity measure? Or a risk assessment of some kind?
Experience tells me I need to be extra careful around certain things - user input, code generation, anything with a publicly exposed surface, third-party libraries/services, financial data, personal information (especially of minors), batch data manipulation/migration, and so on.
But is there any accepted means of formally measuring a system and ensuring that some level of test quality exists?
I am not sure what the common/agreed upon rules are. Seems like it depends on the team lead or manager to decide. Some orgs have better engineers, experience, systems and others don’t.
I used to follow the 100% coverage because I was told to do so in my start. I found myself chasing semi-colons rather than null references. Luckily, I had a team mate with which we argued a lot about what we did, do, and will do and he helped me. (In a friendly manner, not like Dinesh and Gilfoyd from Silicon Valley).
Now, I start my tests by going over how the user will use it, e.g. the happy path. Then happy path away. It seems to cover most cases. It helps if you know the business too. (Think messaging system that is intentionally and strictly simple, or one that has a lot of Unicode and language support… fucking emojis hurt me cause I forget they exist even though I use them all the time, I always forget).
Alas, no matter what, I always miss some test case or a very imaginative user will find a way to show me how wrong I am.
In the end, I think the best, no matter how big or small the project you’re building is, to do many small PRs (with tests) to your team. This way, things are tested in increments and helps prevent PR burnouts. This I need to get better at myself.