115
From the 80's to 2024 - how CI tests were invented and optimized
graphite.devWhile Google started automating its build tests in 2003, the engineering industry took longer to do the same. But automation was sorely needed: Software systems are growing larger and ever more complex… To make matters worse, new versions are pushed to users frequently, sometimes multiple times each day. This is a far cry from the world of shrink-wrapped software that saw updates only once or twice a year. The ability for humans to manually validate every behavior in a system has been unable to keep pace with the explosion of features and platforms in most software. - Software Engineering at Googlehttps://abseil.io/resources/swe-book/html/ch11.html#testing_overview Add author/source Sun Microsystem’s engineer, Kohsuke Kawaguchi, was key to ushering in the next era of testing. In 2004, he created “Hudson”https://community.jenkins.io/t/lets-thank-kohsuke-the-creator-of-jenkins/168 (later renamed to Jenkins in fun Oracle drama). At his day job, Kohsuke “got tired of incurring the wrath of his team every time his code broke the build.” He could have manually triggered tests before each code contribution, but instead, Kohsuke chose the classic engineering solution and created an automated program. The Hudson tool acted as a long-lived test server that could automatically verify each code change as it integrated into the codebase.
I have to say, pretty much every CI/CD tool, build automation or whatever you want to call it, sucks.
Somehow they all manage not to offer boilerplate actions since that would be too restrictive, yet they also stand in your way if you want to have advanced features.
I often end up writing pipeline steps/jobs thinking “how is that not already built-in?”.
And my absolutely biggest pain point: why the fuck is there not a single tool that lets me execute pipelines locally? Why do I have to have 200 commits all saying a variation of “pipeline test”?
I agree so, so much; and I’ve been saying similar things for years.
But I recognize that it’s probably a hard problem. For one thing, auth is almost never going to work the same way in CI as it does locally.
…still, though, I feel like there could be some much nicer tooling here.
It can’t be that hard to create a test harness, so I can plug my own variables into the runner context.
Even just an 80% solution would speed up development tremendously.
Well, for GitHub Actions, there’s this: https://github.com/nektos/act
https://github.com/nektos/act
That actually looks quite promising. You don’t happen to know of a similar project for Gitlab runners?
I do not. I’m sorry.
The sentiment of the first half of your comment is the cause of the problem you describe in the second half. Why /should/ the CI tool have any “steps” built in? Use a task runner, or script in your repo for any task you expect CI to do. Configure CI to run the same command you would run locally.
It’s not the same thing. Sometimes there are rules/conditions in the pipeline you need to test. Sometimes there are child pipelines. There really should be a way to simulate a pipeline run locally.
I totally agree, with some caveats:
haven’t tried it myself, but I’ve heard https://earthly.dev/ is supposed to address the execute locally problem