Explanation for newbies:

  • Shell is the programming language that you use when you open a terminal on linux or mac os. Well, actually “shell” is a family of languages with many different implementations (bash, dash, ash, zsh, ksh, fish, …)

  • Writing programs in shell (called “shell scripts”) is a harrowing experience because the language is optimized for interactive use at a terminal, not writing extensive applications

  • The two lines in the meme change the shell’s behavior to be slightly less headache-inducing for the programmer:

    • set -euo pipefail is the short form of the following three commands:
      • set -e: exit on the first command that fails, rather than plowing through ignoring all errors
      • set -u: treat references to undefined variables as errors
      • set -o pipefail: If a command piped into another command fails, treat that as an error
    • export LC_ALL=C tells other programs to not do weird things depending on locale. For example, it forces seq to output numbers with a period as the decimal separator, even on systems where coma is the default decimal separator (russian, dutch, etc.).
  • The title text references “posix”, which is a document that standardizes, among other things, what features a shell must have. Posix does not require a shell to implement pipefail, so if you want your script to run on as many different platforms as possible, then you cannot use that feature.

  • KSP Atlas
    link
    fedilink
    14 minutes ago

    Nushell has pipefail by default (plus an actual error system that integrates with status codes) and has actual number values, don’t have these problems

    • @panicnow
      link
      149 minutes ago

      I even use powershell as my main scripting language on my Mac now. I’ve come around.

    • Badabinski
      link
      fedilink
      53 hours ago

      Lol, I love that someone made this. What if your input has newlines tho, gotta use that NUL terminator!

      God, I wish more tools had nice NUL-separated output. Looking at you, jq. I dunno why this issue has been open for so long, but it hurts me. Like, they’ve gone back and forth on this so many times…

    • @[email protected]
      link
      fedilink
      536 hours ago

      People say that if you have to explain the joke then it’s not funny. Not here, here the explanation is part of the joke.

      • @[email protected]
        link
        fedilink
        42 hours ago

        It is different in spoken form, written form (chat) and written as a post (like here).

        In person, you get a reaction almost immediately. Written as a short chat, you also get a reaction. But like this is more of an accessibility thing rather than the joke not being funny. You know, like those text descriptions of an image (usually for memes).

      • @marcos
        link
        55 hours ago

        I really recommend that if you haven’t, that you look at the Bash’s man page.

        It’s just amazing.

        • @felbane
          link
          44 hours ago

          I think you mean amanzing.

          I’ll see myself out.

  • Badabinski
    link
    fedilink
    606 hours ago

    set -euo pipefail is, in my opinion, an antipattern. This page does a really good job of explaining why. pipefail is occasionally useful, but should be toggled on and off as needed, not left on. IMO, people should just write shell the way they write go, handling every command that could fail individually. it’s easy if you write a die function like this:

    die () {
      message="$1"; shift
      return_code="${1:-1}"
      printf '%s\n' "$message" 1>&2
      exit "$return_code"
    }
    
    # we should exit if, say, cd fails
    cd /tmp || die "Failed to cd /tmp while attempting to scrozzle foo $foo"
    # downloading something? handle the error. Don't like ternary syntax? use if
    if ! wget https://someheinousbullshit.com/"$foo"; then
      die "failed to get unscrozzled foo $foo"
    fi
    

    It only takes a little bit of extra effort to handle the errors individually, and you get much more reliable shell scripts. To replace -u, just use shellcheck with your editor when writing scripts. I’d also highly recommend https://mywiki.wooledge.org as a resource for all things POSIX shell or Bash.

    • @[email protected]
      link
      fedilink
      7
      edit-2
      4 hours ago

      Putting or die blah blah” after every line in your script seems much less elegant than op’s solution

      • Badabinski
        link
        fedilink
        64 hours ago

        The issue with set -e is that it’s hideously broken and inconsistent. Let me copy the examples from the wiki I linked.


        Or, “so you think set -e is OK, huh?”

        Exercise 1: why doesn’t this example print anything?

        #!/usr/bin/env bash
        set -e
        i=0
        let i++
        echo "i is $i"
        

        Exercise 2: why does this one sometimes appear to work? In which versions of bash does it work, and in which versions does it fail?

        #!/usr/bin/env bash
        set -e
        i=0
        ((i++))
        echo "i is $i"
        

        Exercise 3: why aren’t these two scripts identical?

        #!/usr/bin/env bash
        set -e
        test -d nosuchdir && echo no dir
        echo survived 
        
        #!/usr/bin/env bash
        set -e
        f() { test -d nosuchdir && echo no dir; }
        f
        echo survived
        

        Exercise 4: why aren’t these two scripts identical?

        set -e
        f() { test -d nosuchdir && echo no dir; }
        f
        echo survived
        
        set -e
        f() { if test -d nosuchdir; then echo no dir; fi; }
        f
        echo survived
        

        Exercise 5: under what conditions will this fail?

        set -e
        read -r foo < configfile
        

        And now, back to your regularly scheduled comment reply.

        set -e would absolutely be more elegant if it worked in a way that was easy to understand. I would be shouting its praises from my rooftop if it could make Bash into less of a pile of flaming plop. Unfortunately , set -e is, by necessity, a labyrinthian mess of fucked up hacks.

        Let me leave you with a allegory about set -e copied directly from that same wiki page. It’s too long for me to post it in this comment, so I’ll respond to myself.

        • Badabinski
          link
          fedilink
          54 hours ago

          From https://mywiki.wooledge.org/BashFAQ/105

          Once upon a time, a man with a dirty lab coat and long, uncombed hair showed up at the town police station, demanding to see the chief of police. “I’ve done it!” he exclaimed. “I’ve built the perfect criminal-catching robot!”

          The police chief was skeptical, but decided that it might be worth the time to see what the man had invented. Also, he secretly thought, it might be a somewhat unwise move to completely alienate the mad scientist and his army of hunter robots.

          So, the man explained to the police chief how his invention could tell the difference between a criminal and law-abiding citizen using a series of heuristics. “It’s especially good at spotting recently escaped prisoners!” he said. “Guaranteed non-lethal restraints!”

          Frowning and increasingly skeptical, the police chief nevertheless allowed the man to demonstrate one robot for a week. They decided that the robot should patrol around the jail. Sure enough, there was a jailbreak a few days later, and an inmate digging up through the ground outside of the prison facility was grabbed by the robot and carried back inside the prison.

          The surprised police chief allowed the robot to patrol a wider area. The next day, the chief received an angry call from the zookeeper. It seems the robot had cut through the bars of one of the animal cages, grabbed the animal, and delivered it to the prison.

          The chief confronted the robot’s inventor, who asked what animal it was. “A zebra,” replied the police chief. The man slapped his head and exclaimed, “Curses! It was fooled by the black and white stripes! I shall have to recalibrate!” And so the man set about rewriting the robot’s code. Black and white stripes would indicate an escaped inmate UNLESS the inmate had more than two legs. Then it should be left alone.

          The robot was redeployed with the updated code, and seemed to be operating well enough for a few days. Then on Saturday, a mob of children in soccer clothing, followed by their parents, descended on the police station. After the chaos subsided, the chief was told that the robot had absconded with the referee right in the middle of a soccer game.

          Scowling, the chief reported this to the scientist, who performed a second calibration. Black and white stripes would indicate an escaped inmate UNLESS the inmate had more than two legs OR had a whistle on a necklace.

          Despite the second calibration, the police chief declared that the robot would no longer be allowed to operate in his town. However, the news of the robot had spread, and requests from many larger cities were pouring in. The inventor made dozens more robots, and shipped them off to eager police stations around the nation. Every time a robot grabbed something that wasn’t an escaped inmate, the scientist was consulted, and the robot was recalibrated.

          Unfortunately, the inventor was just one man, and he didn’t have the time or the resources to recalibrate EVERY robot whenever one of them went awry. The robot in Shangri-La was recalibrated not to grab a grave-digger working on a cold winter night while wearing a ski mask, and the robot in Xanadu was recalibrated not to capture a black and white television set that showed a movie about a prison break, and so on. But the robot in Xanadu would still grab grave-diggers with ski masks (which it turns out was not common due to Xanadu’s warmer climate), and the robot in Shangri-La was still a menace to old televisions (of which there were very few, the people of Shangri-La being on the average more wealthy than those of Xanadu).

          So, after a few years, there were different revisions of the criminal-catching robot in most of the major cities. In some places, a clever criminal could avoid capture by wearing a whistle on a string around the neck. In others, one would be well-advised not to wear orange clothing in certain rural areas, no matter how close to the Harvest Festival it was, unless one also wore the traditional black triangular eye-paint of the Pumpkin King.

          Many people thought, “This is lunacy!” But others thought the robots did more good than harm, all things considered, and so in some places the robots are used, while in other places they are shunned.

          The end.

            • Badabinski
              link
              fedilink
              12 hours ago

              No worries! Bash was my first language, and I still unaccountably love it after 15 years. I hate it and say mean things about it, but I’m usually pleased when I get to write some serious Bash.

        • @pivot_root
          link
          13 hours ago

          Exercise 6:

          set -e
          f() { false; echo survived; }
          if ! f; then :; fi
          

          That one was fun to learn.


          Even with all the jank and unreliability, I think set -e does still have some value as a last resort for preventing unfortunate accidents. As long as you don’t use it for implicit control flow, it usually (exercise 6 notwithstanding) does what it needs to do and fails early when some command unexpectedly returns an error.

          • Badabinski
            link
            fedilink
            12 hours ago

            I personally don’t believe there’s a case for it in the scripts I write, but I’ve spent years building the || die habit to the point where I don’t even think about it as I’m writing. I’ll probably edit my post to be a little less absolute, now that I’m awake and have some caffeine in me.

            One other benefit I forgot to mention to explicit error handling is that you get to actually log a useful error message. Being able to rg 'failed to scrozzle foo.* because service y was not available' and immediately find the exact line in the script that failed is so nice. It’s not quite a stack trace with line numbers, but it’s much nicer than what you have with bash by default or with set -e.

    • @renzevOP
      link
      English
      216 hours ago

      I’ve been meaning to learn how to avoid using pipefail, thanks for the info!

    • fmstrat
      link
      fedilink
      English
      35 hours ago

      Yup, and set -e can be used as a try/catch in a pinch (but your way is cleaner)

      • Badabinski
        link
        fedilink
        64 hours ago

        I was tempted for years to use it as an occasional try/catch, but learning Go made me realize that exceptions are amazing and I miss them, but that it is possible (but occasionally hideously tedious) to write software without them. Like, I feel like anyone who has written Go competently (i.e. they handle every returned err on an individual or aggregated basis) should be able to write relatively error-handled shell. There are still the billion other footguns built directly into bash that will destroy hopes and dreams, but handling errors isn’t too bad if you just have a little die function and the determination to use it.

        • @[email protected]
          link
          fedilink
          24 hours ago

          “There are still the billion other footguns built directly into bash that will destroy hopes and dreams, but”

          That’s well put. I might put that at the start of all of my future comments about bash in the future.

          • Badabinski
            link
            fedilink
            22 hours ago

            Yep. Bash was my first programming language so I have absolutely stepped on every single one of those goddamn pedblasters. I love it, but I also hate it, and I am still drawn to using it.

    • @renzevOP
      link
      English
      94 hours ago

      I learned about it the hard way lol. seq used to generate a csv file in a script. My polish friend runs said script, and suddenly there’s an extra column in the csv…

    • @mrvictory1
      link
      44 hours ago

      I had os-prober fail if locale was Turkish, the seperator thing could explain that.

  • @[email protected]
    link
    fedilink
    English
    136 hours ago

    Shell is great, but if you’re using it as a programming language then you’re going to have a bad time. It’s great for scripting, but if you find yourself actually programming in it then save yourself the headache and use an actual language!

    • Phoenixz
      link
      fedilink
      23 hours ago

      So what IS the difference between scripting and programming in your view?

      • @[email protected]
        link
        fedilink
        English
        12 hours ago

        No clear line, but to me a script is tying together other programs that you run, those programs themselves are the programs. I guess it’s a matter of how complex the logic is too.

      • @[email protected]
        link
        fedilink
        English
        126 hours ago

        Your scientists were so preoccupied with whether they could, they didn’t stop to think if they should

        • Badabinski
          link
          fedilink
          65 hours ago

          Honestly, the fact that bash exposes low level networking primitives like a TCP socket via /dev/TCP is such a godsend. I’ve written an HTTP client in Bash before when I needed to get some data off of a box that had a fucked up filesystem and only had an emergency shell. I would have been totally fucked without /dev/tcp, so I’m glad things like it exist.

          EDIT: oh, the article author is just using netcat, not doing it all in pure bash. That’s a more practical choice, although it’s way less fun and cursed.

          EDIT: here’s a webserver written entirely in bash. No netcat, just the /bin/bash binary https://github.com/dzove855/Bash-web-server

    • XIN
      link
      fedilink
      96 hours ago

      The few times I’ve used shell for programming it was in strict work environments where anything compiled was not allowed without a ton of red tape.

        • @[email protected]
          link
          fedilink
          35 hours ago

          For more complicated input/output file handling, certainly.

          Little shell scripts do great though if all you need to do is concatenate files by piping them.

          It’s like the Internet, it’s not one big truck but a series of tubes.

          • @[email protected]
            link
            fedilink
            English
            24 hours ago

            Yep, in my mind piping together other commands is scripting not programming, exactly what shell scripts are for!

            • @[email protected]
              link
              fedilink
              115 minutes ago

              With enough regex and sed/awk you might be able to make even complicated stuff work. I’m not a regex guru (but I do occasionally dabble in the dark arts).

    • @renzevOP
      link
      English
      7
      edit-2
      6 hours ago

      Alpine linux, one of the most popular distros to use inside docker containers (and arguably good for desktop, servers, and embedded) is held together by shell scripts, and it’s doing just fine. The installer, helper commands, and init scripts are all written for busybox sh. But I guess that falls under “scripting” by your definition.

  • @Phoenix3875
    link
    65 hours ago

    A lot of people call set -euo pipefail the “strict mode” for bash programming, as a reference to "use strict"; in JavaScript.

    In other words, always add this if you want to stay sane unless you’re a shellcheck user.

    • Badabinski
      link
      fedilink
      65 hours ago

      People call set -euo pipefail strict mode but, it’s just another footgun in a language full of footguns. Shellcheck is a fucking blessing from heaven though. I wish I could forcibly install it on every developer’s system.

  • @[email protected]
    link
    fedilink
    46 hours ago

    I have written 5 shell scripts ever, and only 1 of them has been more complex than “I want to alias this single command”

    I can’t imagine being an actual shell dev

    • @marcos
      link
      45 hours ago

      only 1 of them has been more complex than “I want to alias this single command”

      I have some literal shell aliases that took me hours to debug…

        • @marcos
          link
          22 hours ago

          Nah, it’s in the past.

          But people, if you are writing a command that detects a terminal to decide to color its output or not, please add some overriding parameters to it.

    • @renzevOP
      link
      English
      5
      edit-2
      4 hours ago

      No, sorry. I’m a python dev and I love python, but there’s no way I’m using it for scripting. Trying to use python as a shell language just has you passing data across Popen calls with a sea of .decode and .encode. You’re doing the same stuff you would be doing in shell, but with a less concise syntax. Literally all of python’s benefits (classes, types, lists) are negated because all of the tools you’re using when writing scripts are processing raw text anyway. Not to mention the version incompatibility thing. You use an f-string in a spicy way once, and suddenly your “script” is incompatible with half of all python installations out there, which is made worse by the fact that almost every distro has a very narrow selection of python versions available on their package manager. With shell you have the least common denominator of posix sh. With Python, some distros rush ahead to the latest release, while other hang on to ancient versions. Even print("hello world") isn’t guaranteed to work, since some LTS ubuntu versions still have python pointing to python2.

      The quickest cure for thinking that Python “solves” the problems of shell is to first learn good practices of shell, and then trying to port an existing shell script to python. That’ll change your opinion quickly enough.

  • blaue_Fledermaus
    link
    fedilink
    -86 hours ago

    I’m glad powershell is cross-platform nowadays. It’s a bit saner.

    Better would be to leave the 1970s and never interact with a terminal again…

    • TimeSquirrel
      link
      fedilink
      23 hours ago

      What about embedded work? You gonna run a full graphical GUI on a network router?

    • @[email protected]
      link
      fedilink
      English
      13 hours ago

      Quick, how do I do for i in $(find . -iname '*.pdf' -mtime -30); do convert -density 300 ${i} ${i}.jpeg; done in a GUI, again?

    • @[email protected]
      link
      fedilink
      10
      edit-2
      6 hours ago

      What else are you going to do, though?

      If you have some particular and complicated task then sure you’d probably write a program for it in a specific high-level language. But that isn’t what the shell is for.

      If you’ve already got a bunch of apps and utilities and want to orchestrate them together to do a task, that’s a good shell use case.

      Or if you have a system that needs setup and install tasks doing on it to prepare for running your actual workload, that’s also a task which the shell is ideally suited to.

      Shell scripting always has a place, and I can’t see it being made obsolete any time soon.

      • @renzevOP
        link
        English
        46 hours ago

        People keep on telling me that python is a “scripting language” but honestly I would rather use the shoddiest and most barebones shell you would give me than python if I had to make a script. It all boils to interfaces, and there are more programs, libraries, and daemons that have a shell interface as opposed to whatever other “better” language there is out there. Trying to write scripts with Python or ruby or whatever will just boils down to plumbing data between external programs in a less succinct syntax. What good is type safety, try/catch, and classes if all the tools that you’re using are taking in and spitting out raw text anyway?

    • @ccunning
      link
      56 hours ago

      I’m a former (long long ago) Linux admin and a current heavy (but not really deep) powershell user.

      The .net-ification of *nix just seems bonkers to me.

      Does it really work that well?

      • @renzevOP
        link
        English
        23 hours ago

        The .net-ification of *nix just seems bonkers to me.

        It IS bonkers. As a case study, compare the process of setting up a self-hosted runner in gitlab vs github.

        Gitlab does everything The Linux Way. You spin up a slim docker container based on Alpine, pass it the API key, and you’re done. Nice and simple.

        Github, being owned by Microsoft, does everything The Microsoft way. The documentation mentions nothing of containers, you’re just meant to run the runner natively, essentially giving Microsoft and anyone else in the repo a reverse shell into your machine. Lovely. Microsoft then somehow managed to make their runner software (reminder: whose entire job consists of pulling a git repo, parsing a yaml file, and running some shell commands) depend on fucking dotnet and a bunch of other bullshit. You’re meant to install it using a shitty setup.sh script that does that stupid thing with detecting what distro you’re on and calling the native package manager to install dependencies. Which of course doesn’t work for shit for anyone who’s not on debain or fedora because those are the only distro’s they’ve bothered with. So you’re either stuck setting up dotnet on your system, or trying to manually banish this unholy abomination into a docker container, which is gonna end up twice the size of gitlab’s pre-made Alpine container thanks to needing glibc to run dotnet (and also full gnu coreutils for some fucking reason)

        Bloat is just microsoft’s way of doing things. They see unused resources, and they want to use them. Keep microsoft out of linux.

      • @[email protected]
        link
        fedilink
        English
        45 hours ago

        What type of .net-ification occurs on *nix? I am current linux “admin” and there is close to 0 times where I’ve seen powershell not on windows. Maybe in some microsoft specific hell-scape it is more common, but it’s hard to imagine that there are people that can accept a “shell” that takes 5-10 seconds to start. There are apps written in c# but they aren’t all that common?

        • @ccunning
          link
          15 hours ago

          I honestly don’t know but there must some translation happening between .net objects and *nix.

          And .net “core” started supporting Linux like a decade ago. I’m guessing they’re related ¯\_(ツ)_/¯

      • blaue_Fledermaus
        link
        fedilink
        16 hours ago

        I usually only use the terminal when I don’t find an alternative, but powershell usually feels a bit saner than bash.

        I’ve also tried nushell, also nice, but in a few situations powershell despite usually being verbose was more “elegant”.

    • @renzevOP
      link
      English
      56 hours ago

      Better would be to leave the 1970s and never interact with a terminal again…

      I’m still waiting for someone to come up with a better alternative. And once someone does come up with something better, it will be another few decades of waiting for it to catch on. Terminal emulation is dumb and weird, but there’s just no better solution that’s also compatible with existing software. Just look at any IDE as an example: visual studio, code blocks, whatever. Thousands of hours put into making all those fancy buttons menus and GUIs, and still the only feature that is worth using is the built-in terminal emulator which you can use to run a real text editor like vim or emacs.