Self-host your compiler, you cowards!

Several times this year, I’ve read about new systems programming languages, usually created with robotic help. That’s great! New languages expand how we think about programming. I’ve noticed their compilers are almost always written in Rust.

Rust is a fine systems programming language in its own right, but a systems programming language, the kind of language meant for writing things like compilers, ought to be used for its own compiler before it’s used for any other serious work.

I’m kidding about the cowards part of course. I’m sure all language designers are brave and kind, even if some look like serial killers.

@programming

@[email protected]

#programming #languagedesign #rust #compilers

  • ryokimball@infosec.pub
    12·
    3 days ago

    Bootstrapping.

    Similarly, you should be able to build the OS on the OS, which is why Android is still not a viable operating system compute-wise.

    • QuetzalcutlassEnglish
      6·
      3 days ago

      Google recommends at least 64 gigabytes of RAM for building modern versions of Android (though apparently you can get away with 16 gigs with heavy use of memory compression). Has any Android device ever even come close to having that amount?

        • QuetzalcutlassEnglish
          2·
          2 days ago

          There’s a breakdown in the link in my previous comment:

          Why does AOSP build consume so much memory

          From my experience, building a custom ROM based on AOSP indeed requires about 30-40GB of memory. It may be more for ROMs with more requirements, such as LineageOS-based ROMs.

          Just lower the number of jobs!

          Well, this doesn’t work. Why? The Android build process occurs in multiple stages:

          • Build dependency analysis: The build system (Soong) analyzes de build dependencies by parsing the .bp blueprint files. A dependency graph is generated to determine the build order, with relationships between thousands of modules. As far as I know, the number of jobs doesn’t matter: everything is loaded in memory, and that uses about 30 GB of memory or more with Android 14 (this includes RAM and swap). This step uses a huge fixed amount of memory, regardless of the number of jobs.

          • Compilation and Linking: these steps can be run in parallel, so the number of jobs will determine how much memory will be needed. From my experience, with 8 jobs, less than 30 GB are used.

          • Image generation: It is quite RAM-hungry, but I remember it consumes less than 10 GB on my build setup.

            • QuetzalcutlassEnglish
              3·
              2 days ago

              Apparently there are thousands of code modules and Soong keeps literally everything in memory while calculating dependencies? Which makes me wonder what the hell the blueprint files actually do if it’s still so absurdly heavy.

              Considering how the requirements went from four gigs to over thirty over the years, it wouldn’t shock me if RAM use grows exponentially with the number of modules and they just put up with it because the Google devs use cloud machines with practically infinite resources for building and therefore don’t have any impetus to fix it.

      • ryokimball@infosec.pub
        3·
        2 days ago

        I’ve seen some with 24, more commonly 12 or 16 though. Plus I’ve been several versions of Android running on PC

  • HaraldvonBlauzahn@feddit.org
    52·
    2 days ago

    That’s actually a disadvantage of Rust: Because the Rust compiler is written in Rust, it is hard to bootstrap on new platforms. Which is for a systems language used to build kernel device drivers a serious limitation, and also makes it harder to tackle Thompon’s “Trusting Trust” problem.

    Better to compile the new language to C first, which compiles and runs even on a smart lightbulb.

    This does work well. It is how C++ was created, and even some Lisp implementations go this way.

    • Miaou@jlai.lu
      5·
      2 days ago

      Rustc is built atop llvm, and there’s been a lot of work on the GCC front-end. Your comment makes the situation sound worse than it is

      • calcopiritus
        1·
        1 day ago

        I don’t see how LLVM helps with bootstrapping.

        Yes, it helps with supporting newer platforms, since LLVM is a big project and will probably have support for the new platforms relatively fast.

        But as long as any part of the whole process is in rust, it means that you need a rust compiler to build the rust compiler, thus needing to solve the bootstrapping problem.

        Of course, every compiler has this issue (even if not self-hosted, you still have to bootstrap the compiler of the language your language is written in).

        The only solution for this is to have a compiler-chain where the first level is a simple compiler that can be relatively easily written in the assembly language of the new platform.

        EDIT: or of course I could’ve read any other comment. Of course you can use one of the intermediate representations that are portable to build the first compiler. In which case you don’t need a rust compiler, just need LLVM. Makes sense then.