I really don’t understand how complex programs like compilers and browsers have come to an existence.

From little what I know about 8086, it resembles a digital abacus in a way that we can set, push and do all the fancy magic with registers and store them in memories using buttons.

For the like of me, I cannot figure out how an OS was created out of thin air? Do they keep pushing, adding and popping registers back in the day to create OS or compiler? What about the TTY? Was there no such thing as booting? What about file systems? Partitions? How did any of that even work in the first place?

  • @agent_flounder
    link
    English
    32 months ago

    Good question about filesystem. Let me think… yeah, probably think about it as a kind of database, but more like an API for a database that keeps track of file names, file sizes, where the files are stored on the storage device.

    It’s a collection of functions to do useful things like create, read, write, rename files, and list files available.

    It is an abstraction layer on top of the storage device functions.

    If we created tape storage, like on a Commodore 64, we wouldn’t have a filesystem. Our functions would just write or read a sequence of bytes from wherever the tape was, currently.

    With no filesystem, the user has to remember where on the tape the programs are stored (using the tape counter), what the program was (jupiter lander or tic tac toe or whatever), and keep track of what sections of the tape are free. Back in the day I kept this written down in a notebook. It was a pain.

    A filesystem would do this for us. But instead of a sequential storage device like tape, we need to create a random access, block storage device. We’ll make a floppy drive.

    We will divide the storage capacity (100KB, say) into blocks–sets of bytes–of 1KB each. So each disk will have 100 blocks.The drive will read or write to any specified block on command. We can say “read block 7; read block 73; write block 2”. Any order. Doesn’t matter.

    Let’s call our filesystem SimpleFS (SFS). SFS will keep track of the list of files, the names of each file, where those files are stored, how long the files are, what blocks are free on the disk, etc.

    Keeping this dead simple, SFS reserves Block 0 for the list of files (aka file catalog) and for maintaining a list of free blocks. We can use the first 100 bytes to represent the free/used status of the corresponding 100 blocks on disk where 0 represents free and non-zero means used. So if byte 5 is 1 then block 5 has been filled with data.

    The remaining bytes contain the file catalog. Each catalog entry contains a 10 character file name, the file size (16-bit) in 2 bytes, and the starting block # of the file. SFS will only store files in contiguous blocks.

    So now we can write our SFS functions.

    • FORMAT will initialize the storage by writing 110000…0 to the block map to show it is empty and formatted with only block 0 and occupied. And it writes 0’s to the catalog, block 1, to zero it out.
    • DELETE will clear out the catalog entry corresponding to the specified file name, by setting the starting block to zero.
    • LIST sequentially reads the list of files out of the catalog displaying name and size, skipping over directory entries with a 0 starting block.
    • RENAME would change the name of the specified file in the catalog to the new specified name.
    • LOAD would read the specified file from disk into the specified memory location. So LOAD(“myfile”,$1200) would go to disk, find the catalog entry for “myfile”, find out how many bytes to read in, the starting block #, and then read from those blocks into memory.
    • SAVE writes a sequence of N bytes from RAM to the disk, creating a file with the specified filename. So, you’d call SAVE(“myfile”, 73, $1200) to save 73 bytes starting at hex addr $1200 to a file called “myfile”. You can imagine that this function would have to find a free directory entry, find a series of 73 free blocks on disk, copy data to said blocks, then update the block map and the catalog entry.

    You can imagine some flaws with this simplistic design. For example, requiring all the file blocks to be contiguous is rather limiting. Imagine if you want to append to a file but there’s no space available after the last block?

    So you might want to eventually redesign SFS to be able to keep a list of non-contiguous blocks. There are various ways to do it. Since the directory is fixed, maybe you create a new data structure that contains a 0-terminated list of blocks. You refactor your functions to be able to use a non-contiguous set of blocks with this new data structure. And maybe you need a DEFRAGMENT function to reduce fragmentation that will result.

    Having only a flat directory kind of sucks too, so you could redesign it so that you could have subdirectories, each with their own file catalog. Then add functions to deal with path names, change directories, etc.

    Also, it kind of sucks having to know ahead of time how many bytes your program occupies. So instead, lets make it look more like unix. We use an OPEN and CLOSE function. Open can be used to write or read or append to a file. We don’t have to know the file size, just write bytes one at a time.

    We can then revise our editor to read our program a byte at a time from the filesystem and save it a byte at a time to the filesystem. And we can revise our assembler to read the assembly file, bytewise, and output the machine code directly to disk one byte at a time. The SFS can figure out how to update the size data in the catalog and manage which blocks get used.

    Another flaw is that if any of these save operations are interrupted before they’re done, we’re going to lose data and have a mess. So we probably need to figure out a way to reduce the potential damage and a way to repair the filesystem.

    Anyway, hopefully this gives you some ideas of how this stuff can be done.

    I’m vaguely familiar with how some of the classic filesystems work, like DOS FAT, Apple II’s DOS (I forget what it was called), C64’s disk OS (which if I’m not mistaken was actually programmed on the very intelligent but expensive 1541 disk drives), Minix/Linux-sorta-kinda as taught in some of my CompE classes.

    • velox_vulnusOP
      link
      fedilink
      12 months ago

      Wow, so by this, one could assume that there’s a lot of technical debt. And it’s almost impossible to keep track of each and every part that goes into a modern OS. Right now I do not have the luxury to play with RISC-V, but maybe in the near future, I’ll be buying one of their boards to learn hardware in depth.

      • @agent_flounder
        link
        English
        22 months ago

        Based on your comment about technical debt, I should clarify.

        My descriptions aren’t historical at all. They’re just meant to show how it might be possible for you to do this as a personal project. I felt like that might kind of “scratch the itch” of your original question.

        The actual history of operating systems is fairly complex with many companies having developed them independently over time, but perhaps borrowing concepts from previous operating systems and academic papers.

        For example, Windows NT was built more or less from scratch. However, I think I read that a couple things are inspired by prior work of the devs on VAX/VMS.

        But yeah modern operating systems are pretty complex. Too much for any one person to fully and perfectly keep track of.

        Having written all this I think it would be cool to develop the process I described into a series of courses where you learn how computers work, hands on, from discrete transistor logic to logic chips to ICs and building an old school 8-bit computer, all the way up to writing the OS.