(github.com)
I used to study a lot of hobbyist OS development in my late teens. It was awesome, I still try doing small kernels from time to time (last one was a RISCV small kernel that printed a message to my partner).
Given the myriad of resources available, how did you manage to keep the project engaging rather than copying others people code?
Regarding language, I love C for its basic syntax and “straight to the point” style. Maybe I’d consider some other languages for userspace applications.
A goal for me from the start was always that I wanted to write everything myself, no porting of any software, and for me that’s the entire point of the project. I mostly adhered to the rule of copy ideas not code.
See the message from Linus Benedict Torvalds on 26/08/1991
https://groups.google.com/g/comp.os.minix/c/dlNtH7RRrGA/m/Sw...
All I can say is LOL
Technical debt is as sure as death and taxes. Good software grows over time, like a person growing up. Sometimes you have parts that are needed for a time but not permanently, like baby teeth. That's what technical debt is like.
Other times there are parts that are absolutely ideal long term, but it's a long time before they even make sense. And sometimes things only develop much later than we expect, like wisdom teeth. These are software features that won't make sense until you write the thing that you don't realize yet is only temporary.
I have the same goal of writing everything myself from scratch. It's a very important goal to me for a reason I don't fully understand yet.
Congrats on writing an entire OS, and getting it booting on real hardware! That must be such a rewarding feeling in its own right, even if no one understood how. And it looks so very cool.
You've given me more motivation to announce my project soon. Maybe next week. Thanks.
Have you played around with Zig yet? I hear that would be a good replacement for C here. Did you have any thoughts on porting your OS to that, or did you have any branches some Zig code made its way into?
Each approach has a clear set of pros and cons, as others have explored in this thread. In the end, I think what ultimately matters is whether we're satisfied (and proud) of a project after a significant amount of progress has been made.
I also took the "I want to write everything myself" approach with my project since I wanted it to be a demonstration of my abilities. I wanted to prove to myself and others I could create a project of significant size and scope. It seems like this often means working without libraries and having to reinvent the wheel over and over again. It's tedious, but it certainly adds to the satisfaction you get as you look over your work and think "this is truly mine; I did this."
And yes, copy ideas, not code, is one of those things I do my best to adhere to. Sometimes, at least with JS, you run into problems, look into solutions, and it turns out there's really only one sensible way to do something; what you end up writing feels like a boilerplate solution to the problem. I often find myself going back and forth through search results from places like Stack Overflow and trying to siphon out the best techniques and ideas from multiple examples to mesh together into some usable code in a style I prefer.
Finally, since I failed to say this eariler, congrats on your project! I'm gonna have to try it out as soon as I get some extra time.
Having no plan is often better when you're facing a deep pit of work, pick at it every now and again, whatever you feel like working on that day.
vs. having a clear Todo list that is longer than a DNA chain and just giving on it after item 5 because 6 is boring.
Oh, and congrats, looks great.
I don't know anything about your font system; I'm assuming it's fixed width bitmap fonts? I don't know how hard it would be to make these changes within your codebase so if it's too much work then don't worry about it.
I've been building a web OS site from scratch for a while now (not as technically impressive as what you're doing) and I just got through a total rewrite of the font classes, so it's at the top of my mind.
I think people skimming a project often just don't quite get how much we pour into our projects, and how very much it cost us. But it's okay, they don't have to understand just how cool it is what we made. At least we know.
When I saw your video of your project, I was thinking to myself, "wow, that's incredibly cool, and I can relate to the steps he's doing!" Just trying to be the voice here of someone who thought that way, in case there are no others.
Such as https://robey.lag.net/2010/01/23/tiny-monospace-font.html for example.
http://jdebp.uk./Softwares/nosh/guide/terminal-resources.htm...
If you want a quick improvement over that 8by8 IBM ROM font, I suggest two things:
1. Switch to a 16by16 square.
2. For the ASCII range, go with one of the old home computer fonts, rather than IBM ROMs. Viznut has .hex files for Commodore PET and BBC Micro graphics modes squirrelled away inside Unscii.
I've done the work of upscaling the 8by8 PET font to 16by16, and it works quite nicely as a monospace square font. These old home computer fonts were of course designed to. Whereas in the world of VGA you're soon in trouble with glyphs designed for 9by16.
For vector fonts, the interesting question is "if I render this at a different resolution and then scale it, will it line up?" which fundamentally has no answer that can satisfy everyone. Most other difficulties are merely a Small Matter of Coding (and providing appropriate APIs).
- “Look, I wrote an operating system!”
- “Meh the font is weak”
Peak HN IMO :D
The power of the crowd compells you.
The power of the crowd compells you.
My point was: I would not expect such a response anywhere else but HN :)
Someone in the crowd was already there and is wearing the t-shirt - that’s why I come here <3
https://github.com/joexbayer/RetrOS-32/blob/development/grap...
https://github.com/joexbayer/RetrOS-32/blob/development/grap...
There are some tutorials out there too, but a lot of them have bugs and you basically recreate their OS.
Regarding studying opcodes, I never went that deep, closest I got was looking them up for my C compiler, so I know the most common basic ones.
https://forum.osdev.org/ https://wiki.osdev.org/Expanded_Main_Page
You have a short loop that scans through the prefixes, checks for a REX prefix (if you handle 64-bit mode), reads the opcode (1-3 bytes), reads the MOD/RM byte if there is one (use a table lookup), reads the SIB byte if there is one (table), reads offset if there is one (table), reads immediate if there is one (table).
It's probably easiest if you use an "expanded/normalized" opcode internally so the 1-3 opcode bytes + the 3 extra bits from some MOD/RM bytes + prefix info (for certain SIMD instructions) map to a single 16-bit opcode (likely around a couple hundred to a thousand opcodes in total).
You have a table that maps those to mnemonics + operand info. You have some tables that map 0-7 (or 0-15) to AL/AH/... and AX/BX/CX/... and CS/DS/ES/... and various system registers.
Not that much code all in all. Several tables. You can squeeze them and bit pack them to your heart's content if you want.
Once you have that, a simple assembler isn't so hard.
I've been thinking about doing a long term hobby project of creating a personal computer - one where I create the entire software stack myself and can know the provenance of every bit that goes into the system (though I can't do the same with the hardware, unless I can get enough performance out of an FPGA to run everything on it).
Until now it seemed unrealistic but you proved it's possible! And looks like you're even implementing a hobby C compiler: https://github.com/joexbayer/C-Compiler
The C compiler is built for the OS, and works inside it. The project became just so big that I wanted to take it out of the OS repo. Especially because it works on Linux too.
Current it has lots of limitations such as only int and char, no switch statements etc. The biggest change from default C, is allowing functions in structs and implicit passing a struct to a function if it’s a struct function. You can read more here:
Did a little testing myself.
Tried it on some bare metal, looks pretty good.
There were two phases, wouldn't fit in a single comment.
Wrote RetrOS-32.img to a blank SSD and it boots in CSM/Legacy mode.
That's a nice milestone right there :)
This was on a tiny Lenovo desktop about as old as an x240. These are low-power, run on the same Thinkpad A/C adapter, so there are some similarities to a laptop.
But the cursor wasn't very responsive and wouldn't come down below the very top rows either so could not sign in. Using mainstream wired mouse & kybd. (The PC has 8GB memory, in the login box it showed the nominal 15mb, but extended memory was a negative number, about 8GB itself.)
Then I remembered things like the x240 often have a touchpad that is recognized as a PS/2 mouse, not USB. Moved the SSD to an old tower that has a PS/2 mouse and kybd and then could sign on. Cursor was still quite a bit sluggish, but it worked. This one has 12GB memory and the reported extended was about 464mb so it looked realistic in the login box.
It boots! It works nominally! It's some kind of Golden IMG already, so congratulations!
Back at the Lenovo I do boot to DOS from time to time, and I enable Legacy USB in the BIOS so I can use the USB mouse for DOS which had no concept of USB. After booting DOS, if I want to use the mouse I still have to load DOS mouse drivers, just like the 1980's but this firmware setting fools the old pre-USB DOS mouse driver so it will handle the USB mouse & kybd that people are using now. This setting has been there for ages since the PS/2 sockets on PC's started becoming scarce. Checked the BIOS, and Legacy USB was enabled the whole time.
However, on this Lenovo there was an additional setting, for "USB Virtual KBC Support". I don't know if I ever saw this along with Legacy USB settings in one BIOS, up to now would have assumed they were both the same thing but using different terminology. This setting was not only disabled by default, the text says this one auto-disables once an XHCI driver has been loaded. Then definitely found it to auto-disable after booting Windows.
Enabled Virtual KBC, booted to RetrOS and the USB mouse could log in. More sluggish cursor than on the PS/2 machine though.
Plus there were some untyped characters that would sometimes appear when logging in, or admin would duplicate itself so username said adminadmin after you typed in the password below.
Still has a huge negative number in the login box for extended mem, that was not a show-stopper, mem appears normal when looking at the system stats once logged in.
I would now imagine this Virtual KBC setting emulates how mouse & kybd are presented to a VM?
Interesting finding that DOS does not require this setting for a USB mouse to work on bare metal, but RetrOS does.
Have not put it through its paces thoroughly, focusing on the bare metal aspect.
Key sectors containing things like the Partition Table, or a bootsector (Boot Record) are also supposed to be ones that do not change whether or not there is an OS (or any files) in the filesystem or not.
Break that down a little bit, a partition table should not change whether or not the partition(s) are formatted or completely blank. Not supposed to matter whether you are going to use the partition for DOS, Windows, Linux or whatever.
Then a bootsector doesn't even exist until its partition is formatted to begin with, then it reflects the "geometry" that the formatting process seemed to think was best for that filesystem based on the information it had available.
Neither of these should change after that, and be completely independent of whether there are any files in the filesystem.
It's probably best to build on as firm a foundation completely consistent with regular old PC's and it really looks like this is going pretty well using FAT16.
Once you have copied the IMG to a SSD, you could look at either IMG or hardware in DMDE and see the same thing. You do have to kind of figure I checked the IMG for potential bootability before I did anything on bare metal ;)
It is intriguing. There were other anticipated possibilities for an IMG to be bootable, like floppy mode or "large floppy" mode, which start with only a bootsector located at sector 0, no partition table is needed since there is never more than one partition.
Sector 0 right now is reminiscent of a FAT16 bootsector, looking at it in "FAT/FAT32/NTFS Boot Record" Mode, it displays real well. With FAT, you will see there are two different entries for Total Sectors.
By the time FAT16 came around, the upper Total had already been deprecated. You are using it for a full total of 65535 sectors which works so far but it should be in the lower Total Sectors entry for this template on FAT16. The upper entry should be 0 instead.
Then interestingly, looking at sector 0 in Partition Table mode, you do have a (un-needed at this time) partition table, in the usual position within the final bytes of sector 0. The 65535 total sectors are independently designated there (same total as above in the bootsector code that takes up the bulk of the bytes in sector 0 above) but there is no CHS geometry in place, and DMDE flags a "Relative Sector" of 0 as incorrect because it's expected to always be something other than 0. There's a number of structural features that can cause DMDE to flag them other than this too.
With sector 0 in DMDE you hit Editor > Partiton Table and all you're seeing interpreted is the last dozens of bytes of sector 0, it's only meaningful if there actually is a partition table in sector 0. There would be a maximum of 4 primary partitions in place. You can change this part all you want without having an effect on the code existing in the earlier bytes of sector 0 (such as an MBR ordinarily). But if you've got partitions the partition table better be sensible or it could fail to point directly to where the target code is.
No real partitions at this point so you can just go into Edit Mode and zero out the top primary partition line:
80 0E 0 0 0 0 0 0 0 65535
You have to hit File > Apply Changes. Then the partiton table is completely gone. And it will still boot just the same.
Random finding; there's a gfx_destory_window in sector 284 that looks like a typo.
Nothing in the OS should ever be hardcoded to rely on specific sectors, instead everything needs to be relative to the "Hidden Sectors" in the FAT which determine the offset which is the actual bootsector location for a primary partition.
The bootloader needs to be independent, and the OS itself needs to function regardless of what sector its partition starts on.
Start with a zeroed SSD, open it in DMDE, hit Editor > Partition Table and there is nothing but zeros as expected. Type in this, analogously to the above:
80 0E 0 32 33 4 52 48 2048 65536
Also manually put the required AA55 signature at the final 2 bytes. Apply Changes. You have just partitioned it a very mainstream way.
The Relative Sector value shown is supposed to be incorporated into the bootsector at that position as the Hidden Sector offset, which it uses to find its way into the filesystem.
Therefore ideally after you copy the IMG to the SSD starting at sector 2048 (rather than starting at the now-proven, wonderful sector 0), it would be like any other OS on any other partition, with your functional bootsector right there on 2048 where the partition table expects it to be, the only thing to change is your original "Hidden Sectors" of 0 so that is what to edit.
In DMDE Partition Table view, double click on the 80 0E...2048... line and it will take you to sector 2048 as expected, plus autointerpret it as a bootsector, displaying the correct template for FAT bootsectors.
Edit > Edit Mode and where it says Hidden Sectors 0, carefully change that to 2048. Drive Number is supposed to be 80 most of the time too. Apply Changes. The partition should be self-contained now. All you should need to do is boot to it.
You will still need a Master Boot Record (MBR) within the first 440 bytes of sector 0 as usual, which can interpret the partition table and jump to the designated Active partition whose bootsector at 2048 will take over from there. Regardless of whether the OS was originally "installed" to a volume starting at 2048, it needs to be able to work after recovery to a different sized volume which starts at something different like sector 512000 for instance some other time too.
I copy a current MBR from a working bootable MBR-layout USB, bytes 0-439 to the SSD.
And we're off to the races on a real partition.
The success was however, limited. After choosing a resolution, no more display :(
There is mostly no code yet in the 65535-sector 32mb volume, only sectors 0 - 3097 in this IMG are nonzero so far. This is normal at this scale.
Getting it going does seem to depend on specific code always existing somewhere in sectors 1 - 3097 rather than the OS relying on the Hidden Sectors offset within its own bootsector to find that same code located further beyond 2048. This is easy to have happen since hidden sectors has always been zero. VMs tend to abstract this since they are not on real partitions, and they don't always want to make it easy to move to bare metal anyway.
However there is really nothing wrong if the specific bootloader for this OS is permanently stored outside the filesystem at this time and/or at a hard-coded address for the time being, Grub did that sometimes so it would be outside the filesystem and not touched by DOS or Windows normally. I wanted to see if the OS itself can be easily relocated as an IMG to any partition within reach of FAT16. Formatted or not the IMG will overwrite what was there so you get your familiar format parameters and the only thing you're supposed to need to edit is the hidden sectors to match where the partition actually starts when it is different than original. But that didn't quite do it.
So next, additionally copied IMG sectors 1 - 2000 to the SSD starting at sector 1 of the SSD. (Didn't want to overwrite sector 0 with its new partiton table & MBR, nor the other good code now starting at 2048, which didn't leave enough room for all 3097 sectors of data in the IMG to fit in that space any more.) Incidentally, 3097 sectors is only about 1.5+ mb in the whole OS at this point, if it could be trimmed to 2880 sectors it would fit on a regular 1.44mb floppy. Partitons-R-not-us at all for that. Well, you've pretty much accomlished that and more on USB so it really is something to be proud of.
Sectors 1-2000 are completely outside the filesystem as I have it partitioned now.
Now it works, regular Windows MBR on sector 0 finds the Active partition like normal, jumps to 2048 as indicated by the partition table as expected, where it first encounters your bootsector/loader? code and proceeds nominally from there. No more failure of display. Any dependency on code being located in the original dev sectors 1-2000 is fulfilled now because a copy of it is there too, regardless of it being outside the current filesystem which now starts at 2048.
Appears to work with only sectors 1-1000 in place too. I would maybe leave the bootloader in that position and make sure the OS can do its thing starting at 2048 (everything relative to the hidden sectors value within your bootsector, at the time) without relying on any sectors below 2048 after the kernel has loaded.
At this point I should be able to reformat my FAT16 SSD partition using MS-DOS, which will not only clear off the RetrOS, but also rejigger the parameters seen on the template to be completely conventional, I would expect quite a bit different since yours are not very mainstream values. No problem if that's what your OS likes right now. Then I could easily copy io.sys, msdos.sys, and command.com (in that order can be best) to the freshly formatted partition and DOS would boot fine in its familiar landscape, never touching your code in sectors 1-1000 since that's not in the filesystem and not routinely accessed by DOS. When I was done with DOS, then I could copy your IMG back starting at 2048 again, and it would boot right up, without any experimentation, since the code it occasionally depends on is still there somewhere between 1-1000.
So you're ready to switch over to bare metal like this and tie up the loose ends in the real world!
In the long run you want your own simple bootloader to be a file on your filesystem that your bootsector simply seeks after the standard generic stuff jumps to your bootsector, and never depend on anything in earlier sectors than your bootsector after that.
Whatever OS you put on a partition, fundamentally it should be able to exist as just a bunch of stored files unless you boot to it.
Another milestone will be when you can just copy the files of your OS to a regular FAT16 volume that was formatted using MS-DOS itself. Do not skip this step. Even 20-year old Windows XP FAT16 is no substitute, and Linux is also non-ideal. Plus different sized partitions than 32mb will need to be taken in stride. The only thing that needs to be special is the bootsector itself, mainly the part that seeks your OS bootloader file specifically.
Use the mainstream old generic stuff to lead to your bootsector, use DMDE to craft a custom bootsector easily for any partition when needed without having to write your own Format routine, and concentrate on the OS functioning well within its own volume anywhere like the best of them!
I'd love a non-linux minimal single-user OS that boots in <1 second
What about Haiku OS? It’s supposed to be for that exact use case…
Rock on!
Kudos for the project.
Brb, hacking your thinkpad
An example of a radically different way to think is TempleOS.
Forget practicality.
Forget "go to market" strategy.
Forget target fit.
Just build and learn.
> I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu)
Just kidding. This is really impressive.
There are two types of people that I think of as gods: drummers (but not the bad ones) and OS developers (even the bad ones).
Congrats!
Are you planning on cardbus/pcmcia support and wifi?