It starts with house rules for your game. The game engine doesn't handle a particular situation or maybe it does, but it handles it badly in your opinion and the opinion of your fellow players.
After you have enough house rules, and you've played a ton of game systems and read even more rule books you start to think that you could write your own game engine. You and just about every other serious gamer ever.
I'm not saying don't try, it's an excellent exercise in discovery. Rules are almost always some sort of compromise between realism and fun. Spending some time trying to model your own game engine can help you understand why rules you've used before work the way they do, when you start to see the bigger picture of what the rules are there to do.
I recommend, before you start from a blank slate you take a look at a game engine that has been adapted to a variety of genres such as Cortex Plus or Apocalypse World. Any system really that provides you the tools you need to move it's basic mechanics into another genre or adjust it's rules to fit a narrow scope of narrative style.
Make your own variations using these sorts of game engines first, so you have some experience with making up everything except the core mechanics. Then, once you've got that handled, then try making something from whole cloth. A game engine all your own, with your own methods and madness.
Perhaps you will publish your game engine, and perhaps a world setting, so that others can enjoy your work. If nothing else you have got a reason to run play tests and get people together to role-play, and really that;s what this hobby is supposed to be about right?
So give in to your mad dreams of designing the best game system ever. It's fun and every gamer should do it. At least once.