Well, to begin with this, I am just going to begin by saying that I am not responsible for any damage the driver cause ( BSOD, loss Of Data and so on ). Get a virtual machine so that your main PC does not get affected or damaged. I also do not advise or encourage the use of this information for an ill use or just damaging or infecting other’s PC or properties that do not belong to you.
What We Will Be Using
So, I wouldn’t say that there are many ways of protecting a user-mode program. The most common methods are just setting up callbacks and making a bootkit, something that loads during or directly after boot. This makes it so that your callbacks are registered before everything, meaning you can monitor everything right after boot. Other then that, there are other ways like signature scanning also known as memory scanning. This just allows you to defend yourself against programs that could be commonly used, things such as debuggers, cheat engine or pretty much anything you like. The downside to this is that you must have a copy of the program you like to defend yourself with. Things like Hooking would also be advised if you like to monitor some APIs calls in your programs you would most likely want to hook/detour some APIs. Things like checking for Test Mode also helps, just make sure that the user can’t just load up a driver and just screw up your whole protection. Things like replacing your callbacks with dummy callbacks or just emulating or making a dummy driver pretending to be your real one. Other things such as a heartbeat system should also be used.
So, first things first, callbacks. Let us just review some useful callbacks you could use.
These are the commonly used callbacks, yes there are many other callbacks such as PoRegisterDeviceNotify and many more. But for now, let’s just stick to these few. Might do a part 2 if this isn’t enough.
Let us start with the most powerful and most used callback, ObRegisterCallbacks. If you don’t really know what this exactly does, there are two proper use of this callback. You can either monitor what handles are made, thread and processes, or just “strip” their permissions. You can also see which process the handle belong to. You might ask why is this useful? well, it is as simple as giving an access violation when trying to use OpenProcess() to create a handle to your chosen program. How? as simple as just getting rid of their permissions. There are two things you can do, before and after the handle has been created. Meaning, you are able to just strip the handle’s permissions such as Read and Write before the handle is even created. If you still don’t realize how powerful this is, well think of it as this. If you like to do anything or just interact with the program ( Read it’s memory or Write it’s memory or even inject ), you need a handle. So stripping the handle itself or even some permissions can really affect every program trying to do something with your program. This is the most basic and easiest method of isolating or protecting your program. The sad thing is that this isn’t bulletproof, things like dummy callbacks can be placed or people can even start their own callback in the same altitude and just cause a collision when you try to register the callback. An easy fix is just checking if the callback got registered properly. ( Checking what it returns ).
Lets move on to PsSetCreateProcessNotifyRoutine. Well, as the name suggests, its a callback for processes created or destroyed. This can go amazingly with your signature scan or memory scan as you would be notified every time a process is opened/ran. You can send this information to your user-mode program and just scan it for any bad signatures. Other then that, you can monitor if your own user-mode program has been closed by something. If you are using ObRegisterCallbacks and this happens, there is something really wrong here, it either crashed or someone is using ZwTerminateProcess. If this happens just try to reopen your program. If you are making an antivirus, just make sure to not allow anyone to close your program by protecting your process using RtlSetProcessIsCritical. This callback is quite easy to understand and to use, so it shouldn’t really give too many problems, just make sure to unregister it on unload. ( If you like to stay on the computer, just don’t have a unload function )
The next callback would be PsSetLoadImageNotifyRoutine. This is as easy to understand as the previous one. It just notifies you when there is an image being loaded. What’s an image? Drivers and DLLs or any other modules. This could go alongside your signature scanner just like PsSetCreateProcessNotifyRoutine as this basically notifies you whenever something is injected or is a driver is loaded. Remember that there are many ways of bypassing this, so don’t be surprised if something manages to sneak in. I would strongly not advise you to rely on it, it can be easily bypassed and only the most obvious methods can be detected.
Alright, moving on to CmRegisterCallbackEx. Well, to be honest, I have never really used this callback. Would say that this is quite useful unless you want to monitor your registries.
Hook / Detours
Well, hooking or also known as detouring would help you monitor or just redirect, filter or not allow an API of your choice. You might ask, what’s the point of this? Well, imagine if someone tries to use ReadProcessMemory using your process’s handle. Now, imagine being able to check if the handle belongs to your protected process and you being able to cancel or just return a access denied. Now you see why this is useful? I would advise you to at least detour some APIs and monitor it. Things like CreateThread is very useful and CreateRemoteThread. With this, you can monitor whenever your program is trying to create a thread and check their starting address. If it is outside the module, it means its an injected DLL. Either kill it or just suspend it. Hijacking a thread is different, and not so easy to find one, just make sure to strip handles for processes and threads using the callback shown on top. Another thing you can do with this is injecting into every process in the PC ( Expect system programs. Easiest way to check is if their PID is below like 400, remember it’s not constant ), then hooking onto APIs such as ReadProcessMemory and so on. If you don’t like injecting into each process, try a global hook which works system-wide, but there are obviously some cons to this. If you are on a 32-bit processor, you can do things like hooking onto the SSDT table and just control the whole PC. Note that 64 bit PCs has PatchGuard, the only way around is by Mini Filter Drivers if I am not wrong.
I would say that this is really an opinion, just due to the fact of what you are trying to do. If you are developing an Anti Virus, then not running on Test Mode should not be done, however, if you are an Anti Cheat, you should check for Test Mode and not allow the game to run unless Test Mode is turned off. Doing this is quite tricky I suppose. There are things like checking for CodeIntegrityOptions in SYSTEM_CODEINTEGRITY_INFORMATION or just having an unsigned driver, which you can load and check to see if it loads properly or not. Other then that, you need to be quite creative. Test Mode isn’t the easiest thing to check even for me and some of the major companies like Battleye and such.
Signature / Memory Scans
Well, this is the most basic function to really use. There are really a lot of things you can do with this and this is quite useful to many. However, there are cons like having a team of reverse engineers to try to break the program to find good signatures which really is not easy. Why? Usually, smart cheat/virus programmers usually encrypt, pack or just use cryptors on their program to get away with signature scanning and this will also make reverse engineering extremely painful. Therefore, this should not be really the main way of scanning for viruses. Obviously, this does get rid of some old or unchanged viruses, and if you find or scan for the right signatures, you might even detect every update that comes after for the cheat/virus. But like I said, this is easy said than done and extremely easy to bypass. For example, a totally new virus which maybe does the same functions ( delete system32 ) can be written in so many ways that there is really no way of finding or getting a signature which detects every program which tries to delete system32. Its been like that forever and would stay that way.
Problems / Things To Think About
I would say that the best Anti Virus or the best Anti Cheat, that people often use, is always the most aggressive. Bringing up the problem of, “false detection” or “false ban”. For me, I would say the best Anti Cheat / Anti Virus would be a good balance of defensive and aggressive. Imagine you getting banned for just having Cheat Engine opened in the background, or having your Anti Virus scan all unsigned programs ( programs without digital signature ) and just blocking them for not trusting them. This would defiantly not be fair, but this would block out so many Viruses. Therefore would you consider this the best or the worst? See my point? Anyways, if you would like to create or make the best Anti Virus or Anti Cheat, you need to find the best balance. Things like not ban for opening Cheat Engine but banning if Cheat Engine has opened up a handle to your program. But this would bring up some weaknesses, like what if I just somehow bypass your checks for going through the handles? So, every non-aggressive move would have a side effect and every aggressive would leave some with a false ban.
Well, hopefully, you learn something. Let us recall what we talked about, Callbacks, hooking / detouring and test mode. Would say that if you really wanted to make a program which either protects your PC or protects your programs, these 3 would be somewhat enough. Obviously, other things like packing your games/programs would make it much harder to reverse engineer it and other small steps like setting up a HeartBeat system which checks if the driver is loaded properly and the other way round. Don’t overlook something, and you should do fine. Just try to change view somethings and try to think of the ways your ways can be exploited. This is important if you really want to make a bulletproof protection. Disclaimer, by bulletproof i meant, hard enough to not make the attacker continue or just give up. Remember that everything can be bypassed, it is just about how hard you make it or just how many holes you cover.