Hill Climb Racing Automation

So, While I was little maybe in my early teens I used to play this game. At that time, it was in mainstream at least in India, apart from the Subway Surfers. I was more into Hill Climb Racing because i was always into car racing but partly because I was better at it than i was in Subway Surfers.

Background:

This game offers variety of tracks to play on. You can select any car that you want which is unlocked to play with. For the game controls you are only provided with Gas and Brake. Now this game does follow some rules of physics, but it overly exaggerates the effect of Gas pedal and Brake pedal. In a normal scenario, applying brake makes front of the car lighter, and applying brake makes back of your car lighter.

So this games' physics exaggerates this effect and gives rotational velocity while using any of the given pedal. This rotational velocity actually comes in handy in some of the tracks.

Problem Statement:

I'll try to make Bot to play this game, and earn money (My extrinsic motivator)

Gameplay:

This is my gameplay, Notice how application of Gas pedal makes the front of the car lighter and application of brake makes back of the car lighter and it starts to flip

So after moving the mss and various other window capture, I decided to go window api screenshot

it was giving me the flexiblity to capture any window, not the whole screen. So it was good and the FPS was also respectable, and with just few optimizations the FPS jumped to pretty good.

Now i'm struck at template matching for object detection. Without nay preprocssing HSRV and all it is capturing the car,

but not keeping the car withing the bounding box and many times i would say half of the time the bounding box is 

pointing ti the sky on thee right part of the screen rather on the car which is on the left part of the screen.

So template matching is working but not as nicely as i would have hoped for. So let's try some optimization

So i tried fo rtemplate matching

result = cv.matchTemplate(image, template, cv.TM_SQDIFF)

result = cv.matchTemplate(image, template, cv.TM_SQDIFF_NORMED)

result = cv.matchTemplate(image, template, cv.TM_CCORR)

result = cv.matchTemplate(image, template, cv.TM_CCORR_NORMED)

result = cv.matchTemplate(image, template, cv.TM_CCOEFF)

result = cv.matchTemplate(image, template, cv.TM_CCOEFF_NORMED)

For SQDIFF i even tried with already grayscale image, and even without background, filling the background with white , to create a drastic difference. 

Also i tried with others as well, to provide an image without background but it's not working.

Best is CCOEFF it matches the car, but jitters very much, idk why so for now it's threshold is 0.65

But let's see tomorrow we can change HSRV and try to do template matching with that.

After trying out HSV and incorporating it into the screen capture with object detection using template matching, I encountered a a roadblock. Now I tried to template match for object detection this created a very huge problem for me as, when the car moves in the game, the camera rolls backward according to speed. Now this is very problematic because my template has the car in certain angle i.e. 0 degree from the ground meaning totally flat to the ground. The game is over when car hits the floor at 180 snapping the neck of the driver and finsihing the game. So with template matching, this would not be able to observe since there is actually very large bonus points are given if you rotate in mid air, meaning, flip 360 in mid air in any direction. So this risk you take to flip gives out the reward in monetary terms so to collect the data, i have to actually at all times collect the data of rotation, but since this is not possible with template matching. I'll drop this pipeline now. 

Now there are two options to explore

1. YOLO

2. Feature Matching using ORB

Now i tried with feature matching with orb as well, but the problem is That Feature mapping is not performing as I would have wanted it to the reason that I saw was because when I was featured matching it the icon for gasoline actually very closely matches with the color of Offer that's Why it was getting confused moreover I was constantly observing that whenever anything is printed on the screen while playing the game suppose air time or neck flip bonus or any kind of bonus or any kind of flashing text is generated it is actually outllined by black color And the same color was of the car's front so it was again getting very confused with thatI think feature matching would have Formed very well if the game was little less flashy there are coins there is the meter for gasoline there is the icon to get the gasoline and fill up your tank also there is some gems as well also whenever the fuel starts to get low flashing right signals appear on the screen signaling that the fuel is low that color it is little less red but it again could be a problem also after so much of Of playing out with different configurations most of the time feature matching was happening from the hat of the driver....closely matches with other icons such as Highest record and full screen icon so again if I was increasing the number of feature matching it was getting confused between all those things if I was reducing it then I was not properly satisfied with the feature matching so feature matching using org didn't turn out as well as I would have liked it too also to feature match I tried to do it in grayscale in rgb and bgr is different configuration in the colors as well, but upto no avail.

This is the demonstration of Feature Matching with ORB.

So tomorrow we start on YOLO.

Tomorrow came a little late, but....

Second Method is using YOLO. A plain Vanilla YOLO Model was able to identify the car on many instances. Not on each and every frames, but on every 5th - 10th frame it was able to identify car as "car" or sometimes as "truck". So that is very good. But the main benefit of yolo was that it was able to identify gasoline icon very well. 

If i had decided to move forward with feature or template matching I had to make so many other templates for coins (different type of coins), gasoline and car, Instances where car is flipped upside down which marks "GAME OVER" would be difficult to match and differentiate with normal car. And it would be little difficult to get rotational angle of the car. So we can use feature matching or template matching for other icons, which wouldn't be recognized well by YOLO. But for now YOLO seems to be a good promise. And it would be very good, if it still manages to identify car even if camera pans out or zoom in.

So let's see.