How can i make a realistic simple wall paint material not look like a toy?

If the walls are supposed to look painted, then you could add a very small noise bump to them (to simulate the slight stippling effect you get from a paint roller).

You could also try using the microroughness node group, to add some slight reflection variance to the wall.

I used this effect on a scene I downloaded from blendswap in the link below. The effect is subtle on the walls, but IMO definitely adds some realism.