I don’t think there’s any magic involved there. You could get that “effect” by simply dialing back stroke opacity, and avoiding the blur or smear tools. This naturally forces you to layer up the paint. Create transitions using intermediate shades and values, rather than trying to blend them. Apart from that, shape dynamics, like size jitter, will give some randomness to strokes.
depending on what version you’re using, the Brush settings have a “wet edges” effect that might do some of what you want. other than that, you could try some of the filters, eg Watercolour, or fiddling w/ the brush settings for opacity/jitter/hue jitter etc etc.
You may also consider using the free and open application Krita for this sort of thing. While Photoshop certainly is very good at what it does, Krita is better suited towards creating this kind of artistic effect.
