Polygons are drawn as triangles on the screen and that’s how they’re rendered too. Triangles are great for that because they are always planar. It means that no matter how you arrange three vertices in a triangle, the face between them is always flat.
Quadrilaterals and n-gons can be non-planar. If you move a vertex out of alignment in a quad or n-gon, you get a face that is deformed without having an actual edge on the fold. Those are then triangulated for drawing it on the screen and for rendering, and you can end up with triangulated shading when the lights don’t hide it. You’re not in control of the triangulation either - it might not get a new edge where the fold happens.
Bottom row: a triangle, quadrilateral, n-gon
Middle row: a triangle and two non-planar polygons
Top row: same as previous but properly triangulated
The solution is to add more geometry to the shape so that you don’t get non-planar polygons, or even if you do, the angle between polygons after triangulation is so small it doesn’t result to unwanted shading.
Viewport and rendered versions, from left:
- original with weird shading
- same as previous but triangulated
- fixed version that has subdivision surface modifier (in simple mode) added to give more geometry to the initial shape. All cube edges have been marked as sharp (ctrl+E -> mark as sharp) and those are split with edge split modifier to fix the shading on high angle edges.