OpenGL simple antialiased polygon grid shader - opengl-es

Question

How to make a test grid pattern with antialiased lines in a fragment shader?
I remember I found this challenging, so I'll post the answer here for my future self and for anyone who wants the same effect.
This shader is meant to be rendered "above" the already textured plane in a separate render call. The reason I'm doing that - is because in my program I am generating the texture of the surface through several render calls, slowly building it up layer by layer. And then I wanted to make a simple black grid over it, so I make the last render call to do this.
That's why the base color here is (0,0,0,0), basically a nothing. Then I can use GL mixing patterns to overlay the result of this shader over whatever my texture is.
Note that you needn't do that separately. You can just as easily modify this code to display a certain color (like smooth grey) or even a texture of your choice. Simply pass the texture to the shader and modify the last line accordingly.
Also note that I use constants that I set up during shader compillation. Basically, I just load the shader string, but before passing it to a shader compiler - I search and replace the __CONSTANT_SOMETHING with an actual value I want. Don't forget that that's all text, so you need to replace it with text, for example:
//java code
shaderCode = shaderCode.replaceFirst("__CONSTANT_SQUARE_SIZE", String.valueOf(GlobalSettings.PLANE_SQUARE_SIZE));

Answer 1

If I could share with you the code I use for anti-aliased grids, it might help the complexity. All I've done is use the texture coordinates to paint a grid on a plane. I used GLSL's genType fract(genType x) to repeat texture space. Then I used the absolute value function to essentially calculate each pixel's distance to the grid line. The rest of the operations are to interpret that as a color.
You can play with this code directly on Shadertoy.com by pasting it into a new shader.
If you want to use it in your code, the only lines you need are the part starting at the gridSize variable and ending with the grid variable.
iResolution.y is the screen height, uv is the texture coordinate of your plane.
gridSize and width should probably be supplied with a uniform variable.
void mainImage(out vec4 fragColor, in vec2 fragCoord) {
// aspect correct pixel coordinates (for shadertoy only)
vec2 uv = fragCoord / iResolution.xy * vec2(iResolution.x / iResolution.y, 1.0);
// get some diagonal lines going (for shadertoy only)
uv.yx += uv.xy * 0.1;
// for every unit of texture space, I want 10 grid lines
float gridSize = 10.0;
// width of a line on the screen plus a little bit for AA
float width = (gridSize * 1.2) / iResolution.y;
// chop up into grid
uv = fract(uv * gridSize);
// abs version
float grid = max(
1.0 - abs((uv.y - 0.5) / width),
1.0 - abs((uv.x - 0.5) / width)
);
// Output to screen (for shadertoy only)
fragColor = vec4(grid, grid, grid, 1.0);
}
Happy shading!

Answer 2

Here're my shaders:
Vertex:
#version 300 es
precision highp float;
precision highp int;
layout (location=0) in vec3 position;
uniform mat4 projectionMatrix;
uniform mat4 modelViewMatrix;
uniform vec2 coordShift;
uniform mat4 modelMatrix;
out highp vec3 vertexPosition;
const float PLANE_SCALE = __CONSTANT_PLANE_SCALE; //assigned during shader compillation
void main()
{
// generate position data for the fragment shader
// does not take view matrix or projection matrix into account
// TODO: +3.0 part is contingent on the actual mesh. It is supposed to be it's lowest possible coordinate.
// TODO: the mesh here is 6x6 with -3..3 coords. I normalize it to 0..6 for correct fragment shader calculations
vertexPosition = vec3((position.x+3.0)*PLANE_SCALE+coordShift.x, position.y, (position.z+3.0)*PLANE_SCALE+coordShift.y);
// position data for the OpenGL vertex drawing
gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}
Note that I calculate VertexPosition here and pass it to the fragment shader. This is so that my grid "moves" when the object moves. The thing is, in my app I have the ground basically stuck to the main entity. The entity (call it character or whatever) doesn't move across the plane or changes its position relative to the plane. But to create the illusion of movement - I calculate the coordinate shift (relative to the square size) and use that to calculate vertex position.
It's a bit complicated, but I thought I would include that. Basically, if the square size is set to 5.0 (i.e. we have a 5x5 meter square grid), then coordShift of (0,0) would mean that the character stands in the lower left corner of the square; coordShift of (2.5,2.5) would be the middle, and (5,5) would be top right. After going past 5, the shifting loops back to 0. Go below 0 - it loops to 5.
So basically the grid ever "moves" within one square, but because it is uniform - the illusion is that you're walking on an infinite grid surface instead.
Also note that you can make the same thing work with multi-layered grids, for example where every 10th line is thicker. All you really need to do is make sure your coordShift represents the largest distance your grid pattern shifts.
Just in case someone wonders why I made it loop - it's for precision sake. Sure, you could just pass raw character's coordinate to the shader, and it'll work fine around (0,0), but as you get 10000 units away - you will notice some serious precision glitches, like your lines getting distorted or even "fuzzy" like they're made out of brushes.
Here's the fragment shader:
#version 300 es
precision highp float;
in highp vec3 vertexPosition;
out mediump vec4 fragColor;
const float squareSize = __CONSTANT_SQUARE_SIZE;
const vec3 color_l1 = __CONSTANT_COLOR_L1;
void main()
{
// calculate deriviatives
// (must be done at the start before conditionals)
float dXy = abs(dFdx(vertexPosition.z)) / 2.0;
float dYy = abs(dFdy(vertexPosition.z)) / 2.0;
float dXx = abs(dFdx(vertexPosition.x)) / 2.0;
float dYx = abs(dFdy(vertexPosition.x)) / 2.0;
// find and fill horizontal lines
int roundPos = int(vertexPosition.z / squareSize);
float remainder = vertexPosition.z - float(roundPos)*squareSize;
float width = max(dYy, dXy) * 2.0;
if (remainder <= width)
{
float diff = (width - remainder) / width;
fragColor = vec4(color_l1, diff);
return;
}
if (remainder >= (squareSize - width))
{
float diff = (remainder - squareSize + width) / width;
fragColor = vec4(color_l1, diff);
return;
}
// find and fill vertical lines
roundPos = int(vertexPosition.x / squareSize);
remainder = vertexPosition.x - float(roundPos)*squareSize;
width = max(dYx, dXx) * 2.0;
if (remainder <= width)
{
float diff = (width - remainder) / width;
fragColor = vec4(color_l1, diff);
return;
}
if (remainder >= (squareSize - width))
{
float diff = (remainder - squareSize + width) / width;
fragColor = vec4(color_l1, diff);
return;
}
// fill base color
fragColor = vec4(0,0,0, 0);
return;
}
It is currently built for a 1-pixel thick lines only, but you can control thickness by controlling the "width"
Here, the first important part is dfdx / dfdy functions. These are GLSL functions, and I'll simply say that they let you determine how much space in WORLD coordinates your fragment takes on the screen, based on the Z-distance of that spot on your plane.
Well, that was a mouthful. I'm sure you can figure it out if you read docs for them though.
Then I take the maximum of those outputs as width. Basically, depending on the way your camera is looking you want to "stretch" the width of your line a bit.
remainder - is basically how far this fragment is from the line that we want to draw in world coordinates. If it's too far - we don't need to fill it.
If you simply take the max here, you will get a non-antialiased line 1 pizel wide. It'll basically look like a perfect 1-pixel line shape from MS paint.
But increasing width, you make those straight segments stretch further and overlap.
You can see that I compare remainder with line width here. The greater the width - the bigger the remainder can be to "hit" it. I have to compare this from both sides, because otherwise you're only looking at pixels that are close to the line from the negative coord side, and discount the positive, which could still be hitting it.
Now, for the simple antialiasing effect, we need to make those overlapping segments "fade out" as they near their ends. For this purpose, I calculate the fraction to see how deeply the remainder is inside the line. When the fraction equals 1, this means that our line that we want to draw basically goes straight through the middle of the fragment that we're currently drawing. As the fraction approaches 0, it means the fragment is farther and farther away from the line, and should thus be made more and more transparent.
Finally, we do this from both sides for horizontal and vertical lines separately. We have to do them separate because dFdX / dFdY needs to be different for vertical and horizontal lines, so we can't do them in one formula.
And at last, if we didn't hit any of the lines close enough - we fill the fragment with transparent color.
I'm not sure if that's THE best code for the task - but it works. If you have suggestions let me know!
p.s. shaders are written for Opengl-ES, but they should work for OpenGL too.