Suggestions on how to fix holes appearing in 2D image rotation shader?

I have a HLSL shader I am using in unity and the idea is to take a 2D image and a depth map of that image, and rotate them based on an euler input. Right now it's rotating exactly how I want it to, but I am having an issue with holes appearing at certain steps. Currently it iterates through a cube area starting at z = -1, stepping by 0.01f to z = 1 and finding a texel from the original image that best rotates to a point, prioritising closer points to account for occlusion. This means that some pixels are skipped though if there are no pixels close enough. I added a fidelity variable to try and account for this, but it leads to smudging. Does anyone have a solution or a more elegant way for achieving the effect I'm aiming for?

Relevant texture coordinate to sample
float2 releventTexCoord = i.uv;
bool has_seen = false;
float _zStep = 0.01;

Traverse through depth values from z = -1 to z = 1
for (float z = -1.0; z <= 1.0; z += _zStep)
>                 {
Set current position at depth z
currentWorldPos.z = z;

Inverse rotate the current position to determine the corresponding UV
float3 rotatedUVPos = RotateVectorByQuat(currentWorldPos, inverseQuat);

Check if the rotated position lies within the 0-1 UV range
if (rotatedUVPos.x >= 0 && rotatedUVPos.x <= 1 &&
rotatedUVPos.y >= 0 && rotatedUVPos.y <= 1)
>                     {
Sample the depth map for this UV coordinate
float sampledDepth = tex2D(_DepthMap, rotatedUVPos.xy).r * _DepthIntensity;

Rotate this depth value to world space
float3 depthWorldPos = float3(rotatedUVPos.x, rotatedUVPos.y, sampledDepth);
float3 projectedWorldPos = RotateVectorByQuat(depthWorldPos, quat);


Check if this rotated position projects to the current fragment
if (abs(projectedWorldPos.x - i.uv.x) < _fidelity*_zStep &&
abs(projectedWorldPos.y - i.uv.y) < _fidelity*_zStep)
>                         {
If the depth is higher, update the relevant texture coordinate
if (projectedWorldPos.z > maxDepth)
>                             {
has_seen = true;
maxDepth = projectedWorldPos.z;
releventTexCoord = rotatedUVPos.xy;
>                             }
>                         }
>                     }
>                 }

I was hoping I could add a float that measures the closest appropriate texel but right now I am chock full of the cold and cannot figure out how best to do it. My gut is to just do a linear distance and heavily weighting the z distance to account for the occlusion.