Removing bytes from File in (C) without creating new File

I can suggest two options:

• Ring buffer
  Use a memory mapped file as your logging medium, and use it as a ring buffer. You will need to manually manage where the last written byte is, and wrap around your ring appropriately as you step over the end of the ring. This way, your logging file stays a constant size, but you can't tail it like a regular file. Instead, you will need to write a special program that knows how to walk the ring buffer when you want to display the log.
• Multiple number of small log files
  Use some number of smaller log files that you log to, and remove the oldest file as the collection of files grow beyond the size of logs you want to maintain. If the most recent log file is always named the same, you can use the standard tail -F utility to follow the log contents perpetually. To avoid issues of multiple programs manipulating the same file, your logging code can send logs as messages to a single logging daemon.

So... you want to change the file, but you cannot. The reason you cannot is that other programs are using the file. In general terms, you appear to need to:

1. stop all the other programs messing with the file while you change it -- to chop now unwanted stuff off the front;

2. inform the other programs that you have changed it -- so they can re-establish their file-pointers.

I guess there must be a mechanism to allow the other programs to change the file without tripping over each other... so perhaps you can extend that ? [If all the other programs are children of the main program, then if the children all O_APPEND, you have a fighting chance of doing this, perhaps with the help of a file-lock or a semaphore (which may already exist ?). But if the programs are this intimately related, then @jxh has other, probably better, suggestions.]

But, if you cannot change the other programs in any way, you appear to be stuck, except...

...perhaps you could try 'sparse' files ? On (recent-ish) Linux (at least) you can fallocate() with FALLOC_FL_PUNCH_HOLE, to remove the stuff you don't want without affecting the other programs file-pointers. Of course, sooner or later the other programs may overflow the file-pointer, but that may be a more theoretical than practical issue.