iOS - Scale and crop CMSampleBufferRef/CVImageBufferRef

Question

I am using AVFoundation and getting the sample buffer from AVCaptureVideoDataOutput, I can write it directly to videoWriter by using:

- (void)writeBufferFrame:(CMSampleBufferRef)sampleBuffer {
    CMTime lastSampleTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);    
    if(self.videoWriter.status != AVAssetWriterStatusWriting)
    {
        [self.videoWriter startWriting];
        [self.videoWriter startSessionAtSourceTime:lastSampleTime];
    }

    [self.videoWriterInput appendSampleBuffer:sampleBuffer];

}

What I want to do now is to crop and scale the image inside the CMSampleBufferRef without converting it into UIImage or CGImageRef because that slows down the performance.

Vukašin Manojlović · Accepted Answer · 2020-12-02 12:44:59Z

36

If you use vimage you can work directly on the buffer data without converting it to any image format.

outImg contains the cropped and scaled image data. The relation between outWidth and cropWidth sets the scaling.

int cropX0, cropY0, cropHeight, cropWidth, outWidth, outHeight;

CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);                   
CVPixelBufferLockBaseAddress(imageBuffer,0);
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
                                    
vImage_Buffer inBuff;                       
inBuff.height = cropHeight;
inBuff.width = cropWidth;
inBuff.rowBytes = bytesPerRow;

int startpos = cropY0*bytesPerRow+4*cropX0;
inBuff.data = baseAddress+startpos;

unsigned char *outImg= (unsigned char*)malloc(4*outWidth*outHeight);
vImage_Buffer outBuff = {outImg, outHeight, outWidth, 4*outWidth};

vImage_Error err = vImageScale_ARGB8888(&inBuff, &outBuff, NULL, 0);
if (err != kvImageNoError) NSLog(@" error %ld", err);

So setting cropX0 = 0 and cropY0 = 0 and cropWidth and cropHeight to the original size means no cropping (using the whole original image). Setting outWidth = cropWidth and outHeight = cropHeight results in no scaling. Note that inBuff.rowBytes should always be the length of the full source buffer, not the cropped length.

edited Dec 2, 2020 at 12:44

Vukašin Manojlović

3,8573 gold badges23 silver badges33 bronze badges

answered Apr 8, 2012 at 13:02

Sten

3,8642 gold badges30 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

21 Comments

Eyal Over a year ago

Hi Sten, I searched the guide for a cropping example but couldn't fine one, can u give an example how to crop the buffer directly?

AlexeyVMP Over a year ago

vImage has no cropping functionality

Nils Ziehn Over a year ago

I know this question/answer is old, but how do I get the outBuff now to a CMSampleBuffer?

Sten Over a year ago

Nils, the data of the cropped file is in outImg. So you can use that to create a pixelbuffer e.g. with CVPixelBufferCreateWithBytes. Then you can use that to create the CMSampleBuffer.

Sten Over a year ago

I have added an image instead of the link to the old PDF-guide.

|

Jeroen · Accepted Answer · 2019-10-09 08:48:10Z

11

Note: I didn't notice that the original question also requested scaling. But anyways, for those who simply needs to crop CMSampleBuffer, here's the solution.

The buffer is simply an array of pixels, so you can actually process the buffer directly without using vImage. Code is written in Swift, but I think it's easy to find the Objective-C equivalent.

First, make sure your CMSampleBuffer is BGRA format. If not, the preset you use is probably YUV, and ruin the bytes per rows that will later be used.

dataOutput = AVCaptureVideoDataOutput()
dataOutput.videoSettings = [
    String(kCVPixelBufferPixelFormatTypeKey): 
    NSNumber(value: kCVPixelFormatType_32BGRA)
]

Then, when you get the sample buffer:

let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!

CVPixelBufferLockBaseAddress(imageBuffer, .readOnly)

let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer)
let cropWidth = 640
let cropHeight = 640
let colorSpace = CGColorSpaceCreateDeviceRGB()

let context = CGContext(data: baseAddress, width: cropWidth, height: cropHeight, bitsPerComponent: 8, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)
// now the cropped image is inside the context. 
// you can convert it back to CVPixelBuffer 
// using CVPixelBufferCreateWithBytes if you want.

CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly)

// create image
let cgImage: CGImage = context!.makeImage()!
let image = UIImage(cgImage: cgImage)

If you want to crop from some specific position, add the following code:

// calculate start position
let bytesPerPixel = 4
let startPoint = [ "x": 10, "y": 10 ]
let startAddress = baseAddress + startPoint["y"]! * bytesPerRow + startPoint["x"]! * bytesPerPixel

and change the baseAddress in CGContext() into startAddress. Make sure not to exceed the origin image width and height.

edited Oct 9, 2019 at 8:48

Jeroen

2,0932 gold badges27 silver badges55 bronze badges

answered Jul 28, 2015 at 17:45

yuji

7257 silver badges22 bronze badges

8 Comments

vkalit Over a year ago

"...without converting it into UIImage or CGImageRef because that slows down the performance."

yuji Over a year ago

I did "crop and scale the image inside the CMSampleBufferRef without converting it into UIImage or CGImageRef ". I just save it as CGImageRef for further usage (eg. show on screen). You can do whatever you want with the cropped context.

Crashalot Over a year ago

Hi 黃昱嘉 what is the best way to contact you? Would like to ask a quick question. Thanks!

user924 Over a year ago

and how to convert CIImage back to CMSampleBuffer fast?

Leonid Silver Over a year ago

Hi, @yuji thank you for your answer! Could you please show how to recreate CVPixelBuffer? there is a lot of params I don't understand when trying to do it. And I would be super grateful if you could advice anything to read about all this magic

|

Cliff · Accepted Answer · 2011-12-13 18:32:27Z

9

You might consider using CoreImage (5.0+).

CIImage *ciImage = [CIImage imageWithCVPixelBuffer:CMSampleBufferGetImageBuffer(sampleBuffer)
                                           options:[NSDictionary dictionaryWithObjectsAndKeys:[NSNull null], kCIImageColorSpace, nil]];
ciImage = [[ciImage imageByApplyingTransform:myScaleTransform] imageByCroppingToRect:myRect];

answered Dec 13, 2011 at 18:32

Cliff

11.5k7 gold badges71 silver badges105 bronze badges

2 Comments

vodkhang Over a year ago

and then, how can I convert it back to CMSampleBuffer, or how can I write it down using self.videoWriterInput?

Tony Million Over a year ago

CIContext has methods to rasterize a CIImage back into a CVPixelBuffer.

Bluewings · Accepted Answer · 2017-06-27 13:18:42Z

Try this on Swift3

func resize(_ destSize: CGSize)-> CVPixelBuffer? {
        guard let imageBuffer = CMSampleBufferGetImageBuffer(self) else { return nil }
        // Lock the image buffer
        CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags(rawValue: 0))
        // Get information about the image
        let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
        let bytesPerRow = CGFloat(CVPixelBufferGetBytesPerRow(imageBuffer))
        let height = CGFloat(CVPixelBufferGetHeight(imageBuffer))
        let width = CGFloat(CVPixelBufferGetWidth(imageBuffer))
        var pixelBuffer: CVPixelBuffer?
        let options = [kCVPixelBufferCGImageCompatibilityKey:true,
                       kCVPixelBufferCGBitmapContextCompatibilityKey:true]
        let topMargin = (height - destSize.height) / CGFloat(2)
        let leftMargin = (width - destSize.width) * CGFloat(2)
        let baseAddressStart = Int(bytesPerRow * topMargin + leftMargin)
        let addressPoint = baseAddress!.assumingMemoryBound(to: UInt8.self)
        let status = CVPixelBufferCreateWithBytes(kCFAllocatorDefault, Int(destSize.width), Int(destSize.height), kCVPixelFormatType_32BGRA, &addressPoint[baseAddressStart], Int(bytesPerRow), nil, nil, options as CFDictionary, &pixelBuffer)
        if (status != 0) {
            print(status)
            return nil;
        }
        CVPixelBufferUnlockBaseAddress(imageBuffer,CVPixelBufferLockFlags(rawValue: 0))
        return pixelBuffer;
    }

Community · Accepted Answer · 2017-05-23 11:47:01Z

1

For scaling you can have AVFoundation do this for you. See my recent post here. Setting the value for AVVideoWidth/AVVideoHeight key will scale the images if they are not the same dimensions. Take a look at the properties here.As for cropping I am not sure if you can have AVFoundation do this for you. You may have to resort to using OpenGL or CoreImage. There are a couple of good links in the top post for this SO question.

edited May 23, 2017 at 11:47

CommunityBot

11 silver badge

answered Dec 16, 2011 at 3:53

Steve McFarlin

3,5961 gold badge27 silver badges24 bronze badges

1 Comment

vodkhang Over a year ago

I can make it automatically scale for me, but it keeps complaining me running out of memory, as you can look at my newest post here stackoverflow.com/questions/8561456/… . It seems that reason is I keep changing the size

Collectives™ on Stack Overflow

iOS - Scale and crop CMSampleBufferRef/CVImageBufferRef

5 Answers 5

21 Comments

8 Comments

2 Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

21 Comments

8 Comments

2 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related