Flutter: InteractiveViewer with a child that does not change its size

I would like to use InteractiveViewer to zoom and pan an image. On top of that image I want to position another widget which should always have a fixed size and mark a specific location on the image. Think of a pin on a map. The map can be zoomed but the pin keeps its initial size.

Something like this:

InteractiveViewer(
  child: Stack(
    children: [
      Image.network('http://placekitten.com/400/400'),
      const Positioned(
        left: 100,
        top: 100,
        child: SizedBox( //    <-- this should not change its size 
          width: 24,
          height: 24,
          child: Icon(Icons.pin_drop, color: Colors.red),
        ),
      ),
    ],
  ),
);

I can't prevent the pin from changing it's size when the image is zoomed.