Firstly the “ character is invalid JS. You need to use the " double quote character instead.
Secondly selecting an element by its inline style is a really bad idea. If you have absolutely no other choice (which would be selecting by class, id, or even just a hierarchical selector) then you could use the attribute selector or filter(). The latter is far more robust.
Thirdly, note that the exact CSS property you're checking against in this case is text-decoration-line, not just text-decoration as that's a shorthand for several other rules.
Lastly, when you create the new h3 element, if you're going to provide a property list, you need to include all properties in there. You cannot have the class in the element string, then other properties within the object.
With all that said, try this:
var $span = $('span').filter(function() {
return $(this).css('text-decoration-line') === 'underline';
}).replaceWith(function() {
return $('<h3 />', {
'class': 'classSample',
'html': $(this).html()
});
});
h3.classSample { color: #C00; }
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<span>Unaffected</span>
<span style="text-decoration: underline">This is now an h3...</span>
<span>Unaffected</span>