I have included data-attribute in your html input.
<h3>Select multi checkbox inside 2 categories max:</h3>
<div class="col-xs-4">
<h5>Category 1</h5>
<input type="checkbox" data-category="Category1" name="tech" value="jQuery" /> checkbox
<br/>
<input type="checkbox" data-category="Category1" name="tech" value="JavaScript" /> checkbox
<br/>
<input type="checkbox" data-category="Category1" name="tech" value="Prototype" /> checkbox
<br/>
<input type="checkbox" data-category="Category1" name="tech" value="Dojo" /> checkbox
<br/>
<input type="checkbox" data-category="Category1" name="tech" value="Mootools" /> checkbox
<br/>
</div>
<div class="col-xs-4">
<h5>Category 2</h5>
<input type="checkbox" data-category="Category2" name="tech" value="jQuery" /> checkbox
<br/>
<input type="checkbox" data-category="Category2" name="tech" value="JavaScript" /> checkbox
<br/>
<input type="checkbox" data-category="Category2" name="tech" value="Prototype" /> checkbox
<br/>
<input type="checkbox" data-category="Category2" name="tech" value="Dojo" /> checkbox
<br/>
<input type="checkbox" data-category="Category2" name="tech" value="Mootools" /> checkbox
<br/>
</div>
<div class="col-xs-4">
<h5>Category 3</h5>
<input type="checkbox" data-category="Category3" name="tech" value="jQuery" /> checkbox
<br/>
<input type="checkbox" data-category="Category3" name="tech" value="JavaScript" /> checkbox
<br/>
<input type="checkbox" data-category="Category3" name="tech" value="Prototype" /> checkbox
<br/>
<input type="checkbox" data-category="Category3" name="tech" value="Dojo" /> checkbox
<br/>
<input type="checkbox" data-category="Category3" name="tech" value="Mootools" /> checkbox
<br/>
</div>
and modified the JS as follows:
$(document).ready(function () {
$("input[name='tech']").change(function () {
maxAllowed = 8;
var count = 1;
var first = $("input[name='tech']:checked:first").data("category");
var cnt = $("input[name='tech']:checked").length;
if (cnt > maxAllowed) {
$(this).prop("checked", "");
alert('You can select maximum ' + maxAllowed + ' categories!');
}
var cat = $("input[name='tech']:checked").each(function(index) {
if($(this).data('category') != first)
count ++;
if(count > 2)
alert("Can't have more than two categories");
});
});
});
Hope it helps.