The predicate on b.meta_key in the WHERE is negating the "outerness" of the LEFT JOIN operation.
One option is to relocate that predicate to the ON clause, for example:
SELECT a.user_id AS ID, a.meta_value AS color, b.meta_value AS size
FROM table AS a
LEFT JOIN table AS b ON a.user_id = b.user_id
AND b.meta_key = 'size'
WHERE a.meta_key = 'color' GROUP BY a.meta_value, b.meta_value
FOLLOWUP
Q: What if color is missing?
A: You'd need a table or inline view as a rowsource that returns every value of user_id that you want included in the result set. For example, if you had a table named "users" with one row per user_id, and you want to return every id in that table:
SELECT u.id AS user_id
, a.meta_value AS color
, b.meta_value AS size
FROM user u
LEFT
JOIN `table` a
ON a.user_id = u.id
AND a.meta_key = 'color'
LEFT
JOIN `table` b
ON b.user_id = u.id
AND b.meta_key = 'size'
GROUP
BY u.id
, a.meta_value
, b.meta_value
We also note that the GROUP BY in your original query does not include the user_id column, so any rows that have the same meta_value for color and size will be collapsed, and you'll get only one of the user_id values that have matching color and size.
Note that if the combination of user_id and meta_key is UNIQUE (in table table), then the GROUP BY isn't necessary.
In the normative relational model, attributes of an entity are implemented as columns. As a good example, the resultset being returned by the query looks a lot like the table we'd expect.
When an Entity Attribute Value (EAV) model is implemented in a relational database, the SQL gets an order of magnitude more complicated.
If you want to return ONLY values of user_id that have either color or size, you could get a list of user_id values with an inline view (query against table). This query is the same as above, but replacing the reference to the user table with an inline view:
SELECT u.id AS user_id
, a.meta_value AS color
, b.meta_value AS size
FROM ( SELECT t.user_id AS id
FROM table t
WHERE t.meta_key IN ('color','size')
GROUP BY t.user_id
) u
LEFT
JOIN `table` a
ON a.user_id = u.id
AND a.meta_key = 'color'
LEFT
JOIN `table` b
ON b.user_id = u.id
AND b.meta_key = 'size'
GROUP
BY u.id
, a.meta_value
, b.meta_value