Jelly, 15 bytes

Øhi$€’s2ḅ⁴÷⁹$4¦

Try it online!

How it Works

Øhi$€’s2ḅ⁴÷⁹$4¦ - main link, takes input without # and all lowercase
   $€             - to each input character:
  i               - return its 1-based index in
Øh                - the string "0123456789abcdef"
     ’            - decrement (otherwise 0 is mapped to 1 and f to 16)
      s2          - split the characters into groups of two
        ḅ⁴        - convert from base 16 to decimal
            $4¦   - on the fourth element:
          ÷       - divide by
           ⁹      - 256

Jelly, 15 14 bytes

Øhi$€’s2ḅ⁴÷4¦⁹

Try it online!

How it Works

Øhi$€’s2ḅ⁴÷4¦⁹ - main link, takes input without # and all lowercase
   $€             - to each input character:
  i               - return its 1-based index in
Øh                - the string "0123456789abcdef"
     ’            - decrement (otherwise 0 is mapped to 1 and f to 16)
      s2          - split the characters into groups of two
        ḅ⁴        - convert from base 16 to decimal
           4¦     - on the fourth element:
          ÷       - divide by
             ⁹    - 256

-1 byte thanks to @EricTheOutgolfer