I am not sure whether there is any better way. If you need to work with large pictures and speed is a problem, then maybe .net is not the right environment.
You may try to resize the image to 1x1 using a method that computes correct pixel coverages, or resize it using a sequence of operations, where you shrink the image to 50% in each step. Hopefully the resizing is optimized.
Another way would be to skip pixels lik taking every 5th pixel in every 5th row. The result won't be precise, but maybe it'll enough for you.