WHEN TO USE PNG:
- If your image has transparency/alpha channel
- If your image is simple/has large areas of the same or similar color
- If your image needs to be lossless, or when a jpg is noticeably worse quality
WHEN TO USE JPG:
- If your image is very large (640x480 or larger)
- If your image is a photograph/real life image
- If the image is extremely complex and png generates a very large file size
Also note:
You can rename jpg files to png files so that osu will actually read them (if you're skinning out spinner background or pause overlay, both of which are perfect as jpgs but might not be recognized with the extension "jpg")
Jpg files can be optimized in the following ways:
jpegtran -optimize -grayscale (self explanatory) -progressive (only if the file is larger than 20 kB)
When saving, use floating point DCT method. This is more accurate, more intensive, and generates a better quality image.
Also when saving, you can turn on or off subsampling. with subsampling on (2x2,1x1,1x1) the image will be smaller, but look worse. It "guesses" what the colors will be to compress more. Use subsampling with photographs/complex images. with subsampling off (1x1,1x1,1x1) it will create a better looking image, but will be bigger. Don't use subsampling with simple images. This is mainly for files that are too big to be pngs.
Don't use quality 98+. Each step up past 95 really gets into the exponential size. It's never lossless, so there's no reason to be that high anyway.
png files can be optimized as well:
scriptpng, truepng, pngout, optipng, etc...
This is not all important, but it can save 20% on average on your images. If you've got an image heavy map, it's a good idea.
I don't think there's an easy way to detect these requirements digitally, so it's really up to the user. One thing you might do is detect if a .jpg file really is a jpg, or if a .png file really is a png. I wouldn't list any of these as errors, just point it out.