No you can't, since the mod for one diff will belong to that diff (so that the mapper can handle the full spread easily). Even if you delete the diff, the mods will still be there. I suggest you just continue for the diff you're currently pushing for without being bothered by the mods.