Inconsistent brushwork is pretty much what you're hinting at, and you're right, you should stick to one or at least very few brushes and never use soft ones (other than for rendering).
Texture by brushwork, not brushes.
Texture by brushwork, not brushes.