Should be just fine if set for the bigger cassette, mainly because you're hopefully never running small x small. That's where you'd notice it if it was problematic. I've lazily done it for both road set ups and mtb chainring swaps and no problems. I think my neo has a 30t on it instead of the 34 on the speedmax right now.

*edited to add: plus your 34t cassette still has an 11t anyway so your derailleur can swing it either way