Yes, chunksize changing is much slower than resync or a 'grow' that makes the array larger.
This is because the data needs to be backed-up continuously in case of a crash. So all the data is written twice, once to the backup file, and once back to the array.
Partly because of this, it also moves data in smaller chunks, so it is seeking back-and-forwards a lot more.
You might be able to increase the speed a bit by increasing stripe_cache_size in /sys/block/mdXX/md/, but don't make it too big or you will exhaust memory. A few thousand is probably OK. Don't expect too much gain from doing this though.