Sizing work is hard, even harder if the quality of the requirements are variable.
Your team need well written stories that don’t dictate a technical solution, but rather outline the problem, acceptance criteria and any useful information.
If you want to solve this issue then regardless of whether you are tracking velocity or cycle time, the solution is refinement sessions.
Refinement sessions are dull when you first start doing them, but they are essential!
To start with your team may feel like they are trudging through mud BUT after a short time (if everyone is engaged). The quality of the stories will improve and the sessions become shorter and more efficient.
What may have started as a full team event may even get switched into a three amigo’s with a rotating cast from the team.
If all goes well, you should start seeing the estimates/cycle time become more consistent.
Hard work, I agree but worth it in the long run.

