I was investigating the bloated code size even when the main function was empty and found another way to reduce the code size
The Vorago IRQ handlers were not declared weak so they were always loaded even if they were not used with no way for the user to override these functions.
I declared them weak now which allows users to override them with a custom function or make them empty to save code size. I was able to save around 5-6kB by doing this.