I replaced Monocle's Font Awesome SVG+JS icons with their SVG sprite icons and it loads soooo much faster now! There was always a visible delay while the JS loaded in the SVG icons. Now it's just handled by the browser with no JS.
I always assumed that it was a slow network causing the issues, but I opened up the browser tools to check and it turns out the slowdown was happening while rendering the page, not when fetching external data. Goes to show it's a good idea to open up the browser tools once in a while to check out what's happening under the hood!