I think you mentioned earlier you tend to go hard on shorter rides and sometimes bonk. Warning: NSCSB ahead. Back in college, I raced XC for a few years. I started in cat 3 (though back then it was A, B, and C instead of Cat1, 2, and 3). The C races were usually 8-10 miles, and they were nothing more than a sprint. I always finished middle of the pack.
Then I got sick the day before the A&M race and didn't think I would enter. Felt a little better the morning of, but I missed the start of my race. So I entered the B race, even though that meant I couldn't ever go back down to C. Those races were 16-20 miles, and were more about endurance...no one could sprint that distance. I didn't do well in that first one as I had bad calf cramps and had to stop a couple times to work those out. got passed by one of the Cat1 women who started 10 minutes after us too. she was fast.
I got much better results after that, usually top 10, with a number of top 5's and even a couple of podiums.
Point being, I realized that my strength was more in endurance riding. It always takes me a few miles to feel good on a ride. If you are a slow-twitch dude, you might want to do some longer rides and see how you feel on those.