PFI is defo very interesting but also very hard, I recommend not taking it lightly at all. Especially if you found both RA and prob a challenge since in my experience it makes prob look like high school level math, it really challenges your understanding. However you seem very driven/motivated so I'm sure you'll get through it provided you allocate enough time to it
The lighter the better if you're taking PFI tbh. Look into MAST30013 if you want something applicable, since a lot of quant finance problems require non-linear optimization methods (it requires DMOR but Im sure you can get that waived if you're willing to learn it over summer, this course should give you the basics ). I do not recommend CA and PFI in the same sem unless you found RA a total breeze, since also CA is limited in its applications in finance