Am I really incapable of learning math?

When I was in the 6th grade, I had some serious issues with fractions. Come to think of it, I wasn't able to memorize the times tables in the 3rd grade. So now, when I try to teach myself math that I should know, I get it long enough to work a few problems and then I forget how to do it, sometimes within hours, but sometimes a few days. Sometimes I'll even start to work something out (for example, the concept of how the RSA algorithm works), and I'll get to a point where it looks to complex for me to try to work any further. So, my actual question is, am I really incapable of doing this, or is this some subconscious excuse for me to be mentally lazy?