Caste problem between me and my boyfriend

We both affair more than 4 years and still continuing blw us but, problem is he is upper caste and me lower caste now he started avoiding me, i am still loving him i can't imagine my marriage with another person bcoz already i had sex with him for many times with safety. I am feeling guilty now. Is there any advice for this and give me an solution please to change my life. I am student. I am basically from poor family i supported by sponsor to continent my study. I father is no more leaving under brother's house. And now leaving with 2 younger bothers in separate house i hope you can understand the short story of mine.