Confused

So. There is this guy that I met about a month ago. He just moved from California to pursue in getting his PhD at the masters program here at my school. I met him once and we got a long really well. Then, he asked for my number the second time we hung out. We then started to text all day for about a week and a half. Our texts would be 3-4 pages long for how much we had in common and how much we had to say. I was really starting to fall for him and it was kinda freaking me out because I haven't had anything serious with anyone for about two years now. Well, one night I stayed over with him because we were talking and my ride had left me, so we just talked and stayed up all night. He then ended up kissing me and we talked/kissed for the rest of the night, nothing else. So then the next day he doesnt text me like he did and when i texted him, he acted wierd toward me. So then I figured, "oh all he wanted was ass" and I dropped it. But then my friend/his roommate told me that he didnt want to get serious right now and he didnt want to move fast with me because he didnt want to hurt me. So i believed that but its still so awkward around him, there are times when I see him that he doesn't talk to me at all. Then there are times when he talks a lot to me. Then, there are times when I catch him staring at me from the corner of my eye and there are times where it seems like he doesn't even look my way. Its SO CONFUSING! Then, last night I just took a step and asked him why we didnt talk anymore. He said that I didnt text him anymore so he though I didnt want to talk to him. So now we are talking again but I dont know..is it even worth it?