Seiji has grown up in a family that is typically away on business in foreign locations, and thus has grown up mainly with his sister as his only family shown (although it is assumed his family was around as a child and began travelling when he reached adolescence). He has always been a very kind child, and would take anything anyone did to him in stride without so much as a single tear or any anger in his heart.
His younger sister, Rin Sawamura, would typically be the one defending him from bullies. Being she was capable of beating anyone up, she became Seiji's protector. She was also his greatest tormenter, loving to torment and tease her older brother (which eventually caused him a fear of water due to bathtub torment as young children). Eventually Rin took it upon herself to make herself become Seiji's physical teacher, forcing him from a young age to train very harshly. Eventually he gained an attitude, and eventually challenged a group of recurring bullies. He lost the encounter, but Rin remarks it was incredible to watch, as he never gave up despite being 5 on 1.
Seiji grew up using his physial prowess to knock down anyone. Despite his slender form, he has always been very strong. As his age went up, so did his strength, and has supposedly never lost a fight since. He is shown taking on entire enemy gangs single handed and knocking most, if not all, of them out in a single blow unarmed. This eventually led him to become good friends with Osamu Miyahara. Osamu uses Seiji as a threat against others, and never loses confidence in his protector. Miyahara remarks in one instance that the most incredible thing about Seiji is that he only takes on the bad guys, never harming weak or the innocent. He finds it incredible, and has thus grown up with a small sense of justice and believes the same idea.
Seiji's power is not all glamorous however, as he is also feared by his entire student body. As a result, he has been lonely and only had male friends. Desperately wanting a girlfriend, it came to the point where he would ask out any girl regardless of familiarity or if they even went to his school. This has resulted in 19 rejections before the series begins, with the 20th resulting in the first chapter.
Seiji's initial personality comes across as a cold, rude, and thuggish brute. He doesn't seem to have any indication towards caring about other individuals, and seems to desire and demand fights. Midori is treated rather poorly at first, with Seiji wrapping her totally in bandages, however Seiji quickly comes to find this reprehensible, and begins treating Midori with more respect. He also becomes very protective of her, angrily (yet unitentionally) knocking out Shuichi for trying to get her into a bikini and stood up to an actor who wanted to buy her. This eventually turns in love.
Seiji seems to enjoy doing nothing of importance and actually enjoys human interaction. Scenes involving him hanging out with Ayase, or other people and just talking show Seiji as relaxed, involved, and happy. In other words, Seiji is a people person who shut himself off due to violence.
Seiji owns a rather extensive collection of "adult magazines" that Midori gets rid of within the first few chapters, and shows a rather lecherous enjoyment of adult material, both film and paper. Still, after Midori shows up, it is shown that he becomes frustrated (as he cannot take care of this with a girl on his right hand).
Seiji is very absent minded and oblivious, often missing the intent of Ayase's attempts at attention. He also seems to always be attracted to women who are either disinterested or he doesn't know at all. He is also very innocent, and always seems to be unable to notice details or information by complete accident.
Seiji is also very selfless, willing to sacrifice his own safety in order to protect others. He willingly takes blow in one episode in order to protect others from harm, and drops whatever he is doing to save others in danger. He even saves Ayase at the beginning of the series from being taken away by a group of thugs despite her chastising him in front of the classroom earlier that day. This, coupled with another event, is what leads her to fall for him in the first place.
Seiji does have quite an ego, and has on several occasions taken on tasks he couldn't do, or tried things that would have otherwise been a bad idea.
Above all else Seiji is shown to truly care about other individuals. Although he doesn't start out this way, it comes to show that he indeed does care about the people in his life, and is more than willing to go above and beyond to aid them in whatever troubles they are having.