There is at least one reference that suggests Sid did serve in the war but at least based on FOTSW, he didn't grow up with the trio. I do agree with Peripheral that Sid was certainly at times restraining himself from getting involved because of Ivy. Of course, a lot of this is really just how Clarke wrote the episodes and I think its pretty clear he liked adding new characters all the time rather than keep using the same established one. So while Sid or Wally could have substituted for a guest star playing someone new in a lot of episodes, it was more fun to play with a quirky unknown. It also means the episodes where Sid or Wally did play a bigger role were more memorable.
An example would be "Getting on Sidney's Wire," which I think is a classic example of what makes the show great and is one of my favorite episodes. If we were seeing Sid all the time, spending an entire episode with him would have been less interesting.