This is just conjecture on my part so take it with a pinch of salt
I think affiliate links are probably part of what google looks for when it comes to flagging up spam (obviously not all they look for, legitimate sites have affiliate links too), it's one box they can tick to class you as spam, so you don't want your site ticking any more negative boxes.
Are you confident the rest of your site doesn't tick any more negative boxes? thin/duplicate content, bad links? etc
A content site with some affiliate links, fine
A affiliate site with some content, no good.