Based on the information from the RationalWiki article that Konrad found, the study in question seems to be one performed by Irina Ermakova and only published in the conference proceedings of the "Epigenetics, Transgenic Plants & Risk Assessment" conference in 2005 in Germany, partly sponsored by Greenpeace (Conference proceedings, the study in question starts on page 41).
In a correspondence on that article in Nature Biotechnology the following criticism of the study are listed:
Although Ermakova states that she received "soy clearly labeled as GM
and non-GM soy," she still has not established the identity of the
material tested, which is of paramount importance to an animal feeding
study.
and
The Archer Daniels Midland (ADM) catalog states and B.C. contacted ADM
on October 20 and November 5, 2007, to verify that they do not
sell—and have never sold—a 100% GM-soy product containing the
RR-40-3-2 line to which Ermakova refers.
The identity of the soy was not tested in the study, it was not verified if the GM and the non-GM soy were actually comparable. The company that the GM soy was allegedly procured from even states that they never sold such a pure GM-soy.
The authors are also state that the weight and number of pups are indicative of diet or environmental problems. They also criticize that the cause of death was not determined in the study.
The authors also cite three studies that contradict the results obtained by Ermakova and found no negative effect of the transgenic soy on rats:
The authors conclude:
We conclude then, that Ermakova's research relied on experimental
designs that fall short of internationally accepted norms, with
animals handled in such a way that even control lines were negatively
affected. The feeding studies used materials that were characterized
inadequately, incorrectly or not at all. Thus, no scientific
conclusions can be drawn from the work.