Higher average speed will lead to more accidents. This is because braking distance doesn’t just double from 100 km/h to 200 km/h. In fact, it quadruples. It’s nonlinear, but people think it’s linear. Insufficient following distance is the second-biggest cause of accidents (the biggest is excessive speed). Translate this link.
You might have thought that big traffic jams are mainly caused by accidents. But in fact, it’s the difference in speed that will on higher speed increase non-linearly probability of abrupt braking, insufficient car-to-car distances and longer braking distance of cars. Modern traffic systems try to decrease the human factor as much as possible in order to reduce braking distance. Then you can increase speeds. This is because several cars will behave like a railway train with fixed distances between train carriages.
The second point you mention to adapt to avg. speed (> speed limit) to drive safer is kind of misleading and mirroring this linear thinking (esp. for very old drivers with reduced reflexes), as people will tend to choose non-linearly too small distances to the car in front with increasing speed. Trucks in the right-hand (slow) lane are very unlikely to cause traffic jams. This is because they behave mostly like the "road train" that was described in the hyperlink in the previous paragraph.
We do a lot of traffic research here in Germany. This is because some of our speed-limit signs are computer-controllable, depending on traffic density, average speed, and traffic-jam messages from drivers. These limits get adapted temporally on highly frequented highways, but only downward from the maximum speed limit. Also you probably know of new car features like "Autonomous cruise control system", that will use computer control to control the speed and distance to cars in front to you.
So of course if the speed limit is 100 km/h, one shouldn’t drive 50 km/h in the rightmost lane. Small speed-limited two-wheeled motor vehicles (max. 50 km/h) are forbidden on German highways. As far as I know, in Europe the highway speed limit of around 120 km/h and an advisory speed limit around 80 km/h is the best compromise of fast traffic and human factor. In Germany we have, in some areas, no general speed limit, since we are car fanatics and have a strong lobby :) However, most European countries have a general speed limit of around 120 km/h.