Quick answer: it's harmless. Maximum radiation dose is 20 times smaller than the safety limit.
Long version:
Well it all depends on several factors:
- spectral characteristic of IR source,
- beam shape,
- spatial energy density (energy distribution)
- exposure time
- temporal characteristic (continuous/impulse)
- current eye adaptation state (number of opened rods/cones)
Calculating it is not an easy task, even for the professional (you can find ISO standard, which is under code 10110 if I'm not mistaken, however you'd need to pay an access fee).
Generally speaking, you could assume that typical devices with output power under 20mW are safe to look at for several seconds if the beam diameter is above 2mm. It might partially blind you (glare effect for visible spectrum and NIR) and maybe cause minor, insignificant damage. Below is how well are different wavelengths (IR is above 950nm) detected by human eye:
EDIT:

http://www.dtic.mil/dtic/tr/fulltext/u2/663246.pdf
You can find necessary standards here (most importantly ISO 11254-2):
https://www.iso.org/obp/ui/#iso:std:iso:10110:-17:ed-1:v1:en
Simplified calculations
Assuming classical IrDA diode like:
http://www.excelitas.com/downloads/DTS_CR50IRDA.pdf
and typical eye (pupil and focal length) with ideal optical transmission case (no power loss in air and eyeball) as well as ideal transformation:

[ https://en.wikipedia.org/wiki/Geometrical_optics ]
We have a set of variables:
- pupil diameter: 7mm (average)
- eye lens focal length: 18mm
- eye diameter: 24mm
- lighting angle: 160 deg
- IR light power: 20mW
- wavelength: 870nm
Assuming that you touch the IrDA port to your eye, the distance between light source and eye would be 5mm, then, using the above equation with given data, the light limited by 7mm pupil will focus 7 mm behind lens, so basic geometric analysis gives as spot diameter on retina equal to (24-7)7/7 = 17mm. At 5mm distance, diode irradiates total field of pi*(5mm / cos(160/2 deg )^2=830pi mm^2 but only pi*(7mm/2)^2 = pi*12.5 mm^2 is transmitted through eye pupil.
Concluding: your retina would be exposed to 1,5% of 20mW power on 900 mm^2 surface of retina which is 3 * 10^(-4) W over 9 cm^2 = 3,3 * 10^(-5) W/cm^2 where the laser standard (more restrictive) for wavelength of 900nm is 7 * 10^(-4), so it is 20 times smaller than the safety limit.
[ according to "Safety with Lasers and Other Optical Sources: A Comprehensive Handbook" by D. H. Sliney, J. Mellerio ]