Shrinking balloon in liquid nitrogen

Applying the ideal gas law to the gas above the liquid nitrogen in a vacuum flask suggests that the volume of a balloon placed in it would reduce to one-fourth its volume. Actually, it reduces much more than that - virtually removing all the gas in the balloon. Why doesn't the ideal gas law apply in this case?

It is interesting to observe the amount of liquid present inside the balloon once it has been in contact with the liquid nitrogen.