The question cannot be answered with a simple 'yes' or 'no', be it about Polish or any other language, as "language difficulty" is not readily definable.
For example, there is an effort among linguists to use "language markedness" to predict language difficulty.
Jakobson (1941, 1963), however, observed that the marked members of
oppositions were acquired later by children and were found in fewer
languages, suggesting that they are not only more complex in their
abstract structure, but also more difficult for language users. Source.
However, it's not usually a problem during first language acquisition (L1 acquisition):
Although there is great variation between individual children and the rate of their language acquisition, there is little variation in the pattern of development between languages. One language is not more difficult than another, as we can establish by observing the ease with which children acquire different languages by the same age. Virtually every child develops linguistic and communicative competence, and it is learned naturally and in context, not arranged in an easy-to-difficult sequence. Source.
As for second lanquage acquisition (L2 acquisition), linguistic distance often plays a more important role than language markedness in whether a language will be perceived as difficult. Speakers of other Slavic languages will find a lot of similarities in Polish so they might not view Polish as particularly difficult.
According to
Markedness Differential Hypothesis (Eckman 1977)
a) Those areas of the target language which differ from the native
language and are more marked than the native language will be
difficult.
b) The relative degree of difficulty of the areas of the target
language which are more marked than the native language will
correspond to the relative degree of markedness.
c) Those areas of the target language which are different from the
native language but are not more marked than the native language will
not be difficult.
As far as English speakers go, there has been some research that focused on measuring the "distance" between English and other languages.
The paper by Hart-Gonzalez and Lindemann (1993) reports language
scores for 43 languages for English-speaking Americans of average
ability after set periods (16 weeks and 24 weeks) of foreign language
training. <...> The range is from a low score (harder to learn) of
1.00 for Japanese to a high score (easier to learn) of 3.00 for Afrikaans, Norwegian and Swedish. The score for French is 2.50 and for
Mandarin 1.50. These scores suggest a ranking of linguistic distance
from English among these languages: Japanese being the most distant,
followed by Mandarin, then French and then Afrikaans, Norwegian and
Swedish as the least distant.
According to this report, the score for Polish is 2.00. (Compare with some other Slavic languages: Russian - 2.25, Serbo-Croatian - 2.00, Czech - 2.00, Bulgarian - 2.00.)
Additionally, there might be other factors that can make a certain language "harder to learn" for L2 learners. For example, motivation or attitutde.
A second affective factor, which is formed by the cognitive
development of a person, that can make second language acquisition
difficult for an adult is attitude. Young children are not cognitively
enough developed to possess attitudes towards races, cultures, ethnic
groups, and languages. As the child reaches school age, attitudes are
acquired. It is agreed that negative attitudes towards the target
language, target language speakers, the target language culture, and
the social value of learning a second language can impede language
learning while positive attitudes can enhance learning (Ellis, 1994;
Brown, 1994). Source.
