As per context from the Bannon-hosted interview, the claim was made in the context of the US FDA approval for vaccinating the 12-15 y.o., so I'll address it [only] in that regard.
The claims is unsubstantiated from what I/we know so far, simply because there were zero deaths in the vaccinated group in the clinical trial that led to this (provisional) approval.
No thromboses or hypersensitivity adverse events or vaccine-related anaphylaxis was seen. Few participants in any cohort (≤0.4% through 1 month after dose 2) had serious adverse events, and none were considered by the investigators to have been vaccine-related. No deaths were reported.
Now in the video, immediately after the quoted claim he goes on to talk about VAERS data, but this is not directly in support of his previous claim (on youth) but to make another one comparing VAERS data with that from previous years (i.e. on other vaccines), which is another anti-vaxxer meme by now.
Entertaining that he might have done a quick calculation based on VAERS data in youth... that still doesn't pan out for me to anywhere near his 50x claim. In a recent CDC report "8.9 million U.S. adolescents aged 12–17 years had received Pfizer-BioNTech vaccine" and the "CDC reviewed 14 reports of death after vaccination". The usual anti-vaxxer method (from the previously linked question) is to assume all are caused by the vaccine, which would give 14 / 8.9M = 1.57 deaths per million vaccines. On the other hand, again from CDC data (warning: it's a dynamic page, numbers may change in the future--see footnote below for what data was used) in the same 12-17 age group there are 125+102 = 227 deaths from Covid-19... out of (approx.) 1.136M + 0.773M = 1.909M infections in the same age group, so that's 227 / 1.909M = 118.9 deaths per million infections. So instead of 50:1, as Yeadon claims, the ratio I came up with is 1:75, in the opposite direction, even using anti-vaxxer methodology.
The CDC insofar could not attribute any of those VAERS deaths as being caused by the vaccine; the report additionally says:
Impressions regarding cause of death did not indicate a pattern suggestive of a causal relationship with vaccination; however, cause of death for some decedents is pending receipt of additional information.
Despite his credentials, Yeadon has made quite a few other unsubstantiated claims about the dangers of the Covid-19 vaccines and generally downplayed the risk from the virus. Some of his more recent claims have been so improbable... that there was fact checking that he even said that... e.g. he prognosticated that vaccine booster recipients will die in two years and that it's part of a "mass depopulation" programme amounting to "deliberate execution potentially of billions of people".
Screenshots of the (dynamic) CDC data used "Data as of: Sunday, August 22, 2021 1:18 PM ET. Posted: Sunday, August 22, 2021 3:03 PM ET":

As a comment on the other answers, the mortality rate for a given disease (unlike the case or infection fatality rate) is obviously very dependant on the prevalence of the disease, meaning the mortality rate goes to zero as prevalence goes to zero, because the denominator for the mortality rate is the whole population, not just those infected (or showing symptoms). And surely enough, disease prevalence is a considered a factor in deciding whether to vaccinate a population, even if IFR/CFR for that disease is (or would be) high in case of an outbreak, e.g. there's no campaign or general recommendation to vaccinate Americans against Ebola, but there is one for those who work in specific areas. But it's a textbook case of faulty generalization to say something like "Americans don't get vaccinated against Ebola, therefore Ebola is not dangerous/deadly".
Even in the case of Covid-19 vaccination, some countries, e.g. Germany progressed from a recommendation (in June) to vaccinate just the adolescents who had additional risk factors to vaccinating everyone over 12 (in August). I haven't found a direct quote from STIKO (my German sucks), but they've been paraphrased as taking prevalence into account:
The advisory board to the German government said the U-turn had been made due to an evaluation of new scientific observations and data that showed young people face a high risk of getting Covid due to the more transmissible Delta variant.
and also the US rollout having had few problems in terms of side effects in children, plus the delta variant spreading...
STIKO said that they adjusted their recommendations based on new data about the potential side effects of the currently-available coronavirus vaccines, particularly after the large-scale vaccination rollout across adolescent populations in the USA. According to the committee, almost 10 million young people have already been vaccinated in the US. [...]
In addition, now that the Delta variant of COVID-19 is the dominant strain in Germany, STIKO said that mathematical modelling shows there is a significantly higher risk of young people getting infected in the event of a possible fourth wave in the autumn.
So, yeah, disease prevalence matters in such decisions, but it's not clear at all [to me] from the Bannon interview that Yeadon is talking about this issue.