Scientists fact-checked the UFO orbit study — and the math didn't hold up

Full text

The announcement made headlines. In October 2025, a group of researchers published a paper in a well-regarded, peer-reviewed astronomical journal claiming the extraordinary: They’d found evidence of objects orbiting the Earth years before the launch of any human-made artificial satellites. In another paper, they also claimed links between when these objects appeared in the sky and the dates of nuclear tests in the 1950s. If any of these results were correct, it’d represent evidence of orbiting non-human technological artifacts and an indication that whatever “piloted” these artifacts was interested in our nuclear capabilities. It’s easy to see why the papers grabbed headlines and interest.

So, the big question hovering around these works is — like all newly published scientific papers — can their conclusions hold up to scrutiny?

The first step in answering that question has now been taken. In June, a paper, soon to be published in Publications of the Astronomical Society of Australia, appeared online and offered an in-depth critique of those explosive claims. Given the importance of the story for UAP/UFO studies, I want to walk you through some details of the original papers and this recent response. Understanding the details is, of course, important on its own. Were we being visited by alien spacecraft in the 1950s? That seems like an important question. There are, however, important lessons in this story concerning how science can, and should, explore the contentious UAP topic. It also raises issues about the difficulties that come when science encounters UAP/UFO culture in general. We’ll explore those too.

The VASCO project

Let’s start with the original papers that set the story in motion. Both were published by astronomer Dr. Beatriz Villarroel (a researcher at the Nordita in Stockholm) and her collaborators. For a number of years, Villarroel has spearheaded a project called VASCO (Vanishing and Appearing Sources during a Century of Observations), which is based on a really good idea. The central method is to use old photographic observations of the sky to look for signatures of advanced technology (now generally called “ technosignatures ”). It’s a clever concept because the farther back you go, the less human-made stuff there is in the sky to confound the search. So, the logic argues that it might be easier to find non-human technosignatures in archived data.

The two papers VASCO researchers published in 2025 relied on observations taken in the early 1950s using the Palomar Observatory Sky Survey (aka POSS-I). The observations were captured on hundreds of photographic plates taken at Mount Palomar in California, and the images were later mosaiced together to form a sky map. Images of the same part of the sky were taken twice, and close in time. One plate used a red-light-sensitive emulsion, and one used a blue-light-sensitive emulsion. Villarroel’s excellent idea was to explore digitized versions of these plates and look for “transients” — things that appeared at specific locations in one of the plates but not the other (since they were taken at different times). The hypothesis was that, perhaps, these transients could be sunlight “glinting” off an orbiting object. The plates had been digitized so Villarroel and her collaborators could use computers to explore large portions of the sky and analyze the statistics of the transients.

The most important part of the first 2025 VASCO paper had the team looking at the difference between plates in Earth’s shadow and those that were not. If a reflective technological artifact were in the shadow, then there’d be no sunlight to produce glints. Therefore, those shadowed plates should have fewer or no transients. After analysis, the team claimed that not only did they find a deficit of transients in the shadowed plates, but the effect was significant at about the “22 sigma” level. (Here, sigma refers to how far away the signal — the detection of something or some feature — is from the average of the background/noise.) Usually, a “5 sigma” result is the gold standard for claiming you’ve made a discovery. A 22-sigma result is so high that it’s virtually unheard of in science. The team did note that when they used more stringent vetting of the transients, the significance dropped to the 7.6 sigma level.

A scientific paper on its own, however, is just a single step in the scientific process.

In the second paper, Stephen Bruehl (Professor of Anesthesiology at Vanderbilt University) and Villarroel looked at the timing of nuclear tests and the dates when transients were observed on the POSS-I plates. Bruehl and Villarroel claimed that there was a statistically significant overlap between when the nuclear explosions happened and when the transients were seen (i.e., the plates). In other words, it seemed like the things creating the transients (alien spaceships, perhaps) were clustering their appearances around nuclear tests.

So, of course, these papers made news. A scientific paper on its own, however, is just a single step in the scientific process. The next step is for the community to pore over the paper and see if the work holds up. Was the data set well prepared? Are the methods used to analyze that data well thought out and mathematically legitimate? Are the conclusions the authors draw from their analysis appropriate, or is there some flaw in the chain of reasoning?

A critique of the VASCO studies

That next step just came in with a new paper called “ Critical Evaluation of Studies Alleging Evidence for Technosignatures in the POSS1 – E Photographic Plates,” which was recently accepted by the Publications of the Astronomical Society of Australia (PASA). As the title implies, the authors do not find the VASCO studies convincing.

Before we dive into their critique, one noteworthy thing about the paper is its author list. It includes several working scientists who have been at the forefront of taking UAPs seriously. The lead author is Wesley Watters of Wellesley College. He is an astronomer who has written several papers reviewing UAP sightings and arguing they deserve a deeper, agnostic analysis . He has also been a member of Avi Loeb’s Galileo Project research team , which wants to study UAPs scientifically. Another author is Kevin Knuth , a physicist at the University at Albany, SUNY. Knuth published a well-known study of Navy-pilot UAP sightings exploring whether the trajectories seen by the pilots showed evidence for extreme physics at work. Knuth is also a research affiliate of the Galileo Project. What is striking about the author list is that it represents a group that cannot be seen as hostile to the basic idea that UAPs are a valid scientific problem or that they might represent technosignatures. This observation tells you nothing about the quality of their analysis and the critique that follows. It does, however, mean this is not a group of scientists with a bias against UAPs as a whole (in my experience, most astronomers do have this bias).

The Watters paper begins with an important historical example. Using photographic plates to look for things in the sky that appear and then disappear is not a new idea. For decades, astronomers tried to use photographic plates to find optical traces of the enigmatic gamma-ray bursts (GRBs), powerful explosions involving massive stars. As Watters and team show, even though a huge effort was put into the project, there were just too many “defects” on the plates that had nothing to do with the sky: problems with the photographic emulsion; dust grains on the plates; pits in the glass plate itself; errors introduced when plate copies were made, as well as other issues. It turns out that separating the defects and other issues from the real things captured by the telescope is incredibly difficult. This is especially true when you need to do it at scale — meaning you want to do statistics with tens or hundreds of thousands of potential transients. For the astronomers looking for GRBs, the photographic plates were simply not up to the job. That seems to be an important lesson for the VASCO project.

The defect problem is so bad that the Watters paper doesn’t use the term transient at all since so many of the “things” in the POSS survey aren’t transients (i.e., things seen in the sky). Instead, they begin their analysis by asking about “Selected POSS 1-E Features,” or SPFs. The Watters paper then asks several questions. What can we tell about the SPFs that the VASCO team pulled off the POSS-I plates? How did their choices of what to use effect their analysis and their claims?

It turns out that these are not easy questions to answer because the VASCO team did not publish the dataset used in this study. That means Watters and company had to go through an exhaustive effort to determine which parts of the POSS-I survey were used and which features were selected to be explored by the VASCO group as possible transients . Once this work is done, Watters and company go on to show that this analogous dataset is full of irregularities (something also shown in an earlier paper by Nigel Hambly of the University of Edinburgh). SPFs show up more often on the edges of the plates than in the center, or features cluster on the plates in irregular ways. Worse, when mapped across the whole sky, there are big stripes and voids where no features appear at all, while in other places the features inexplicably bunch up. All this points to the real possibility that the VASCO data set simply has not been vetted carefully enough. If so many, if not most, of the features are not true astronomical transients but plate defects, that is a real problem.

Even if some of the features on the plates are light from the sky, the Watters analysis of the analogous VASCO dataset revealed a deeper problem: the statistics. To pull statistically significant conclusions from the data, as the VASCO team hopes, the dataset as a whole must have certain properties. In particular, overall, it must be what’s called “uniformly random in distribution.” This means the features must, on average, be sprinkled randomly around in the same way as stars in the sky appear to be. Technically, this means they could be generated by what’s called a Poisson process . If you want to find statistically significant patterns in a big data set, the most common methods require that you start with data that at least approximates being uniformly random. Only in that case can you apply many of the usual mathematical methods of statistics, the methods that assure you’re not getting fooled by patterns you think you are finding .

It’s not that the VASCO idea was bad. It was a great technosignature scientific project — but the dataset does not seem up to the task.

Because of all the problems identified in the Watters paper, the VASCO dataset cannot be considered to be uniformly random. Worse, some of the conclusions rely on small numbers of data. For example, for the all-important calculation of the Earth shadow deficit, only 11 out of the 645 plates the VASCO team used are in the Earth’s shadow. That’s not a lot of data to begin with, and it makes doing statistics shakier. But then, through careful analysis, the Watters team found that only five of these plates were fully shadowed, with the rest only partially shadowed. Worse, it appears that the majority of SPFs occur in a single plate. So now there is even less data to work with statistically, and that one plate dominates the analysis. Finally, some of the shadowed plates were in those troubled bands where something weird was going on.

When Watters and collaborators repeated the shadow analysis, they found no significant deficit of features. The VASCO result had disappeared. These problems all stem from the misapplication of the statistical methods. Thus, that spectacular claim of a 22-sigma result is unsupportable. There was no statistically significant deficit of SPF’s in the Earth’s shadow.

The Watters team then turned to the claimed correlation between the dates of features appearing on the POSS-I plates and nuclear tests. That result also disappeared in terms of significance. The problem was that the VASCO team was not using the right number of days over which the POSS-I observations had been taken (they used 2,718 days rather than the actual 312-day time period over which their data were drawn; the POSS-I survey did not record observations every single day. Once this correction was included, the statistically significant correlation between nuclear tests and transient observations collapsed. What did exist was a relationship between when the telescope was used and when nuclear tests were done (possibly due to clear weather extending across the western states). But this had nothing to do with when features appeared on those plates.

Finally, the Watters paper concludes by showing how the whole VASCO project is plagued by a kind of circular reasoning embodied by their overuse of the term “transients” rather than “features.” Circular reasoning involves assuming the conclusion of an argument as part of its premises. The VASCO team wants to claim that some features on the POSS-I plates are actual things in the sky that only appear once and then are gone. They use the analysis of the shadow deficit and nuclear tests to claim these are real transients. But there is no independent validation that the bulk of these “transients” are not stuff screwed up on the plates themselves. Thus, the argument doubles back on itself: the shadow-deficit analysis assumes that the plate features are real transient events, and that same analysis is cited as proof that they are real transient events.

The bottom line is that the Watters paper offers a strong critique of the two news-making VASCO studies. It’s not that the VASCO idea was bad. It was a great technosignature scientific project. Rather, like the failed GRB studies, the POSS-I dataset does not seem up to the task. So, what does it all mean?

Science and UFO culture

The first thing to note is that this is not the end of the story. Science is like the blues — it’s a call-and-response kind of thing. A scientist named Bob publishes a paper. Another scientist named Alice publishes a critique. It’s then Bob’s job to address those critiques in the next published paper. If that’s done well, then the game goes on, and Bob’s evidence and arguments get better over time until, perhaps, the scientific community recognizes it as a proven fact and even uses that fact to do the next set of studies. So, the question now is how the VASCO team responds to these criticisms.

People arguing that peer review and publication are not necessary are like players showing up to a soccer game demanding everyone use the rules of baseball.

Another key point is where this call and response get played out. Over the past 400 years, science has figured out a system that works pretty well. It’s called peer review. Bob writes a paper and submits it to a journal. The journal then sends it to other scientists who are knowledgeable in the field (hopefully experts) for review. The reviewers make comments and suggestions that must be addressed . If the paper is judged to be too deeply flawed, then it may be rejected. This process is not perfect, but it does ensure at least a first level of quality control. Most importantly, it has given us wonderful things like quantum mechanics, molecular biology, and so on.

Raising this point gets us to the sociology of the overlap between UFO culture and science. A number of times, unrefereed papers that have not been submitted to any journal have appeared online, claiming to respond to the Watters paper’s deep analysis. While there is nothing in principle wrong with this, it’s definitely not science. Instead, it’s asking interested scientists to go chasing after work that has not been vetted by anyone and stands to be a waste of time. Certainly, these papers can’t be considered part of the all-important literature on which science has built its long and incredible success. In other words, it won’t endure. People arguing against this — that peer review and publication are not necessary — are like players showing up to a soccer game demanding everyone use the rules of baseball. That’s not how that game is played, and it’s played that way for good reasons.

This takes us deeper into the difficulty for scientists attempting to deal seriously with UFO culture. Advocates for the claim that UAPs/UFOs represent technosignatures of alien visitation often complain that scientists ignore the evidence and are unwilling to take the subject seriously. This was certainly not the case with the VASCO papers. Dr. Villarroel was, for example, invited to present her results at a meeting of the important Goddard Space Flight Center technosignature lecture series. The meeting was polite, with lots of good discussion.

When, however, scientists find fault with UAP/UFO data — as is the case with the Watters study — they are attacked as “part of the problem” or “government shills.” Online and elsewhere, it can get so unpleasant that at some point, a scientist might ask, “Why bother?” Everyone has interesting, fun research projects. Why spend time on this?

In this way, the UAP/UFO community needs to ask itself a difficult question. Does that community want their subject to be part of the actual scientific process — where beliefs can get legitimately challenged — or do they just want to hold on to their beliefs no matter what? There is a word for the second option. It’s called religion.

I once asked Adam Riess, an astronomer who won the Nobel Prize for showing that the Universe’s expansion was accelerating, what it felt like when he made the discovery. He was just a post-doctoral researcher when he was at the telescope collecting data for the epoch-making result. What did he think as he saw the Nobel-worthy conclusion come out of his data?

He told me he was terrified. The smartest people in the world were going to tear through everything he had done, looking for mistakes. Riess’ apprehension shows exactly why science works and why the VASCO team should have expected — and welcomed — critiques like those of Watter’s and collaborators. They are claiming something extraordinary, explosive even. It’s the duty of the rest of the scientific community to look for all the ways they might be wrong. That, after all, is the only way we can know if they are right. Isn’t the question “Are we alone?” worth that kind of effort?

This article A study claimed UFOs orbited Earth in the 1950s. Then scientists checked the math. is featured on Big Think .