All Vedanta schools must have Atman at the core of all sentient beings (Bhagavad Gita 2.17-30). If it does not, then it is not Vedanta. Regardless of whether it is Dvaita, Advaita, Visishtadvaita, Shuddhādvaita or Bhedabheda, all have Atman and Brahman.
Similarly, all Buddhist schools must accept that all phenomena is not Atman or not Self, regardless of whether it is unconditioned phenomena (Nirvana) or conditioned phenomena. If it does not fulfill this, then it is not Buddhism. This is confirmed in the essay "Vedanta and Buddhism: A Comparative Study" by German indologist Professor Helmuth von Glasenapp. This essay includes a definition of Atman or Self, that is consistent with Bhagavad Gita 2.17-30. This applies to all Hinayana / Theravada, Mahayana and Vajrayana schools.
This topic of Atman is a fundamental irreconcilable difference between Vedanta and Buddhism.
By this definition, since Adi Shankaracharya preached Atman at the core of all beings, he was definitely an astika (a believer) and therefore his teachings fall under the umbrella of Hinduism. This can be seen in his compositions: Atma Bodha, Vivekachudamani, Aparokshanubhuti, Tattva Bodha and his commentaries.
The Buddha rejected the concept of an eternal Atman at the core of any being and also rejected Brahman, so he was a nastika (a non-believer).
Another criteria is the acceptance of the authority of the Vedas as revealed scripture (shruti). Adi Shankara accepted the Vedas, so he was an astika. Meanwhile, the Buddha did not accept the Vedas as evidenced in the Canki Sutta, so he was a nastika.
It can be seen in this answer that even non-Vedanta schools that are classified as astika (believer), all accept Atman.
So, based on these two criteria, Advaita Vedanta does not fall under Buddhism, and instead falls under Hinduism.
From Professor Helmuth von Glasenapp's essay "Vedanta and Buddhism: A Comparative Study":
Nothing shows better the great distance that separates the Vedanta and
the teachings of the Buddha, than the fact that the two principal
concepts of Upanishadic wisdom, Atman and Brahman, do not appear
anywhere in the Buddhist texts, with the clear and distinct meaning of
a "primordial ground of the world, core of existence, ens realissimum
(true substance)," or similarly.