That feeling is temporary. You're helping them by leaving.
Sooner or later, the rest of them will leave also.
People have been leaving Scientology since its inception. Before that, people left Dianetics, some of the better remembered are J.A. Winter (who wrote the Introduction for 'DMSMH'), and John Campbell (who published the first article on Dianetics in his 'Astounding Science Fiction' magazine). They left in 1950 and 1951.
Then came the transition to Scientology, after Hubbard lost the rights to the name 'Dianetics', and decided to drive former business partner, supporter and benefactor, Don Purcell, bonkers by concocting - pretty much overnight - "Whole Track Maps," and then the book 'History of Man' ("This is a cold blooded and factual account of your last 60 trillion years.") And more left.
Richard DeMille, who wrote some pieces now attributed to Hubbard, left around 1953. There were many others.
Around 1954, Hubbard, applying his "religion angle" announced that Scientology was a "Church," and more left.
In 1959, L. Ron Hubbard Jr. left.
Lots of people left in the early 1960s with the era of 'Security Checking', deciding that any group that insists that its members undergo metered interrogation, with such questions as "Have you ever had any unkind thoughts about L. Ron Hubbard?" was not for them.
By 1965, Hubbard wrote 'Keeping Scientology Working', invented the SP Doctrine and SP Declares, Disconnection, the Fair Game Law, and the very confidential, "deadly serious," and vital to your survival, history of the universe&your mind, and starting calling himself 'Source', and many more people left.
In 1967 came the Sea Org and Xenu, and more left.
By the late 1960s and early 1970s, the Commodore and his ashtray-carrying 13 year old girl servants were a law unto themselves on the 'Flagship', the "safest and sanest place in the universe," and more left.
John McMaster, 'The World's First Real Clear' and the 'Pope' of Scientology, left.
Writer William Burroughs came along, was involved briefly, did the Clearing Course then, upon completion, when asked if he wished to sign up for OT2, said, "No thanks," and left.
Many others left.
Then came the RPF and the RPF's RPF, and more left.
By early 1977, Hubbard had written his "LSD, Years after they had Come of off' HCOB, in which he described people who had ever taken LSD as "zombies," and soon after told them that they were required to wear rubber suits and run around for hours, and more left. (The rubber suits were soon replaced with the more appealing and marketable Purification Rundown.)
During July 1977 came the FBI raids, and two years later came the court-ordered release of thousands of pages of secret Scientology documents, revealing much of Hubbard's secret spying and covert attack tech, and even more left.
Then Hubbard decided that almost everyone was "Clear" and needed to exit Missions (which he was in the early stage of looting), and go "up lines" and spend their money there; and, while in hiding from (real or imagined) subpoena servers, Hubbard even wrote a "Common Sense Moral Code" - making him an authority on "morality" - that advised, amongst other things, against being sexually promiscuous so as to avoid having "ground glass in the soup," and more people left. (And soup consumption amongst Scientologists dropped.

)
This was followed, a few years later, by Hubbard responding to the Mission Holders Meeting of late 1981 by sending his #1 henchman, David Miscavige, to "handle" the next Mission Holders meeting of late 1982, and more left.
Then, roughly around that time, came the partial unearthing of Hubbard's past by way of the "Shannon documents," obtained through the Freedom of Information act (by a curious non-Scientologist), and more left.
By July 1984, there was Gerry Armstrong vs. Church of Scientology, and the further unearthing of Hubbard's past, and even more left.
Then came books like 'Messiah or Madman?' and 'Barefaced Messiah', and many more left.
Then came the Internet, and even more books, and Message Boards and Blogs and YouTube, and more left.
And, now, you have left.
Welcome.