Are stars vital for cinema success?

The simple answer is yes ... and no!
Most of the big hit movies are to some degree 'star vehicles', but then there are plenty of counter examples too, from Paranormal Activity to Bridesmaids, or Billy Elliot for that matter!
More and more films US and UK films are funded by pre-selling distribution rights to foreign markets, a strategy UK social realist filmmakers such as Ken Loach and Mike Leigh relied upon for most of their careers. In their case, they could pre-sell to European markets where they were held in very high regard as auteur filmmakers. More mainstream films, even at the low budget level, will frequently encounter foreign distributors demanding the presence of some star with appeal in their territory, reinforcing the position of stars as vital to a film's prospects even if the evidence of success is very questionable.
At the tentpole level there is also a yes and no answer: big name stars with wide international crossover appeal help sell the movie, and their appearances on major TV shows across different territories/markets, their presence at red-carpet openings, usually helps generate buzz. However, technology can be just as big a factor: Avatar wasn't reliant on big names but rather the combination of groundbreaking CGI/SFX and the brand of director James Cameron, just as Michael Bay's name and the associated guarantee of lots of spectacular explosions that the producer of such successful bilge as the Transformers franchise brings is at least as important as any star name attached.
Here's a few links for further reading on this - useful for reflecting on your own target audience too.

Michael Bay gets narked at Transformers star slagging off the movie!

After enduring the unenthused gripes of his two leads, Shia LaBeouf and Megan Fox, Bay has apparently been forced to his breaking point by the recent comments from Hugo Weaving, who called his voice-acting gig as Transformers villain Megatron a "meaningless" job that he "didn't care about."

EC McMullen Jr gets rather cross at the very idea that stars sell pictures, specifically horror...

Two major factors will sell your genre movie to the fans. One is someone and something they've never seen before (so if you're dry-humping NIGHT OF THE LIVING DEAD, ALIEN, or SAW, you lose). Two is fan recognition. Fans are more apt to go to a movie made by a director, writer, or producer (even a freaking
composer or SFX man), than they are for a star. Celebrities are pretty much on the bottom of the list unless they've really established themselves in the genre like Dee Wallace, Lance Henrickson, or Robert Englund. Writer, Director, and Producer, Eli Roth understands this. When he made CABIN FEVER, he had to rely solely on audience appreciation of his movie. Once that was established, then he could sell his second movie, HOSTEL, featuring a bunch of nobodies (Eli understands that Big Stars won't sell your movie - although he still thinks sex will do it), and the fans plus more audience would turn out. With HOSTEL II, Eli discovered that there was such a thing as Over-hyping your movie.

From discussion forum theforce.net here's user JohnWesleyDowney with a nice yes and no take on this (the writer, a legendary film producer, he mentions is well worth reading - very entertaining, no holds barred insight into how crazy the film biz can be):

I read a book in the 80s by William Goldman and he explained this subject pretty thoroughly. Nothing ever changes. Yes, there are successful movies without stars, but film executives use the stars as a way to justify a project, and as an insurance policy. If a movie fails but it had a star, the executive can say to the Board of Directors, but we had a star in it! And what they hope for is that a star will "open" a movie, which means they make a lot of money initially on opening weekend, and then that feeds the revenue stream later on, because the movie had high visibility opening weekend. It's politics and economics, but it's sad but true.
There are exceptions. A superstar director can open a film (Spielberg's War Horse or Cameron's Avatar), and a story that's extremely well known can "open" a movie without a star, such as the Lord of the Rings.
But for optimum benefit, look what Disney did with Alice in Wonderland. They had an extremely famous story known for many generations, with one of the world's most popular stars, Johnny Depp, directed by a famous director with a following, Tim Burton. That's hard to beat.
I actually agree the story SHOULD be the thing, but it doesn't necessarily work that way in the real world.

Writer/director/producer J Neil Schulman (with some strong language) scoffs at the idea that stars matter, and attempts to take a ration al, evidence-based look at this (he doesn't in the end give that many examples - the reader comments pick up on this, but still some useful points):

As the writer/producer/director of one low-budget feature who’s now working on getting his next movie project off the ground, I am constantly contending with the Myth of the Star-Driven Big Budget Movie.
In a nutshell, I’m repeatedly told that unless the movie I make has a budget high enough to pay for lots of elaborate stunts, pyrotechnics, and CGI — plus several “A List” stars to put on the movie poster — nobody’s going to buy tickets for it.
Oh, really?
...
Every once in a while an independent filmmaker manages to disprove one or more of these premises — not necessarily all of them — by making an inexpensive movie without stars that sells a lot of movie tickets. This makes agents whose income is a percentage of the millions of bucks the stars they represent are paid — and movie execs whose lavish lifestyle is based on skimming off the movies made for megabucks — work overtime coming up with reasons why such movies are freaks rather than marketplace proofs that huge audiences can be found no matter how cheaply the entertainment is made.
...
YouTube has to be more frightening to movie agents and execs than a horse’s head showing up in their beds. You put up a video of a baby biting his brother’s finger and tons more people watch it than bought a ticket for their summer blockbuster or tuned in to the network’s latest season premiere. And here are the scariest words they’ll ever hear: for free. 

Over on vulture.com John Polone goes into some detail with examples showing that stars don't really make the difference (tho' he partially accepts there are exceptions such as Will Smith), but discusses the pressures that lead to stars continually being added to stretched budgets:

The foreign market is frequently cited as the reason that stars matter. Whenever I’m trying to get an independent film going, I am always presented with a list that comes from a sales agent working on behalf of a prospective financier; this list shows the predicted international value of various actors or actresses on whose name the film is to be financed. The problem with this practice tends to be that those who are valuable internationally may not be as worthy to distributors in the U.S., or right creatively for the movie. A top international sales agent gave me the example that, “Jean Claude Van Damme is still huge in Eastern Europe and he may still be the No. 1 star in Turkey. That’s why he continues to get projects financed.” That’s all fine, but I could never get him approved to lead a movie here. If I’m trying to get an independent movie going with a female lead, the first two names that I’m almost always given from whoever is selling the movie around the world, regardless of what the movie is about, are Milla Jovovich and Kate Beckinsale, based on the overwhelming strength of the Resident Evil and Underworld franchises in most international markets. It could be a comedy about a bookish woman from Alabama and the first thing I will hear from the sales agent will be “Jovovich.” If I counter with anyone from Rachel McAdams to Emily Blunt, I will receive a disappointed, “Maybe … if you get someone like Gerard Butler to play the guy who works at the gas station.” 

The argument that the international market, with the added millions it brings in, proves the existence of “movie stars” doesn’t really ring true to me. The list of the movies that did really well internationally is pretty much the same as those that scored in America. And the success of movies that were saved from ruin by disproportionate grosses abroad likely had nothing to do with stars; The Prince of Persia, with its $200 million budget, underperformed in the U.S. with a $91 million gross, but took in $244 million internationally. Was that because of Jake Gyllenhaal or because the concept and a lot of VFX were something that audiences in other countries found attractive? I would maintain the latter, given that other Gyllenhaal movies didn’t do as well internationally; action and CGI are the international language.

This is an international debate: Bollywood producers fall prey to star theory too, as this TimesOfIndia article argues:

The thriller featuring Rajeev Khandelwal as the lead character has been hailed by many critics as an 'outstanding movie'. In recent times there have been other low-budget movies that have also managed to do well though they've had little or no star presence in them. Movies like Bheja Fry, Superstar, Dor and Parzania have all managed to make a mark at the BO. In contrast, mega-star flicks like Tashan, U Me Aur Hum, Halla Bol and Bhoothnath left many disappointed.