On the market open on the ex-div date the share price is momentarily adjusted to reflect the dividend. Then the share price trades like every other stock based on market sentiment (supply/demand - buyers/sellers).
It's the same as non-div payers. On any given day the share price goes up or down based on a myriad of factors, including nonsense market information like some famous trader just sold his shares, some analyst upgraded or downgraded, the 10 year treasury yield notched up a fraction, or the Fed chair might have said something but could have been misquoted.
Day to day market adjustments say nothing about long-term performance. And the fact that a company retains cash vs paying some out to shareholders says nothing about whether management will wisely deploy that cash. It's entirely possible to do both - pay out some cash to the owners of the company and wisely deploy remaining cash to build the business. Imagine the owner of a pizza parlor paying himself a salary and investing remaining profits to build his business. Is he hurting the future prospects of the business by paying himself a salary?