I've hiked this a few times and it is easily a favorite of mine. I had the pleasure of actually getting the summit all to myself one evening I hiked it solo after work. I would not say it's to difficult even with the rock scrambles because as long as you pace yourself both going up and down and are careful of your footing it's a enjoyable hike.
I've done it top to bottom in 3 hours solo but it can also be more of a leisurely hike with a group around 4--4.5 hours total.