We tend to think of black holes as some kind of cosmic vacuum cleaner, constantly sucking in all the material around it. And while it’s true that if you managed to very carefully drop an object into a black hole, you’d never ever get that object back, under normal circumstances, black holes are actually remarkably bad at pulling material that close.
There are two reasons for this; the first is that black holes aren’t actually attractive to anything for any reason other than gravity. Much like our solar system is in a stable orbit around the sun, the vast majority of a galaxy is in a stable orbit around the black hole, with no real reason to go plunging towards the very centre of the galaxy.
The second reason that black holes are bad at being astronomical vacuum cleaners is that they’re really, really, inefficient at getting material close enough to them to cross the event horizon and add to the mass of the black hole. Even small black holes, which exist in great numbers in a galaxy, are much better at tearing a companion star apart than they are at actually growing their own size by consuming the other star.
Material near a black hole tends to form what’s called an accretion disk- a thin, rapidly rotating disk outside the event horizon of the black hole. The gas trying to get to the black hole will speed up the closer it gets to the black hole, and any jostling between gas particles will heat the gas to incredibly high temperatures. At these temperatures, the gas will start glowing in X-rays, which flow out vertically away from the disk. Sometimes this process also causes huge galactic winds, which pushes material vertically away from the galaxy. A significant fraction of the material which could otherwise have made it to the black hole will get pushed straight back out again before it gets particularly close.
But that’s assuming that there’s a lot of material near the black hole, actively falling towards the event horizon. The supermassive black holes in the centers of galaxies have an additional problem - there might not even be any material around in the first place. The Milky Way’s central black hole, for instance, seems to be surrounded by stars, but almost no gas, so there’s no accretion disk around our black hole. In order to be shredded by a black hole, a star would have to come very, very close to the black hole. The star that orbits the black hole in the centre of the Milky Way orbits once every fifteen years (this is really short) and we’ve been (incredibly) able to watch it move around the black hole. It comes within a light-day of the event horizon, and that’s still not close enough to get torn apart or sucked in. (There are videos of the orbiting stars. Go watch them. Here’s the actual data and here’s the physical model from that data. They are both super cool.)
The fastest way for a black hole to grow in size - at least, as far as we know right now - is by crashing into another galaxy. When that happens, after things settle down, the heaviest objects will wind up in the centre, which for two galaxies will be the two black holes. Over time, the two black holes will lose enough energy while orbiting each other to merge into a single black hole. If the other galaxy was about the same mass as the original galaxy, this should double the mass of the black hole in one fell swoop - much more efficient than than by trying to build mass with gas.