A black hole is a rather complicated cosmic object, and one of the most common misconceptions about them is that they’re some kind of universal vacuum cleaner, constantly sucking material in towards their centers.
A black hole is formed when a star, which is at least 8 times more massive than our own sun, reaches the end of its lifetime and implodes on itself in a supernova. Supernovae release a huge amount of energy, and a large quantity of material is blown away from the star. The very core of the star, however, is undergoing a runaway collapse- the star is no longer producing energy in its core due to fusion (as our star currently is), so there’s no outward pressure to balance the inward crushing force of gravity. Gravity is able to overwhelm the repelling force of electrons between atoms (the same force that keeps you from falling through your chair), and also able to crush neutrons together, which is an even more difficult feat. The force of of gravity is so overwhelmingly dominant that the core of the old star is compressed to a density high enough that the escape velocity is greater than the speed of light.
At this point, the runaway collapse should continue, so that the core of the star collapses into a singularity, which is an object which takes up no space (we call it a point source) and is of infinite density. Infinities and physics currently do not play well together, so our descriptions of how space and time behave immediately surrounding the singularity begin to fail. (The singularity itself is currently most accurately described by a large number of question marks.)
Unfortunately for clarity, “black hole” has become a bit of a catch-all term that can apply to any part of the entire black hole. Astronomers tend to use it to refer to the entire area near and within the event horizon (the point of no return for light), which is fairly distant from the actual singularity.
If we are looking much further away from the singularity of the black hole (in other words, beyond the event horizon), physics is still able to describe the shape of the black hole’s distortion to space-time. At these distances, the black hole’s influence on the surrounding space is creating more of a gentle gravitational slope than a steep cliff. In fact, it has no more extreme an influence on the space surrounding it than a star of the same mass. If something happens to be in the area, it might roll towards the black hole, but if it doesn’t come close enough, it might just swing right on past, leaving the black hole untouched. If the material (gas, star, whatever it might be) wasn’t going to collide with the star, it probably won’t make it into the black hole.
Stars are usually relatively isolated – they may form within a cluster of other stars, but these clusters tend to be relatively loose, with large distances between stars. And since black holes form only at the end of an old star’s lifetime, any dust or gas that might have surrounded the star when it formed would have been blown away by a lifetime’s worth of stellar wind. By the time the black hole forms, there’s not going to be a lot of material hanging around for it to get close enough to trap. So there really isn’t an accessible “bulk” of material for the black hole to draw on!
There is, however, one major exception to this idea; stars which are found in binary systems. These are systems where two stars are in orbit around each other, and typically pretty close in age. If one of the two stars goes supernova first, the other star (which is probably also reaching the end of its life) may become a source of fuel to the black hole as it expands. Our star’s atmosphere will reach out to the orbit of Mars by the time it reaches full red giant phase; a more massive star would extend even further. If the ballooning star reaches out far enough, its outer edges might begin to fall towards the black hole instead of staying attached to the star itself. Even then, though, black holes are very, very bad at getting material that’s handed to them this way to get all the way down past the event horizon, which would be the moment the black hole can actually grow in mass.
So, not only are the black holes formed from a single star generally lacking access to a “cosmic bulk” of material, but even their gathering of the closest material is so inefficient that it’s very hard for them to grow quickly.