If they are worm castings, and it sounds like they are, you are in luck. They will only enrich the soil and feed your tree. I'd gently rake some up and put them on other flower beds. Worm casting are black gold and people pay big bucks for them. Even consider starting a road side stand. All the Best And Happy Yardening, Nancy.