Find all upstream callers of a .NET/CLR method

Find all upstream callers of a .NET/CLR method

I'd like to find out all of the upstream callers of a particular method,
over multiple assemblies.
I don't need to resolve late-bound references or virtual method calls,
simple straightforward CIL call references are fine.
I've looked at several options:
Reflector: the Analyzer portion of the tool is not exposed as an extension
point.
FxCop: the CallGraph class requires an FxCop context to run in.
Roslyn: I've no idea where to begin, but anyway I'm not interested in
starting from source, I want to give it an assembly and work with the
bytecode and metadata instead.
VS Add-in: As with Reflector, the Call Hierarchy tool is not exposed to
add-ins.
I guess my only solution is to work-backwards by iterating through every
method in an assembly, looking for call instructions to my source method,
then recursively repeating the process for each caller method.
Assuming my proposed solution is the way to go, what is the best way of
doing it? MethodInfo.GetMethodBody().GetILAsByteArray() seems a bit
hardcore. Are there any libraries that make it simple to work with CIL
(like ASM for Java)?