The argument goes something like this:
1. God is something than which nothing greater can be conceived. (definition of "God")
2. If someone understands the concept of God (i.e. the concept of something than which nothing greater can be conceived) then God "exists in the understanding" of that person. (definition of "exists in the understanding")
3. It is greater to exist in reality than in the understanding alone. (More precisely: if x exists in the understanding but not in reality, and y is exactly like x except that y also exists in reality, then y is greater than x.)
4. The fool understands the concept of God (= the concept of something than which nothing greater can be conceived).
5. Therefore (from 2 and 4) God exists in the understanding of the fool.
6. Suppose for the sake of argument that God exists only in the understanding of the fool (i.e. not in reality as well). (This assumption will form the basis of a reductio ad absurdum.)
7. Then we could conceive of something exactly like what exists in the fool's understanding except that it also exists in reality.
8. The entity that we conceived in 7 would be greater than the entity that exists only in the fool's understanding (by 3)
9. But in that case what the fool conceived was not after all something than which nothing greater can be conceived (after all, we've just conceived of something greater).
10. So we have a contradiction! (Between 5 and 9)
11. So the assumption we made in 6 must be mistaken (since it led to a contradiction).
12. So God exists in reality. (6 was the assumption that God does not exist in reality; since 6 is mistaken, God does exist in reality.)
This is a very strange argument. But what exactly, if anything, is wrong with it? I think things start to go haywire at step 2, with the definition of "exists in the understanding." Of course, since this is a stipulative definition, it can't exactly be mistaken. If "exists in the understanding" is a technical term used in the argument, then Anselm is free to define it in any way he pleases. However, I think that this definition promotes confusion.
Why? Because it encourages us not to distinguish sharply between concepts and things. The concept of Curtis Brown is one thing; Curtis Brown is another thing. If someone understands the concept of Curtis Brown, then what they have in their understanding is a concept, not a person. If we don't make this distinction, then in the middle stages of the argument we can sort of wobble between talking about concepts and talking about objects.
The real problem, I think, then comes in step 3, because it's not clear in step 3 whether x and y are supposed to be objects or concepts. No matter which way we try to disambiguate it, 3 seems implausible; it's only our failure to clarify whether concepts or objects are intended that makes it seem plausible.
Suppose x and y are supposed to be objects. Then we don't have two objects to compare at all! x is said to exist in the understanding but not in reality. But this just means that although the concept of x exists, x itself does not exist. So we do not have two objects to compare in terms of greatness.
So maybe we should understand x and y to be concepts rather than objects. Now the idea will be something like this: the concept of something that exists in reality is the concept of something greater than is the concept of something that does not exist in reality. But this seems quite peculiar. Is the concept of a hamburger any different from the concept of an existing hamburger?
(As far as I can see, I can't conceive of a nonexistent hamburger. I can conceive of a concept of a hamburger that doesn't correspond to reality, but then I'm not conceiving of a hamburger, just of a concept of a hamburger. If we step back to the level of concepts about concepts, I can distinguish between (a) a concept of a hamburger to which no real hamburger corresponds, and (b) a concept of a hamburger to which a real hamburger does correspond. These two concepts are different, but this seems beside the point; in both cases, the concept of the hamburger is exactly the same.)