All DDR2 memory is supposed to be backwards compatible with other DDR2 sticks. DDR2 cannot replace regular DDR though becuase the notch is in a purposely different location. So you should be able to use DDR2-6400, although its likely to run at the 533Mhz speed of 4200 or the 667Mhz speed of 5300, rather than its full speed of 800Mhz - timed to support the maximum speed of your motherboard. This will probably not be noticible by you. I do not recommend mixing DDR2 memory of different types though. If you're doing a replace you're okay - if you're planning an upgrade I'd reconsider replacing all memory sticks instead.
I don't think PCI-Express (or AGP) cares what the video memory size is. The memory is controlled by the GPU on these cards, not the CPU or northbridges. I foresee no problems here assuming you've made the right selection in port size. If its a PciE-16x video card (which it likely is), make sure your motherboard can handle 16x cards (which it likely does) and not just 1x (which some of the earliest PciE motherboards could only handle.) Most motherboards also don't support dual video cards or Crossfire configurations either, so do research before you go down this path.
For this advice I expect you to post some new artwork for us
