After much Internet hunting and many posts saying things like "why wouldn't you just buy a cheap Chinese touchscreen that fits" I figured it out. The drivers were getting some input just not the right one. If this was a Windows machine, it would mean that the wrong drivers were installed. Of course this is Linux so everything is going through Xserver. Luckily I found a calibration program for touch screens that works on the Xserver side and should work on any touch screen that uses Xserver.
First I had to take care of some prerequisites. So I booted Raspbian (easier to work with than Raspbmc) and downloaded the calibrator program (I believe this was made by Tias Guns but couldn't find a definite link).
Next I made sure everything I needed was installed.
sudo apt-get update
sudo apt-get install libx11-dev
sudo apt-get install libxext-dev
sudo apt-get install x11proto-input-dev
When I tried to install libxi-dev, it didn't work... Then I noticed it was installed with one of the other packages.
Next I unpacked and installed the calibration program.
tar xvzf (file, in this case "xinput_calibrator-0.7.5.tar.gz)
sudo make install
Then I launched it >Preferences/Calibration and it ran. Each time it was happy with my touches it spat out the newly set calibration settings (It took three run-throughs for me).
Note: It replaces the setting each time this way.
Next I needed it to apply these settings each time it booted. The program kindly tells you where to drop the settings... only it was wrong. I ended up changing the file /usr/[lib/share]/X11/xorg.conf.d/99-calibration.conf to the settings spit out at the end of the program. Except there's a catch. The setting changes for flipping the axis are not reset every time you calibrate. So if it was changed and you don't have that in the conf file when you reboot, it will not be there. I had to calibrate and change the file, then reboot and check it a few times. I ended up with this:
MatchProduct "eGalax Inc. USB TouchController"
Option "Calibration" "4000 140 435 3600"
Option "SwapAxes" "1"
It's not perfect. I think the right side could be better, but it works!
My next goal is to get Raspbmc to play nice.