For Photoshop...
What I do is create a new image, exactly the size of the cell I want the rounded corner in... (that's another thing, you need to create a a row for each horizontal border you want, and a column for each vertical; so if you want a box, 1 extra row on top and bottom, and one extra column on top and bottom).
So normally I do a border row/column setup that's 10px (wide and high accordingly); thusly the image I use is 10px by 10px.
From there, I turn on the Snap option, select the Circle tool, start it in the upper left corner, and drag down and right until it just meets the top and left edges. Then press Ctrl+T (Transform), to fine tune it. You should see a little circle with 4 lines sticking out of it at the bottom right corner, and the transform bars should go out beyond the image on the right and bottom sides. If not, stretch it until that little circle is the bottom right corner. Press "Enter" when you're done. If a question box comes up, select Ok or Apply to apply the transformations.
Now select the Crop tool, basically, just make it crop out that 10px x 10px space that is your canvas (gets rid of outside object).
This will create the upper left "Rounded Corner" of your table. Make the background color of the image the background color of your page (where the corner will be), and make the circle whatever color you want the border to be. Then just save it, and but the following tag in the upper left cell:
<img src="pathtoimage.xxx" height="10px" width="10px">
Where pathtoimage is the path to where you saved the image, and .xxx is the extension (i.e. .jpg if a jpeg image, .gif if gif, etc.)
the 10px and 10px will change if your image and cell are a different size, you just need to do this in case something else tries to format oddly, this will keep the image the same size.
After that, just rotate the image and re-save (new name) to get your other corners. If you apply any styles to the layer with the circle, you'll need to make sure you don't rasteurize it or anything, because when you rotate it, shadows and stuff will look really awkward (being that light will be coming from one side per corner, but when looking at both corners it will seem as though it's coming from two).
If you want to see an example of this, take a look at the menus on my site:
http://threadsoffrost.atspace.com
You can open up the source code if you want, but I use a lot of CSS (almost no actual formatting on the page itself), so you probably won't get sizes or anything (been a while since I worked on the main page).
Sorry for how long that was, but hope it helps.
Much Luck,
Paul J. Plusa