just like he said, it needs to be converted and re-encoded to a vcd compliant mpeg-1 video stream, and then authored with nero, or another cd burning program that's able to make vcd's, and then it will play in your dvd player.
they don't have chapters, tho, are about vhs quality, only support 2 channel pcm stereo, and only hold about 1 hour per disc
if you want to be able to play avi and wmv files from your computer, and plan to do so on a regular basis, you can get a divx certified dvd player for as low as $40 now. it will say right on the box that it's divx compatible.
the quality will be much better than converting to a vcd.
if it's a short video, you could make what's called a svcd, which is mpeg-2 and much better quality, but also only holds about 1 hour.
there's ALSO what's called a kvcd, which can hold 2 hours of AWESOME quality video, but i have no idea how to make one

only ones i've seen were image files i got off the net.