There is the open source codec x264. For lower resolution images it might be fast enough to do on the fly on modern computers: http://www.videolan.org/developers/x264.html
You can try to use that to implement your own "video server". I'm not sure what you want to achieve and if one can easily expand the image transport. I guess the easiest thing would be to write your own server that receives the ros image topic, converts to h.264, transmits to your own client (somehow) and then generates a ros image topic again (if that is in your desire).
You could also look into gstreamer (if working on linux), which might help with the transmission of the video via RTSP, too. Not sure if it can act as server, but we're using it as client for some IP-Cameras.