If the videos on youtube are owned by you then there is no problem in keeping videos on your site. But if the videos are copyrighted and owned by someone else then you must have the necessary ownership rights or sharing rights. I think that's it.....
Read the disclaimer on youtube. Basically youtube is responsible for all videos and owners of the vidoe give up rights to ownership of the video uploaded onto youtube. It's perfectly okay to have a linked video from youtube on your site.
That may be all well and good, but Youtube has been sued by many major movie and music industries because of things posted on there. Because Youtube didn't have permission to host the music video or movie clip. I would stick with what has been mentioned already - if you don't have the permission of the author, owner, copyright holder, etc - then no.