For the most part, the GPL v2 and LPGL v2 and v2.1 require that, if you distribute GPL’ed or LGPL’ed code, you also make the source code available yourself. With a few exceptions, pointing to an upstream repository is not sufficient. See section 3 of the GPL v2 and section 6 of the LGPL v2.1. (Version 3 of the GPL and LGPL ease this restriction and permit pointing people to third-party servers; see section 6 of the GPL v3.)

So #1 and #3 are okay; #2, not so much.

Update: A few additional notes:

• In practice, if you simply point people to third-party servers, most people won’t care. Complaints have occurred, though; several years ago, the FSF contacted several Linux distros for not making source available. Slashdot and Linux.com covered this.
• If the libraries are distributed under “LGPL v2 / v2.1 or later,” then I believe that distributing library binaries but not the source would be okay, as long as you point people to the source, since LGPL v3 allows that.
• In my own experience, I’ve found it simplest to just make the source for everything available, rather than trying to track the details of when I do and don’t have to distribute things.

In response to your edit: Sections 5 and 6 of the LGPL specifically cover creating “a work that uses the Library.” Based on my understanding, it’s fine to distribute your work as GPLv2 or later without applying the GPL to the libraries. (Your source code itself is not a derivative of the libraries, so the LGPL doesn’t directly apply to your source code, and section 6 allows the resulting binary to use LGPL’ed libraries as long as the license terms are compatible.)

Standard disclaimers apply, I am not a lawyer, etc.

In general, if your application depends on those libraries to work, you must license the entire application as GPL. Since the libraries are already distributed under GPL, the source code for those libraries should already be publicly available, so in theory you shouldn’t need to redistribute the source code for those libraries yourself. The GPL concerns itself mostly with making your code public (including any modifications you made to the libraries), not code that is already public anyway.

There are a few caveats, however; the most important one is that the version of the library that your application works with might not be the same one that is in general distribution, either because you modified it, or it has been modified by the original developers since you incorporated it in your application. Consequently, it is more convenient for developers to have the entire body of actual code that is in use available in the project that you distribute, rather than going to several disparate places in an attempt to re-create the entire project, which might not even compile now.

Standard disclaimer: P.SE isn’t the most appropriate place to ask for legal/license advice

1) Yes; 2) Yes; 3) Yes

You are not required to distribute any library code as long as you make it very clearly which libraries and versions you are using AND you did not make any modifications AND (as Ross pointed out) you provide explicit information on where to get the source for those libraries. If anyone wishes to see the source code for those libraries, they can always go to their respective websites and download them from there. What I didn’t know and just learned from others on this page (+1 to those answers) is that you are also responsible for making sure that source code accessible based on the information you provide. And if third-party library website goes down, then it becomes YOUR RESPONSIBILITY to find another source or host the source yourself.

However, if you had to modify even a single line of code in a library (under both GPL and LGPL) you must distribute the library code that you modified. The motivation here is that if you are benefiting from using third-party code, the community should in return benefit if you find and fix bugs in that third-party code.

Under GPL you are required to make your own source code available, which looks like you are planning to do already, so you are good there. You do not have to package your source with the binaries, but your binaries distribution should reference a location from where your source code could be downloaded.

While I agree with the other two answers, keep in mind, that you are still in fact on the hook for providing sources to all the dependencies you use.

If you say “grab libBlah2.0” from libBlah.org, and the sources are easy to grab from there, no one is going to give you a hard time. But if libBlah.org shuts down and doesn’t host the sources anymore, you are then still responsible for stepping up and making the sources available. Not stepping up in that circumstance is a violation of the terms of the GPL.

Edit: Additionally, not bothering to distribute binaries of the libraries does not get you off the hook. Your project is a derivative work of those libraries, and so you are responsible for providing source to the libraries whether you distribute them as binaries or not, because the GPL/copyright demands so.