I think it's a great idea! And I always bring with me something like a deck of uno cards as a back up activity for a Little that young.
My Little is older but loved taking the ferry to Governors Island. The only thing I would be weary of is sea sickness.