These altars are easy to walk past, so know what you are looking for. In St. Peter's, the altar of St. Leo has Algardi's large marble relief of Leo meeting Attila above it; St. Gregory's remains have been under his altar since 1606; the Choir Chapel is often closed by a gate, and the place of St. John Chrysostom's relics can be seen through it. Bernini's bronze monument to the Chair of St. Peter, in the basilica's apse, is held up by statues of four Fathers and Doctors of East and West: Ambrose and Augustine, Athanasius and Chrysostom. Access to particular chapels in St. Peter's changes with liturgies and events, so check on the day. Entry to the basilica itself is covered in our answer on tickets for St. Peter's.
St. John Chrysostom and St. Gregory Nazianzen were both venerated in Rome for centuries, latterly in St. Peter's Basilica, until Pope St. John Paul II gave a large part of their relics to Ecumenical Patriarch Bartholomew I on 27 November 2004, as a gesture toward Christian unity; they are now enshrined in the Patriarchal Church of St. George at the Phanar in Istanbul. Rome retained a portion of St. John Chrysostom's relics: Pope Benedict XVI said in 2007 that they rest in the Choir Chapel of St. Peter's, where pilgrims can still pray. No source consulted for this answer says that any relic of St. Gregory Nazianzen remains in Rome, so this answer does not claim one does.
At San Clemente the present church stands above a fourth-century basilica and first-century buildings, and the excavated lower levels are visited separately from the working church above. Confirm visiting arrangements with the basilica.
St. Augustine and St. Ambrose are not in Rome: their relics are in Pavia and Milan. St. Monica, Augustine's mother, is buried in Rome's Basilica of Sant'Agostino.