Identification of protective T cell responses against SARS-CoV-2 requires distinguishing people infected with SARS-CoV-2 from those with cross-reactive immunity to other coronaviruses. Here we show a range of T cell assays that differentially capture immune function to characterise SARS-CoV-2 responses. Strong ex vivo ELISpot and proliferation responses to multiple antigens (including M, NP and ORF3) are found in 168 PCR-confirmed SARS-CoV-2 infected volunteers, but are rare in 119 uninfected volunteers. Highly exposed seronegative healthcare workers with recent COVID-19-compatible illness show T cell response patterns characteristic of infection. By contrast, >90% of convalescent or unexposed people show proliferation and cellular lactate responses to spike subunits S1/S2, indicating pre-existing cross-reactive T cell populations. The detection of T cell responses to SARS-CoV-2 is therefore critically dependent on assay and antigen selection. Memory responses to specific non-spike proteins provide a method to distinguish recent infection from pre-existing immunity in exposed populations.