Purpose: We aimed to describe the psychosocial adjustments according to return to work (RTW) trajectories in breast cancer survivors (BCS) using a sequential and temporal approach. Methods: We used BCS data included from February 2015 to April 2016 in the Longitudinal Study on Behavioural, Economic and Sociological Changes after Cancer (ELCCA) cohort. RTW trajectories were identified using the sequence analysis method followed by a clustering. Anxiety and depression were assessed using the Hospital Anxiety and Depression Scale and the EORTC quality of life questionnaire was used at inclusion and all follow-up visits to assess Health-Related Quality of Life (HRQoL). Results: Fifty-two BCS were included in the study among whom four clusters of RTW trajectories were identified and labeled: slow RTW (N = 10), quick RTW (N = 27), partial RTW (N = 8), and part-time work (N = 7). Quick and slow RTW clusters showed slightly lower baseline mean levels of anxiety and higher levels of HRQoL. In the 4 years following diagnosis, BCS in the quick RTW cluster tended to report higher HRQoL in terms of functioning and less symptoms of pain and fatigue while those in the partial RTW cluster showed a lower HRQoL on almost all dimensions. All clusters showed an increase in pain and fatigue symptoms until 6 months followed by a tendency to recover baseline levels. Conclusions: The results of this study suggest that BCS who return to full-time work (slow and quick RTW patterns) recover better than patients who return to part-time work (partial and part-time RTW patterns).