The stellar mass function of the most-massive galaxies at 3 ≤z < 5 in the UKIDSS Ultra Deep Survey

K. I. Caputi 2 M. Cirasuolo 1 2 J. S. Dunlop 2 R. J. McLure 2 D. Farrah 0 O. Almaini 3 0 Astronomy Centre, Department of Physics and Astronomy, University of Sussex , Brighton BN1 9QH 1 UK Astronomy Technology Centre, Royal Observatory , Blackford Hill, Edinburgh EH9 3HJ 2 Scottish Universities Physics Alliance (SUPA), Institute for Astronomy, The University of Edinburgh, Royal Observatory , Edinburgh EH9 3HJ 3 School of Physics and Astronomy, University of Nottingham, University Park , Nottingham NG7 2RD We have analysed a sample of 1292 4.5-m-selected galaxies at z 3, over 0.6 deg2 of the UKIRT Infrared Deep Survey (UKIDSS) Ultra Deep Survey (UDS). Using photometry from the U band through 4.5 m, we have obtained photometric redshifts and derived stellar masses for our sources. Only two of our galaxies potentially lie at z > 5. We have studied the galaxy stellar mass function at 3 z < 5, based on the 1213 galaxies in our catalogue with 4.5-m magnitudes 24.0. We find that (i) the number density of M 1011 M galaxies increased by a factor of >10 between z = 5 and 3, indicating that the assembly rate of these galaxies proceeded >20 times faster at these redshifts than at 0 < z < 2; (ii) the Schechter function slope is significantly steeper than that displayed by the local stellar mass function, which is a consequence of both the steeper faint end and the absence of a pure exponential decline at the high-mass end; and (iii) the evolution of the comoving stellar mass density from z = 0 to 5 can be modelled as log10M = (0.05 0.09)z2 (0.22 0.32)z + 8.69. At 3 z < 4, more than 30 per cent of the M 1011 M galaxies would be missed by optical surveys with R < 27 or z < 26. Thus, our study demonstrates the importance of deep mid-infrared surveys over large areas to perform a complete census of massive galaxies at high z and trace the early stages of massive galaxy assembly. 1 I N T R O D U C T I O N The study of massive (M 5 1010 M ) galaxies at high (z > 3) redshifts allows for the investigation of the first epochs of efficient stellar mass assembly, when the Universe was less that a few gigayears (Gyr) old. It is now accepted that around 2040 per cent of the massive galaxies we know today were already in place by z 2 (Fontana et al. 2004; Caputi et al. 2005, 2006a; Daddi et al. 2005; Labbe et al. 2005; Saracco et al. 2005; Papovich et al. 2006; Arnouts et al. 2007; Pozzetti et al. 2007; Wuyts et al. 2009; Ilbert et al. 2010). Above redshift z 3, massive galaxies are more difficult to find (e.g. McLure et al. 2006; Kodama et al. 2007; Rodighiero et al. 2007) and they become very rare by z 56 (e.g. Dunlop, Cirasuolo & McLure 2007). The galaxy populations discovered at z > 6 seem to almost exclusively consist of intermediate-or low-mass galaxies (M 1010 M ), and their current study is mainly focused on constraining the epoch and sources of reionization (e.g. Bunker et al. 2004; Bouwens et al. 2008; McLure et al. 2009, 2010; Oesch et al. 2010). In fact, with the current instrumentation, only the rest-frame ultraviolet (UV) emission can be observed for the vast majority of these galaxies, which is used to estimate their levels of star formation, and ability to produce and liberate sufficient Lyman photons to ionize the intergalactic medium. Any constraints on the stellar masses of z > 6 galaxies are still poor, due to the lack of sensitive data at wavelengths that map the galaxy rest-frame near-infrared (nearIR) at these redshifts (although see e.g. Labbe et al. 2010 for an attempt). Nevertheless, different pieces of observational evidence suggest that massive galaxies as a significant population only appear at later times. Cosmological models of galaxy formation predict that massive galaxies can be quickly formed at high z in the high-density fluctuations of the matter density field (Cole & Kaiser 1989; Mo & White 1996). Thus, determining the first epoch of appearance and subsequent rise in the number density of massive galaxies with redshift constitutes a very important constraint on galaxy formation models. Investigating the period elapsed between redshifts z 3 and 67 is fundamental for this purpose, as it connects the earliest stages of galaxy formation after the epoch of reionization, with the better-studied period of galaxy evolution at 1 < z < 3, where a substantial population of galaxies are already massive and host very intense star formation and quasar activity. A global picture of the evolution of galaxy formation and growth can be obtained through the study of the galaxy stellar mass function at different redshifts. At low redshifts, the shape of the galaxy mass function is known down to low mass limits (M 108 M ; e.g. Cole et al. 2001; Baldry, Glazebrook & Driver 2008) and is well fitted by a double Schechter (1976) function (Baldry et al. 2008; Bolzonella et al. 2010; Pozzetti et al. 2010). Peng et al. (2010) proposed that the Schechter-function shape observed for the galaxy stellar mass function up to redshift z 2 can be explained as a consequence of mass-driven star formation quenching proceeding proportionally to the galaxy star formation rate in M M galaxies. This mechanism could be at play since earlier times, as preliminary determinations over small areas of the sky indicate that the Schechter functional form could be suitable to describe the bright end of the galaxy stellar mass function up to z 3.5 (e.g. Fontana et al. 2006; Kajisawa et al. 2009). Selecting galaxies by their rest-frame near-IR light constitutes a good proxy for a stellar mass selection. Rest-frame near-IR wavelengths are relatively unaffected by dust and the corresponding mass-to-light ratios have much smaller variations with galaxy age than at shorter wavelengths. For galaxies at z 3, the rest-frame near-IR light is shifted into observed mid-IR wavelengths. The Infrared Array Camera (IRAC; Fazio et al. 2004) onboard the Spitzer Space Telescope (Werner et al. 2004) is currently the most suitable instrument to conduct a mass-selected galaxy survey at high redshifts. The IRAC has operated at wavelengths 3.6, 4.5, 5.8 and 8.0 m until the end of the Spitzer cryogenic mission, and still operates in the two shortest-wavelength channels in the on-going warm campaign. Spitzer data constitute the last opportunity to conduct such mid-IR galaxy surveys until the advent of the James Webb Space Telescope after 2014. The Spitzer Ultra-Deep Survey (SpUDS; PI: J. Dunlop) is a Legacy Program that has provided IRAC (Fazio et al. 2004) and Multiband Photometer for Spitzer (MIPS; Rieke et al. 2004) imaging over more than 1 deg2 centred on the UKIRT Infrared Deep Survey (UKIDSS) Ultra Deep Survey (UDS) field (PI: O. Almaini). The UDS is one of the five on-going surveys which comprise the UKIDSS and is characterized by the existence of deep UV through K-band ground-based photometric data over an overlapping area of 0.60 deg2. The availability of deep and homogeneous-quality multiwavelength da (...truncated)