size-luminosity relation test about descqa HOT 103 CLOSED

from descqa.

Sorry for coming to the party late --- I just got a chance to clean up and run #95 (output here).

But given the discussion above, it seems to me that we should revise the test so that it only tests bulge size for bulge-dominated galaxies and only tests disk size for disk-dominated galaxies. Is that right, @vvinuv @rmandelb @aphearin?

And if so, @vvinuv, can you update the test accordingly?

from descqa.

@yymao attempted to start to write this test during the Sprint Week but failed due to the lack of a cloned Yao. As a result, no progress has been made on this test so far.

from descqa.

Is this test compares the scaling relation of luminosity and size of observed objects to the simulated objects?

from descqa.

@vvinuv Yes!

from descqa.

@vvinuv If you can point us to a good data set for comparison, this would be very helpful. Thanks!

from descqa.

@vvinuv Or, even better, implement this test :)

from descqa.

@evevkovacs We have found the luminosity and size of SDSS galaxies in this paper http://adsabs.harvard.edu/abs/2015MNRAS.446.3943M . There is a catalog associated with that paper and contains luminosity and size of galaxies in g,r and i filters.

from descqa.

I would imagine there are CANDELS or other HST-based datasets that go to higher redshift and fainter magnitudes than SDSS. This would be useful in my opinion rather than only having a z<0.2 validation test.

from descqa.

I found a few issues to validate this test. For protoDC2 catalog the sizes are given for bulge and disk components. Therefore, to have a meaningful comparison to the data we need similar observational data. As far as I know that the two dimensional decomposition of data is available mostly for SDSS galaxies and a few high redshift HST based BCGs (correct me if I am wrong). We could not validate the protoDC2 unless the catalog has single Sersic half light radius. Another issue is that Buzzard catalog gives only FLUX_RADIUS parameter. I am not quite sure whether Buzzards folks (@j-dr et al) have plans to introduce the half light radius of Sersic component.

There are observational data available for galaxies with single Sersic component at higher redshift (van der Wel et al 2014). Since there are no progress is going on this issue I started writing a validation script for this purpose, at least for Buzzard catalog now.

from descqa.

There are 2-component Sersic fits in COSMOS, for galaxy samples going down to i~25.2 (they are getting noisy down there, so I wouldn't go to the flux limit). See http://adsabs.harvard.edu/abs/2014ApJS..212....5M appendix E. It just so happens that the server where the data can normally be downloaded from is down until mid-next week, but I can put it elsewhere for you if this sounds interesting/useful.

from descqa.

@vvinuv there's also one-component size size_true in protoDC2 catalog version >= 2.1.2 (which is a luminosity weighted size given the two components, not exactly the same as single Sersic half light radius but should be a reasonable comparison).

from descqa.

@rmandelb Thanks! I think it could be useful. However, I can wait until mid-next week. @yymao I think the luminosity weighted radius is different from the radius by fitting only single Sersic. I checked somewhere else but don't have a figure now.

from descqa.

@vinuv, re: buzzard, FLUX_RADIUS is an estimate of the half-light radius. FLUX_RADIUS is just the name of the source extractor parameter that the size distributions in buzzard are based on.

from descqa.

from descqa.

(It seems that @vvinuv is now working on this.)

from descqa.

This test is also going to be quite important for CL in order to test the impact of blending on red sequence colors. I expect that we will need to use something at higher redshift as well for validation data as has already been brought up. I might be able to dig up some HSC data that could be useful for this.

from descqa.

@vvinuv - following up on my comment on this thread from about 2 weeks ago (validation data), the server where the dataset that I mentioned is now online:
http://great3.jb.man.ac.uk/leaderboard/data
(scroll to the bottom, http://great3.jb.man.ac.uk/leaderboard/data/public/COSMOS_25.2_training_sample.tar.gz )

@j-dr - the dataset that I just linked to goes to i<25.2. The FITS are good for galaxies out to z~1. Do you think this is sufficient? I see you mentioned HSC, but my concern is that I'm not sure the galaxy radii from cmodel are that useful at high redshift. Have you seen section 6.4 in https://arxiv.org/pdf/1705.01599.pdf ? (It shows plots for axis ratio, but mentions that a similar problem exists for the galaxy sizes.)

Other thoughts on a validation dataset are welcome.

from descqa.

@rmandelb Thanks! Could you tell me how you calculate the intensity at half light radius for bulge and disk. Is that I = 10**((surface brightness * 2 * pi * half light radius - 5 log10(4 * np.pi * DL))/-2.5)?

from descqa.

@vvinuv - The numbers in these files are all observed quantities, so luminosity distance is not relevant. The intensity comes directly from the files.

Have you looked at the README that is packaged in the tarball? It gives equations for various quantities related to the intensities and radii. If you are uncertain after reading the README I'm happy to discuss, but since equations and parameters are in there I wanted to make sure you knew that reference existed.

from descqa.

@rmandelb I have read the README file and it says that SERSICFIT[0]: I, defined as the intensity at the half-light radius. However, I am not quite sure whether intensity means flux or the luminosity.

from descqa.

Everything in those files is flux.

from descqa.

@vvinu What quantities are you using from protoDC2?

from descqa.

Just to confirm something: my understanding for this test is that you want the total flux, not the intensity at a fixed radius. Do you agree? If so, please use the equation given in the README,

flux = 2pi*n*Gamma[2n]*exp(b_n)*q*R_{1/2}^2*I/(b_n^(2n))

(all the factors are explained in the file)

from descqa.

@rmandelb Thanks! I was supposed to use the magnitudes of bulge and disk and corresponding sizes from protoDC2. Are those the correct parameters to use?

from descqa.

I see. This means that 'I' is the surface brightness at R_(1/2)

from descqa.

If you use the flux defined using the equation I copied above, and convert to magnitudes, then you can compare with bulge and disk mags for protoDC2 (and it should indeed be flux, not intensity at fixed radius, that goes into that comparison).

You should also check how sizes are defined. As described in the README, the sizes in the files I sent are half-light radius along the major axis, so there's a sqrt(q) factor to convert to the half-light radius that is the geometric mean of major and minor axis. Depending on the protoDC2 size definitions, you may need this conversion.

from descqa.

Yes, I have checked that in the README file. I think protoDC2 has circular half light radius @evevkovacs @dkorytov

from descqa.

@vvinuv @rmandelb @dkorytov Half-light radii in protoDC2 are obtained by assuming that the disk and bulge components have n=1 and n=4 Sersic profiles respectively, and using the physical (scale) radius of the component, as given by Galacticus, as the characteristic scale in the profile. There is an analytic calculation that can be done to compute the factor that relates the half-light radius to this characteristic scale. These factors are used to convert the Galacticus scale radii into half-light radii, which are the half-light radii of the major axes.

from descqa.

OK, so if you are using the half-light radius of the major axis, then @vvinuv does not need the sqrt(q) symmetrization factor (but he does still need a factor of 0.03 to convert from size in pixels to size in arcsec).

from descqa.

@evevkovacs @rmandelb Thanks for clarification!

from descqa.

@rmandelb I can't get correct magnitude of the bulge. E.g. I get bulge magnitude of 30.61 for the first object in the file when I used -2.5 * np.log10(I) + zero point of ACS. I don't think it is the correct magnitude.

from descqa.

Well, there are some objects that have basically no flux in one component. Have you compared bulge vs. disk flux (before conversion to mags) to see whether you happen to have gotten an object for which 99.9% of the flux is in the disk, or something like that?

Can you confirm what you are using for the zeropoint of ACS? There can be issues there because of different zeropoint conventions.

from descqa.

@rmandelb Please see the figure uses fluxes of bulge and disk.

I am using ACS F814W zeropoint of 25.95928.

from descqa.

Most of the disk and bulge fluxes are in the range of 0.001 to 0.1

from descqa.

Pretty sure I know what the missing factor is, but in a meeting and cannot test my hypothesis until tonight. Made a note, will come back to this.

from descqa.

@vvinuv @rmandelb If you are trying to check apparent magnitudes and are using the observer-frame fluxes supplied in protoDC2, you need to be aware that there is a factor of (1+z) which has not been included in the flux. The observer-frame fluxes are just the integrals over the blue-shifted filter bands. (So need an extra correction of -2.5log_10(1+z) to get the magnitudes. Check with Dan)

from descqa.

@evevkovacs Good to know that!

from descqa.

@vvinuv - actually I was wrong. I can't find a missing factor, but I'm also unable to reproduce your plot. The fluxes I get are typically in the range 1-10, not 0.001-0.01, so my mags are quite a bit brighter as is reasonable.

Can you clarify how you made the plot? I made one just now by using the equation I pasted above, flux = 2pi*n*Gamma[2n]*exp(b_n)*q*R_{1/2}^2*I/(b_n^(2n)), using n=4 and 1 for bulge and disk, and pulling q, R_{1/2}, and I from the bulgefit entry in the file. If you want, I can send you the entire script?

And just to check: you said the plot was flux (which I thought would be calculated using the above equation) but the axis labels say the plot is intensity, which does differ in magnitude but also isn't directly used for calculation of magnitudes (only indirectly, by calculating the flux using the intensity and other quantities). So - I want to make sure we are calculating and comparing the same thing?

from descqa.

@rmandelb I read in the README file and thought the intensity 'I' in the fits file as the flux. It was the mistake. If I calculate flux from 'I' then I will get the flux in the range you mentioned. Please see the attached plot. However, it will be great if you could send me the script you were using. In the below figure the x-axis label is flux from bulge and the y-axis label is flux from disk.

I had trouble in understanding the parameter names in the fits file. I thought the intensity column was the flux and I didn't know that the flux equation should be used to calculate flux from intensity. I deliberately put the axis labels as intensity to let you know that I was using the intensity column from the fits file. If the values calculated by you agree with my values then are we good to go?

from descqa.

I'm sending you the script directly by e-mail so we can make sure that we agree.

from descqa.

Can I ask for some more details on what you'll be computing for this test? The description I see is

Is this test compares the scaling relation of luminosity and size of observed objects to the simulated objects?

(which was answered in the affirmative). Is this simply computing the mean relation, or will you be looking at the scatter/full distribution at fixed luminosity as well? I'm assuming it's the former, but I wanted to check. And if I missed this info please feel free to just point me there, sorry. Thanks!

(Note: this is explicitly not a request for you to do the full distribution if it wasn't in the plan--if we need it and it's not already being done I think it's simpler to check it separately. Just wanted to make sure I'm not duplicating effort/stepping on toes.)

from descqa.

@msimet See here for an example of this test. Also, some discussion on the implementation can be found in #56.

from descqa.

Excellent, thank you very much!

from descqa.

@msimet I think this test is computing the average size - magnitude/luminosity relation and scatter in the size as a function of redshift . May be you could compute the distribution of size at a fixed luminosity. It is all in the Rachel's COSMOS catalog and in the mock catalog.

from descqa.

@vvinuv just want to check the current status of this test. Last time I checked this test only takes Buzzard. Are you now switching the validation data to COSMOS so that we can compare protoDC2 as well? If so, can you update PR #56 to reflect the new changes? Thanks!

from descqa.

@yymao I wrote the validation test (didn't a working version yet) a few days ago and because of some other commitments I will be able to do after two days.

from descqa.

protoDC2 should now have the quantities needed to run this test.

from descqa.

Since the size parameters in two catalogs are different (protodc has bulge and disk sizes and buzzard has only total size) I am not quite sure whether I could write a single script for the validation test. @yymao Do you find a way to generate the plots for two different catalogs?

from descqa.

protoDC2 also has total size (a luminosity weighted total size). I understand that the detailed definitions are different between the catalogs, but as a validation test, I don't think it matters that much. If it does, we can always just skip one of the catalogs.

from descqa.

@yymao I think we need to compare the bulge and disk sizes to the corresponding observed sizes apart from the luminosity weighted size from the catalog. I am not quite sure what the luminosity weighted size means. @rmandelb gave me the COSMOS catalog and the validation test comparing the bulge and disk sizes could be a separate test of magnitude-size.

from descqa.

In that case, you can either implement a new test or update this test to validate disk and bulge sizes and skip catalogs that only have total size.

from descqa.

The catalog that I sent has fits to bulge+disk for all objects, and fits to a single Sersic for all objects. I don't see any reason not to use the single Sersic fit to get a total size. Some galaxies are not well-fit by a single sersic, but the size parameter will still be meaningful at the level we are doing the validation.

from descqa.

@rmandelb I think the issue is that there is no single size for galaxies in the protodc catalog and that is why we are comparing the bulge and disk sizes. @yymao says there is single size for protodc catalog but it is the luminosity weighted size for bulge and disk. I am not quite sure what does the luminosity weighted size of galaxy means.

from descqa.

Using COSMOS data I checked the Sersic size and weighted average of size of bulge and disk (without adding axis ratios) and found that the weighted average agrees with Sersic size with some scatter. See the figure. Since the total and the weighted average of protodc sizes agree I will compare the luminosity-size relation of van der Wel et al and compare the protodc size to Mandelbaum et al.

from descqa.

FYI the validation code for bulge and disk size is working from yesterday.

from descqa.

@vvinuv thanks for the updates!

The luminosity-weighted size is literally combining the bulge and disk sizes using their relative luminosities as weights. See the code here.

So indeed this definition of total size is not exactly the same as fitting a single Sersic profile. But again, I think at the level we are trying validate these quantities, it's probably not a big issue. And I think your plot also suggests the same. (@rmandelb would you agree with this statement?)

@vvinuv BTW, can you elaborate what do you mean by "validation code for bulge and disk size is working from yesterday"? Have you updated the PR?

I think we now agree to validate the total size - luminosity relation in this test. Do we need a different test to test bulge and disk sizes separately?

from descqa.

@yymao - I think I agree with your statement. I didn't comment on the plot before - but I find the errorbars confusing - why are they enormous and should that worry us? How were they calculated @vvinuv ?

from descqa.

@yymao @rmandelb Sorry for the late reply. I tested the validation code to compare bulge and disk sizes for protodc2 and testing it yesterday for some period of time. I will push the script for PR. I have already written the script and I think checking the bulge and disk sizes may not hurt. Also, there is a tilt in the comparison at larger Sersic radius. Please see the figure. @rmandelb sorry for the enormous errorbars in the plot. I was calculating the weighted radius using a filter and didn't apply that same filter on the plot. That is why I didn't catch slight tilt. The errorbars are calculated as the scatter in the bins of Sersic size. I was eliminating the weighted radius less than 1e-2 and greater than 100 pixels which are probably not physical. There are less than 2000 of them out of ~90000 objects, which is a very small number. Please see the updated figure

from descqa.

@vvinuv - are you looking for comments/feedback on https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-02-24&test=size_Mandelbaum2014_BD yet, or are you still working on the test? I had some questions about it, but I didn't want to send something premature if the test is still in the works.

from descqa.

@rmandelb yes I am looking for comments/feedback on the test.

from descqa.

OK, here are some comments on this plot:

• What is the bottom row of blank plots supposed to show?
• I think you only need the bulge/disk legend in a single panel, since it's really the same legend in all plots.
• What does the M_V in the title mean? The bottom axis label says that the test is based on I-band luminosity, so it's not clear how/why the V-band magnitude is involved.
• The label does not make clear what the points vs. shaded region mean - which is the validation data and which is the sim?
• I suspect the errorbars and shaded region are showing the entire population scatter (otherwise the shaded region wouldn't cover a factor of 100 in size!), but I think the error on the mean would be more meaningful and useful.
• For the real data, how were the sizes converted to kpc? Using the COSMOS photometric redshift? Were any cuts applied to the sample before doing this?
• When comparing the bulge or disk size, did you impose any cuts on the bulge-to-total flux? For example, if a galaxy has 99% of the flux in the bulge, then the disk size is not very meaningful. This question applies both to protoDC2 and to the validation dataset.
• Horizontal axis range seems a bit too wide, there is a lot of empty space on the plots.

from descqa.

@rmandelb Thanks for the comments and please see my partial answers to your questions

• The bottom panels are supposed to show the validation of simulated catalog until z=3. Currently, protoDC2 has no galaxies greater than 1 and I think catalog makers are working on that. However, I am sure that the figure should not be plotting in that way for these three redshift bins. I will check if I can do some tricks for that. Also, I haven't checked whether there are enough galaxies in COSMOS beyond z~1.5

• Agree

• M_V is an error, it should be M_I

• The shaded regions are COSMOS and points are simulation. I should specify in the legend some way

• The errorbars and shaded regions are showing for the entire population of galaxies. Did you mean that the errorbar should be calculated from a selected sample?

• I use the photometric redshift to convert the sizes to kpc. I didn't apply any cuts to the photoz. Could you tell me what cuts should I perform?

• Currently, I didn't apply any B/T cuts. However, as you said it is important to take bulge or disk size when the B/T is almost 1 or 0. Should I take only the bulge (disk) size for B/T > 0.85 (B/T < 0.15)?

• I should be taking care of those empty white spaces in the figure.

from descqa.

Hi @vvinuv -

Apologies for the delayed reply. I was under the impression that you were making some updates but it looks like not, so I will respond to the open questions now (and keep responding if there is more to resolve):

• I don't believe we need this validation test for galaxies above z=1.5, and I am not sure the validation dataset will be useful above that. So I would simply restrict the redshifts used for the test.

• You said this:

The errorbars and shaded regions are showing for the entire population of galaxies. Did you mean
that the errorbar should be calculated from a selected sample?

No. What I was asking is the errorbar showing the standard deviation? Or the standard error on the mean? (i.e., standard deviation divided by sqrt(N_gal)) The first corresponds to the population scatter while the second corresponds to how well-determined the mean is, and I think we want to show the second of these.

Currently, I didn't apply any B/T cuts. However, as you said it is important to take bulge or disk size
when the B/T is almost 1 or 0. Should I take only the bulge (disk) size for B/T > 0.85 (B/T < 0.15)?

I think you will find this cut eliminates far too many galaxies. Due to pixel noise and other effects (difficulty in separation of components due to resolution, imperfect galaxy models, etc.) you would eliminate a large number of galaxies this way. I would just split at a cut value of 0.5.

from descqa.

@rmandelb I am really sorry about not replying for a very long time. Please see the link to the recent size comparison https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-23_31

Please see the following answers to your previous and recent questions:

• What is the bottom row of blank plots supposed to show?

• Fixed!

• I think you only need the bulge/disk legend in a single panel, since it's really the same legend in all plots.

• Fixed!

• What does the M_V in the title mean? The bottom axis label says that the test is based on I-band luminosity, so it's not clear how/why the V-band magnitude is involved.

• Fixed!

• The label does not make clear what the points vs. shaded region mean - which is the validation data and which is the sim?

• Fixed!

• I suspect the errorbars and shaded region are showing the entire population scatter (otherwise the shaded region wouldn't cover a factor of 100 in size!), but I think the error on the mean would be more meaningful and useful.

• In the recent figure I am using standard error. However, the validation of van der Wel et al using the scatter. I will update that plot using the COSMOS data.

• For the real data, how were the sizes converted to kpc? Using the COSMOS photometric redshift? Were any cuts applied to the sample before doing this?

• No. As I answered in the previous comment, could you suggest me what is the sensible redshift cut?

• When comparing the bulge or disk size, did you impose any cuts on the bulge-to-total flux? For example, if a galaxy has 99% of the flux in the bulge, then the disk size is not very meaningful. This question applies both to protoDC2 and to the validation dataset.

• I am currently using your suggestion where B/T of bulge galaxies is greater than 0.5. However, I feel that the figure is very busy. Will work on something else.

• Horizontal axis range seems a bit too wide, there is a lot of empty space on the plots.

• Fixed!

• I don't believe we need this validation test for galaxies above z=1.5, and I am not sure the validation dataset will be useful above that. So I would simply restrict the redshifts used for the test.

• Removed data greater than z=1.5

from descqa.

@rmandelb I worked on making the B/T plot less busy. Seems like the average mean of sizes of bulge and disk from simulation don't clearly agree with validation.

https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-24_8

from descqa.

@rmandelb I think there is an issue with the B/T plot which may be reason why the sims are not agreeing with validation. @yymao when I search the GCR catalog for size_bulge_true and size_disk_true for protodc2 it seems like the search gives the same array. Unfortunately, I committed the changes of SizeStellarMassLuminosity.py to master and it was removed when turn to previous commit. It may take a day to update the script in the size-luminosity-test branch.

from descqa.

@yymao I submitted the script for PR

from descqa.

@vvinuv thanks for the updates. I'm working on reviewing/merging PRs this week and will get to yours soon.

from descqa.

@yymao @vvinuv - I think the plot in its current form looks rather worrisome. Before merging the PR, I would like to see what happens once the bulge/disk size question is sorted out. Or has it already been sorted out in a more recent run?

@vvinuv - to answer your question about redshift cuts, I think that once you're eliminating objects above z=1.5, you don't need further redshift cuts. The current set of bins is fine. It was only before (when the plot had panels for higher redshift) that I was concerned about issues with the redshifts.

from descqa.

@rmandelb there's still some technical bugs in PR #95 and I have asked @vvinuv to fix them. We will post the updated run as soon as we have it.

When you say "worrisome" do you mean that if the test is implemented correctly the catalog is worrisome, or do you mean something else?

from descqa.

When you say "worrisome" do you mean that if the test is implemented correctly the catalog is
worrisome, or do you mean something else?

I mean if the test is implemented correctly the catalog is worrisome.

I don't look for super close agreement, but a factor of 3-4 discrepancy in size at fixed luminosity is worse disagreement then I am comfortable with - unless I am simply misinterpreting the plot? To be more specific, on https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-24_8&catalog=protoDC2&right=2018-03/2018-03-24_8/size_Mandelbaum2014_BD/protoDC2/size_Mandelbaum2014_BD.png I am looking at the following:

• top panel: compare the red points with the red lines (the ones that show the bulge size for bulge-dominated galaxies); I'm ignoring the orange points and lines for now since disk size is hard to measure and interpret for bulge-dominated objects. It seems like the simulated bulges are all larger than the real ones by a factor of ~2 and the size has almost no trend with luminosity.

• bottom panel: compare the blue points with the blue lines (the ones that show the disk size for disk-dominated galaxies); I'm ignoring the green for now since bulge size is hard to measure and interpret for disk-dominated galaxies. It seems like the simulated disks are all smaller than the real ones by a factor of 3-4, and have almost no trend with luminosity.

from descqa.

Thanks, @rmandelb this is very useful. Once @vvinuv fixes the PR I'll double check for bugs and then rerun the test to see if your worries persist.

from descqa.

@rmandelb I share the same concern. Eve told me about protodc2 and said that the sizes of bulge and disk are different. In this figure there are something wrong and the script finds only one size. It could be the total size or any sizes of bulge and disk. I will post the correct figure soon. @yymao I will update the PR after solving these issues.

from descqa.

@rmandelb The galaxy shape model is described in Section 5.4 of the protoDC2 note https://www.overleaf.com/8095694zthbdwgqxqzf#/28574272/. We assume n=1 Sersic and Hernquist profiles for the disk and the bulge with the characteristic scales given by the Galacticus scale radii. We compute the half-light radii by integrating over these profiles. The values looked reasonable for protoDC2 v2. We are investigating why the distributions changes so much in v3.

from descqa.

Hi @evevkovacs - that link doesn't work for me (it says it's password-protected and you have to invite me). However, I thought you used to use Sersic n=1 (exponential) and n=4 (de Vaucouleurs) profiles for disk and bulge, not Hernquist for the bulge, so thanks for clarifying; I had misunderstood the model.

from descqa.

@rmandelb The reason we don't use n=4 Sersic for the half-light radius computation is that it gives a very large conversion factor when you integrate the light profile to find the relationship between the scale radius and the half-light radius. The n=4 Sersic has a long tail, and the answer you get depends on where you cut off that integral. So for definiteness, we used Hernquits instead (which looks just like n-4 over almost all of the range) Conversion factors are ~ 1.6 and 1.8 for the disk and bulge respectively. The pdf file for the note is also on https://confluence.slac.stanford.edu/display/LSSTDESC/Extragalactic+Catalog+Development+for+DC2 Click on the first bullet to find the link to
DC2-prototype-catalog_release_2.1.pdf

from descqa.

Hi @evevkovacs - for what it's worth GalSim can very quickly do the scale radius to half-light radius conversion as a function of sersic n (the calculations have been numerically validated for convergence even up to sersic n=6). It can do it without and with truncation of the profile at any point that you specify. I am not saying you have to change something at this stage, but it sounds like this might be a useful utility for your calculations in general since you wouldn't have to code up or test the profile integrations.

from descqa.

Sorry for the long silence. @evevkovacs I checked the script and it is taking correct values for sizes. Sizes are not a function of luminosity even if I multiply those by 1.8 and 1.6 for protoDC2.

https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-05-21_26&test=size_Mandelbaum2014_BD&catalog=protoDC2

from descqa.

@vvinuv Would you be able to run this test on the latest versions of protoDC2? The catalog names are
proto-dc2_v4.4, proto-dc2_v4.5, proto-dc2_v4.5_rescale. To run your test (I'm assuming you have a private version of the code) you need to do this in your local descqa directory:
./run_master.sh -c proto-dc2_v4.4 proto-dc2_v4.5 proto-dc2_v4.5_rescale size_Mandelbaum2014_BD -p ~kovacs/gcr-catalogs_v4/

The last -p points to my catalog reader at nersc, otherwise you won't be able to read these new catalogs. If there is any issue with doing this, could I clone your code from github? I see there is a version in the master branch of DESCQA. Is that the version that you are running? Thanks Eve

from descqa.

from descqa.

from descqa.

from descqa.

@vvinuv I just cloned your repo and switched to the size-luminosity-test branch. When I ran the test, I got an error:
UnboundLocalError: local variable 'ylim' referenced before assignment
[ERROR][2018-05-21 14:52:51,857] Exception occurred when running validation size_Mandelbaum2014_BD on catalog proto-dc2_v4.5. Below are stdout/stderr and traceback:
Traceback (most recent call last):
File "/global/u1/k/kovacs/descqa2-local/descqa_vv/descqarun/master.py", line 348, in run_tests
test_result = validation_instance.run_on_single_catalog(catalog_instance, catalog, output_dir_this)
File "/global/u1/k/kovacs/descqa2-local/descqa_vv/descqa/SizeStellarMassLuminosity.py", line 201, in run_on_single_catalog
ax.set_ylim(ylim)

Could you please let me know when the test runs again, and I will try again. Thanks, Eve

from descqa.

from descqa.

from descqa.

@rmandelb @msimet @aphearin Could you please take a look at the size-luminosity relation in https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-05-21_38&test=size_Mandelbaum2014_BD&catalog=proto-dc2_v4.4 and let us know if the agreement is acceptable?

from descqa.

I'm not sure, actually. A factor of 2 average size difference might be enough for us to care about, and these look within that limit for the bulge sizes with B/T>0.5 and within the disk sizes for B/T<0.5. However, the other component looks more off (but this matters proportionally less because of the bulge vs disk dominance). Will the total size-luminosity test (vs van der Wel) be run on these catalogs as well?

If we have only this to go on, I'd say it's probably okay, but I'd like the backup of the total size test as well, if possible.

from descqa.

@rmandelb and @msimet - I was surprised when I saw this level of discrepancy. The DC2 model for the size-luminosity relation comes directly from the fitting function provided in Zhang & Yang 2017, which is based on SDSS galaxies. So I dug back into the paper, and I found that Youcai and Xiaohu actually provide fitting functions for the size-luminosity relation of two distinct morphological classification: one based on Galaxy Zoo, and another based on B/T in the Simard+11 catalog. You can see the differences in the plot below:

In protoDC2 v4.4 & v4.5, we based our size modeling on ZY17 fitting function parameters of disks/bulges defined by the Galaxy Zoo classification; according to the DESCQA test under discussion here, our bulge-dominated galaxies are too large. If we switched to a model based on the Simard+11 B/T classification, the figure shows that sizes in our disk-dominated systems would hardly change, but bulge sizes will be brought down ~30-40%, depending on luminosity.

Note that these changes will not alter the sizes of very luminous disks, which appear to be significantly larger as reported in Mandelbaum+14 relative to ZY17. Inter-publication variance is particularly large when measurements of size and morphological classification are concerned, even when different profile-fitting codes are run on the exact same dataset; I suspect it will be difficult to bring the model into better agreement with the data beyond the many-tens-of-percent level. I'm bringing this up since @msimet mentioned using an entirely different measurement (van der Wel+14) as an independent consistency test. According to this DESCQA test, our size-luminosity relation appears to be in reasonably tight agreement with van der Wel+14.

CC @katrinheitmann @dkorytov @evevkovacs

from descqa.

@aphearin @msimet The biggest discrepancy between data and the catalog in the DESCQA test is for the bulge component for galaxies with B/T<0.5. (late-type). The catalog and the data agree quite well for B/T>0.5 galaxies. Overall, the disk component sizes are in pretty good agreement with the data for all galaxy types. I don't think switching models above will help, since the biggest difference in the above models is for early-type galaxies and we already have pretty good agreement there. As Andrew points out, since the total size-luminosity relation agrees well, further iteration on the size model does not seem warranted at this time, but the final word should come from the WL WG.

from descqa.

@rmandelb and @msimet - When evaluating whether the current size-luminosity model in protoDC2 is sufficient for purposes of cosmoDC2, I think we should be using this earlier DESCQA test and not the much more recently completed DESCQA test currently under discussion. My reasons are:

1. The first DESCQA test was completed prior to the deadline for WGs to contribute the validation tests that they considered an important priority. In general, it is perfectly fine for new tests to be added to DESCQA, and the extragalactic catalog can and will continue to improve long past completion of cosmoDC2. However, for purposes of cosmoDC2 evaluation, the deadline for a test to be considered for cosmoDC2 evaluation has passed.

2. Deadlines aside, I would like to better understand the scientific motivation for test 2 over test 1. In the first test, the size-luminosity relation is evaluated separately for disks and bulges, which is one of the more basic/common summary statistics studied in the literature. In the second test, the galaxy sample is first split on B/T, and then within each split subsample, the size-luminosity relation is evaluated separately for disk and bulge components. So the main thing that this second test offers is a separate validation for the sizes of bulges in disk-dominated systems, and also the size of disks in bulge-dominated systems. In my experience studying a few different morphology catalogs (primarily Mendel+13 and Meert+15), this latter measurement strikes me as a highly uncertain measurement to make, and so if we are going to prize this measurement highly enough to put into DESCQA, I would like to understand what the downstream scientific motivation is, so that we can effectively modify the model.

CC @vvinuv @evevkovacs @katrinheitmann @yymao

from descqa.

Hi all - apologies for missing some of this traffic; was offline yesterday for personal reasons, then traveling today. A few comments, but since this is long I will say the brief summary is that I think that the results of the size-luminosity relation tests for v4_5 are acceptable:

• @aphearin I hope you will please correct me if there is a misunderstanding, but based on your comments, here's how I see the purpose of this validation test: You are defining the size-luminosity relation in the sims using a very well-measured relation that is valid for galaxies at z<0.2. The WL group for DESC cares about galaxies at z~0.7-1. Hence the primary purpose of the validation test is to provide some input into your method of extrapolating the parameters of that relation to higher redshift, and that's why having a CANDELS (van der Wel+) or COSMOS (Mandelbaum+) comparison is important. Do you agree with this statement?

• Based on my experience, if you have bulge-dominated galaxies in real data with low to moderate S/N (anything less than ~200), you should not try to use the disk sizes for validation tests. If the bulge dominates, then the disk parameters tend to be quite noisy and sensitive to assumptions about both the disk and bulge models. So in a validation test that uses bulge+disk fits, I only tend to use the bulge parameters for bulge-dominated galaxies, and disk parameters for disk-dominated galaxies. If other agree, we could even go so far as to remove the other curves (or de-emphasize them visually in some way). This is not necessary in all cases, but for the COSMOS and CANDELS data that go into both of our test datasets, this is the case. This is another way of saying that I agree with your tendency to trust this test over two of the curves shown on this test.

• I do want to make sure that the curves I do trust on the second version of the test are telling the same story as the first version of the test, and I think that is indeed the case if I compare the plots. In the second version, comparing the dark red/brown points and curves (bulge size for bulge-dominated galaxies) it looks like the points are a bit above the curves for 0.5<z<1, by ~20-30%? And in the first version, at the same redshift range, the red points (bulge size) are similarly above the red curves. The match is better at high redshift in both cases. And the disk curves/points in both versions of the test mostly agree within that ~20% tolerance from what I can see, with some more tension for the highest luminosities. I believe this is the tension you remarked on for high-luminosity disks -- but weak lensers don't care that much about the rare high-luminosity disks, they care more about the more numerous ones at intermediate luminosities. So I think these plots are telling us that the dominant population that we care about is in good shape. That could be the reason the van der Wel+ test shows agreement, since presumably it's also reflecting the fact that at typical luminosities disks dominate (and their sizes are fairly well-matched). Does this interpretation sound reasonable to you?

• I think we may agree on the challenges inherent in a comparison with these published results: the sample selections differ and the methodology for estimating and sometimes even defining sizes differs. In that sense, there will always be a question mark with any DESCQA test of these relations, which means our tolerance for disagreement should allow for tens of % effects.

• In my mind a much less uncertain test would be to produce images from the catalogs, and for a given seeing, compare the observed galaxy sizes for a fixed apparent magnitude and PSF size against those in HSC, using the same apparent size measurement algorithm (which we can do since HSC uses the LSST stack). By imposing the same selection in observed quantities and measurement algorithms, I believe we have a much less ambiguous test that can be interpreted a bit more strictly. The reason I haven't advocated for simply using that as our sole test is that (a) the DESCQA tests provide some input at an earlier stage of the catalog production process and (b) if we find a serious mismatch in the observed size vs. magnitude relation, we still don't know if it's in the intrinsic size vs. luminosity relation at fixed redshift or some redshift evolution effect, so we'd eventually have to dig into these quantities that we're testing in DESCQA anyway. Given the results you have here, I am optimistic that this DM catalog-level validation test on apparent size vs. magnitude will not reveal any surprises.

Again, apologies for the length of this message but I saw a few points that I wanted to address. Comments from others are welcome. Many thanks to @vvinuv @msimet @evevkovacs @yymao and others who have been working on the size tests.

from descqa.

@rmandelb - thanks for getting back to us with a thoughtful answer.

Hence the primary purpose of the validation test is to provide some input into your method of extrapolating the parameters of that relation to higher redshift, and that's why having a CANDELS (van der Wel+) or COSMOS (Mandelbaum+) comparison is important. Do you agree with this statement?

Oh yes, I very much agree we need higher redshift validation data in addition to low-redshift. In particular, and to be clear, in the protoDC2 model, we do evolve the size-luminosity relation at higher redshift: as z increases, we decrease the normalization of this scaling relation (higher redshift galaxies are more compact than their lower redshift counterparts). In previous versions of the catalog, there was no evolution, and the van der Wel+14 validation test looked considerably worse.

Based on my experience, if you have bulge-dominated galaxies in real data with low to moderate S/N (anything less than ~200), you should not try to use the disk sizes for validation tests...So in a validation test that uses bulge+disk fits, I only tend to use the bulge parameters for bulge-dominated galaxies, and disk parameters for disk-dominated galaxies.

Yes, exactly, exactly. I similarly advocate for entirely ignoring the sizes of bulges in disk-dominated systems, and also the sizes of disks in bulge-dominated systems.

... So I think these plots are telling us that the dominant population that we care about is in good shape... Does this interpretation sound reasonable to you?

Yes, I agree that the two tests give a consistent story, and that this story is indeed telling us that the bulk of the population has reasonable sizes. I also certainly acknowledge that there is room for improvement: based on this back-and-forth, I have already tweaked the bulge normalization to be ~30% smaller. But even without this change, I think we are on the same page that the current version is sufficient for cosmoDC2 production purposes.

Thanks again to @rmandelb for weighing in clearly and carefully, and echoing thanks to @vvinuv @msimet @evevkovacs @yymao for contributing to this important validation criteria.

from descqa.

@yymao The most recent run of this test from the master branch here appears to have some issues. The bottom set of plots is missing for one thing. Thanks

from descqa.

@evevkovacs I believe that's a feature... the code always creates 6 panels but only 3 are used for size_Mandelbaum2014_BD (but 6 are all used in size_vanderWel2014_SM_Lum). See these configs:

https://github.com/LSSTDESC/descqa/blob/master/descqa/configs/size_Mandelbaum2014_BD.yaml#L20
https://github.com/LSSTDESC/descqa/blob/master/descqa/configs/size_vanderWel2014_SM_Lum.yaml#L25

But yes, the code should create the number of panels according to z_bins. (cc @vvinuv)

from descqa.

Actually, they are also blank in size_vanderWel2014_SM_Lum see here

from descqa.

from descqa.

@evevkovacs have you run them on buzzard_test? Since protoDC2 only goes to z=1 is not surprising the lower panels are empty...

from descqa.

from descqa.

from descqa.