HELP! in Determining A Sample Size

[RedHot25]

Hi Everyone!

Hoping that someone can help me out with this.

I am looking to send out a survey to 3 provinces (each of their respective professional associations). I don't have the survey specifically developed yet, but it will be all (or super close to all) quantitative in nature.

My question is how do I determine my sample size? I want to take a simple random sample from each of the 3 (I think that is still called simple random sampling?). Do I take the same number from each one, or do I somehow scale it? 1 org has approx 1600 members, another 1100 or so, and another 5400 or so!

Cruising around online, I have found one site that, depending on which scenario I put in, would sample each province for approx. 365 - 370 or so; or, it would sample the big one for approx 359, and the other two for around 310 and 285 respectively?

I'm looking for a "justifiable" type number...I know things may not be perfect, but something that is workable/passable at the momment.

Thanks for all help...a stats saviour would be truly appreciated !!!!

[AaronSJ]

I'm not a stats saviour, but I have some experience in this area. You can approach this in a couple of ways -- the statistical approach is to determine your confidence interval and then send out an appropriate number of surveys to achieve that result based on an expected response rate (e.g. 1% for unsolicited surveys, maybe 10-25% for professional associations). Just by eyeballing the numbers, if you want a high confidence interval (say 95%), you'll probably have to send a survey to almost everyone... And this may be the only option if you will be publishing your results, running a study, etc.

However, if you're approaching it from a business point-of-view, perhaps the first thing to consider is your resources (your budget, the logistics of sending & receiving 1000+ surveys, etc.) What's the maximum number of surveys that you can send? If it's an electronic survey, is there an issue with sending one to each member?

The biggest problem you'll face is estimating your response rate -- and from experience, be conservative. Have similar surveys been done in the past? If so, that can help. Is there a number of completed surveys that would make you comfortable with the results?

I've found that it is very difficult to survey customers/stakeholders with an expectation that you'll receive a statistically significant, evenly distributed response. The approach that many companies use is to survey as many people as possible... and hope for the best. Is that an option?

[RedHot25]

Thanks Aaron!

Well, financial resources might be a bit of a challenge to be able to survey everybody. Also time may be a factor. Also, some people have mentioned to me that the scope of this project may be at a higher level (phd) than the level (masters) that I am at...

That said, I am thinking about choosing 2 provinces' associations - the 5400 one and the 1600 one. There has been some studies done, that I could look at their response rates.

My main question then for each of the 2 associations is (at a basic level) how much do I sample from them? Do I sample the same amount, or a percentage level? I.e. should my sample #s for the 5400 one be higher than the 1600 one?

If I took a 50% sample of each one... 800 and (approx) 2500...ouch, that's a lot!

[AaronSJ]

Originally Posted by RedHot25

My main question then for each of the 2 associations is (at a basic level) how much do I sample from them? Do I sample the same amount, or a percentage level? I.e. should my sample #s for the 5400 one be higher than the 1600 one? If I took a 50% sample of each one... 800 and (approx) 2500...ouch, that's a lot!

I guess the answer depends on how important it is for you to have a statistically significant sample for each group. If you want to be accurate within +/- 5% at a .95 confidence level for each group, you'll need 310 responses from the group of 1600, and 360 from the group of 5400. If you're expecting a 25% reponse rate, you'll need to survey 1240 of the 1600, and 1440 of the 5400... which obviously is a lot. If you're happy with a +/- 10% confidence interval (obviously weaker), you could get away with ~100 answers from each group, or ~400 surveys to each.

If you're looking for the best data possible, given your limitations, I'd just pick a % from each group. Whether you choose the same % from each group is up to you - you could survey 10% of the larger group and 20% of the smaller group. However, you will need to be able to justify this when you present your findings (defend your thesis, etc.), and that could be a problem if you don't have a logical reason for doing so... and you can expect stats geeks to pick up on this (attacking the methodology is the easiest way to disprove findings).

One final thought -- have you considered an electronic (e-mail or web-based) survey? If you have valid e-mail addresses available, this is the most efficient way to collect your data as you can survey the entire population and expect a similar response rate to a written survey, without having the sample size questioned. There are companies that specialize in management of such surveys... they aren't cheap, but it might be comparable to the cost of printing & mailing 1000+ surveys... just a thought.

[flmsfan2004]

I wish you the best of luck. I'm currently finishing up my honours thesis, I conducted an online survey of university undergrads last semsester. At times I feel overwhelmed with my 155 respondent set of data. I could not even imagine having more! In my opinion, doing the survey online is the way to go, especially if your survey is quantative, it makes the data analysis so easy.
Here are a few links that may help you out
http://davidmlane.com/hyperstat/index.html
http://www.tufts.edu/%7Egdallal/LHSP.HTM

[DeanOMac]

I just worked on this in school not to long ago. Aaron has explained it pretty good. You first determine how strong you want your data to be, most go around 95%. Once you have done that, you then determine n (your sample size).
You can approach it a few different ways as well. The 3 that immediately come to mind are: Stratified Sampling, Cluster Sampling or a Simple Random Sample. Stratified is when they are in groups (as they are already) and you select a portion from EACH group. Cluster may not be the best here because you are suppose to randomly select THE GROUPS you will draw from (must be random) and then take a simple random sample from each group. Simple Random Sample is if you just put all into one group not caring what area they are from, and selecting whatever is decided to be your ideal sample size.
Cluster and Stratified should yield more precise (or at the worst, same precision) as a SRS.

Depending on which way you go there are different numbers you would use from each group. Sorry I didn't give you any set numbers but it really does depend.

[RedHot25]

Excellent!

Thanks so much guys (and gals?). I truly do appreciate the help, its nice to be able to bounce things off of people or hear other feedback re: stats. Especially when you are in a field where stats and quantitative stuff is not exactly the big thing shall we say .

Aaron, I can't say enough, you have been a BIG help. Thank you for that!

FF - ha, thanks for the "encouragement" . Anyways, those links are awesome...

DeanOMac - thanks for the input. I have chosen the 2 prof. associations myself (i.e. I chose province 'X' and province 'Y') to take a sample from. From these two provinces, I am going to take a simple random sample (I think roughly following Aaron's great info!). That said, could I still call it "simple random sampling", as long as I denote that provinces 'X' and 'Y' are my two sampling cohorts? Or am what I doing technically called stratified random sampling? I know its splicing hairs, but....

Thoughts?

[DeanOMac]

If I understand correctly, you are doing Cluster sampling actually, stratified you take a SRS from each strata where as you are only doing 2/3.
You had 3 groups and you randomly selected 2 groups to survey (even if you didn't, you should say you did). The problem with cluster in this case, is you only have 3 groups (clusters) and if their are regional differences within the 3, you won't get accurate results. Assuming you pick a proper sample size from each cluster and say the clusters were picked randomly, the only criticsm I can think of is: "Do you have evidence to suggest that Province X would have similar results to province Y and Province Z. Otherwise you are putting to much weight on X and Y's opinions but none on Z and they would almost surely have different opinions since they are from a different region.")

In other words, would all provinces (clusters) have similar results OVERALL. Can you say ALTA, ONT and NFLD's (just made these up, I don't know which ones you are using) professional associations will have similar opinions. That is the only setback I can see. If you have reason to believe yes and can explain your way out of it, I say what you are doing is fine.

[RedHot25]

Originally Posted by DeanOMac

If I understand correctly, you are doing Cluster sampling actually, stratified you take a SRS from each strata where as you are only doing 2/3.
You had 3 groups and you randomly selected 2 groups to survey (even if you didn't, you should say you did). ....

In other words, would all provinces (clusters) have similar results OVERALL. Can you say ALTA, ONT and NFLD's (just made these up, I don't know which ones you are using) professional associations will have similar opinions. That is the only setback I can see. If you have reason to believe yes and can explain your way out of it, I say what you are doing is fine.

Hey Dean,

Thanks for the info. I don't think I am doing cluster sampling. I have chosen to compare two provinces (let's say ALTA and ONT)...this is my own decision (lets say b/c I lived in each one, for example, and that they have not been studied in this way as another example). Sooo....I am taking ALTA and ONT's 2 professional associations, and then from each prof. association, I am taking 1 simple random sample of members from Alta's prof assoc, and 1 simple random sample from Ont's prof assocation...

(i am choosing a specific prof association b/c it is the same one i am in...i.e. they are, just for e.g., the alta association of accountants and the ont association of accountants).

So, in sum....Redhot25 is choosing 2 associations in total: the Alta assoc. of accountants and the ont association of accountants b/c he has lived/worked in both provinces and they have not been studied in this way before, etc. From the Alta association comes 1 random sample of members, from the Ont association another random sample of members. And ta da, we have my study !

[DeanOMac]

Ok, I clearly did misunderstand. I thought you were going to select 2 groups to make generalizations about all 3 groups. If you are doing a head-to-head comparison between two of them, your methodology is great. I think it should work out great for you, although non-response can and usually is VERY annoying, but aside from a few tricks which make only little difference, not much you can do. Good luck, and if you need any more help don't hesitate to ask.

Originally Posted by RedHot25

Hey Dean,

Thanks for the info. I don't think I am doing cluster sampling. I have chosen to compare two provinces (let's say ALTA and ONT)...this is my own decision (lets say b/c I lived in each one, for example, and that they have not been studied in this way as another example). Sooo....I am taking ALTA and ONT's 2 professional associations, and then from each prof. association, I am taking 1 simple random sample of members from Alta's prof assoc, and 1 simple random sample from Ont's prof assocation...

(i am choosing a specific prof association b/c it is the same one i am in...i.e. they are, just for e.g., the alta association of accountants and the ont association of accountants).

So, in sum....Redhot25 is choosing 2 associations in total: the Alta assoc. of accountants and the ont association of accountants b/c he has lived/worked in both provinces and they have not been studied in this way before, etc. From the Alta association comes 1 random sample of members, from the Ont association another random sample of members. And ta da, we have my study !

[RedHot25]

Thanks to everyone that has replied in this thread. I really appreciate the extremely thoughtful answers, responses, comments and questions...

[hockeyfan99]

seriously...this is not a hockey forum....

[RedHot25]

Originally Posted by hockeyfan99

seriously...this is not a hockey forum....

Nope, not this part of it. This is the Off-Topic part....