?

Log in

No account? Create an account

The Toybox

people for the conservation of limited amounts of indignation


Previous Entry Share Next Entry
okay, we are in sga, therefore, we know of the scientific method
children of dune - leto 1
seperis
Thought. Random thought. Actually fairly undoable thought.

In another lj, there's a fascinating meta about correlation between comment number and how good a story is. I really wont' rehash that here, because it will only make me cite the stories I hate most that had high comment counts and send me into blind rages. It's a problem. I deal with it.

So I'm trying to figure out how would a true double blind work in fandom as it stands. The only way I can see that would level all playing fields--and even then, I'm talking a severe difference in level, but close enough--would be a double blind. Anonymous authors, screened comments--and a single writing prompt. Because while I buy that quality of fic has something to do with quantity of feedback--I think it's not as much as we--and I mean, me, the writer--always hopes it will be.

Okay, just thinking. A double blind, if you wanted to test the hypothesis -- a fic with a lot of feedback is (usually) better or at least far more publicly accessible than one that has a lower one. What would be the constants?


My first question would be, how would you define "better"? Because you'd need an objective, quantifiable measure of goodness in fiction, wouldn't you?

Better as defined in this particular instance is writing, stripped of author name, in single pairing or gen format, where the entirety of the story rests on the authors' ability to tell the story well.

Now in that, we'd be left with style preferences, pov preferences, and even, with a single writing prompt, kink preferences.

See, that's why I'm not like, opening this as a challenge or anything. I can't even out all all the variables. For the purposes of this, it would have to come down to the ability of this story to appeal almost entirely on the basis of how the writer writes.

Would something like the anonymous_sga (I have no idea of the exact journals name, sorry) work as a test-case? The stories are posted anonymously, the authors revealed a few weeks later.

Also -- a double-blind involving fandom? I wish you so much luck in trying to achieve that because I have no idea. I think we have too many ways of communicating, not just lj but through chat programs and email and the phone... there are too many avenues to invalidate it.

It *might*--but we can still see the number of commetns, and that's anotehr form of influence on readership. For purity, it would have to also be entirely screened.

And exactly. I'd have to leave at least part of it purely up to author honesty in not communicating to anyone which story they were doing.

What would be the constants?

Fixed length, third person limited, past tense, genfic in a fixed fandom with pre-specified characters, using authors who have both written that and had good feedback before.

Although, how do you define better?

In this particular instance--adn only in this particular instnace, not generalized--the story appealing to the largest number of people totally resting on the way the author wrote the story.

the sga_santa is almost like a double blind in a way because no one knows who is posting what and you can kind of see which stories appealed to the most readers based on the feedback.

Right. And that's a variable--does the number of comments already received influence readership? So it would have to remain screened.

Hmm. I agree with previous posters that if your goal is to see whether more comments = better fic, you would need to define what you mean by "better," and that achieving double-blind is pretty much impossible, because there would be no way at all of knowing whether one or more authors had told other people which story was theirs, and if those people had told people, etc., etc.

I like the idea of what you're trying to accomplish, though. It would be really nice to definitively determine how much impact such things as name recognition, popularity, hot-buttons in fic content, etc., really have. But how can you ever be certain that you've filtered out the name recognition and popularity?

In this case only--and for the purposes of my imaginary double blind--it would totally rest on the author's ability to tell the story and the widest number of people who enjoyed it. But only based on that.

because it will only make me cite the stories I hate most that had high comment counts and send me into blind rages. It's a problem. I deal with it.

*nods verrrrrrrrrrrrrrry enthusiastically*

also, good is kind of subjective. Even among people who read a lot there are sometimes disagreements on the quality of the story.

yes, it is.

but for teh purposes of this particular exercise, removing all other variables, and assumign if someone liked the story, they'd comment--and this is a lot of assumptions--the 'success' of the story would rest solely on teh authors' ability to tell it. Basically, on *them*. So better--in this case, and only this case--would be based on how broadly it appealed.

Here's a problem--in order to approximate the typical behavior of readers/commenters, they would have to not know that there was any kind of comment-count thing going on, or they might comment/not comment more often than they typically would.

And, yeah, who would judge "better"? You couldn't go by comment count, of course.

Actually, in my imaginary double blind, comments *would*.

Okay, the base idea is this--do higher comment counts equal to a better story y/n? We don't know because of the variables--author name, author following, pairing, heck, plotline or kink, the number of comments already received, who recced it. All of these things are influence. So on an ideal double blind--and God knows it would have to be ideal--all of these things are stripped out. No author name, a single story prompt, and no way to see the comments. What would be left is the authors' ability to tell that particualr story, whatever it might be. So--and this is hugely theoretical--people would comment more favorably the broder the appeal of that writer's story, or--and only for the purposes of this particular setup, not in any way in general--and the broader the appeal, theoretically more comments. So we could take from that at very least that a particular author is better *to more people* than to others, which for the purposes of this, would be a better story.

(no subject) (Anonymous) Expand
Oh, good idea! You're gonna run this, right? I want to play!

*casts mind back to setting up studies...*

The order that the fics are in should be randomised and different for every refresh. And they should probably be stripped of all identifying info until after you hit the link - including title and number of comments so far. So that every piece gets a similar hit count.

The fics should be posted in one go - multi chapter wips seem to get more support, and I'm not going to get into the tactics thereof.

Ideally they should be the same length and equally easy-on-the-eyes, but how would you enforce that? It could be a fun extra variable to play with in the stats. No, I do not get any kind of sick pleasure from playing with stats.

Okay. Maybe once.

Also, ideally the commenters shouldn't know they're participating in a study. But again, how?!?

As an interesting aside, if you do run it, can you set up hit counters (that the reader can't see or know about) so we can see whether there's a difference in hit-rate and comment-rate and how that varies with the story quality? That'd be kinda fun to know.

And, uh, obviously people should be able to rate the stories so there's something to analyse there.

Wow, I just came over as a complete stats weirdo, didn't I? Am actually a bit alarmed by how interested I am in this. Really, I'm alarmed. I have almost no idea how you'd institute it. Leave me alone.


but, um, I'm available for brainstorming

I suppose I'm suggesting you need a programmer on board to make up a proper database, fully equipped with randomising and secret comments and counters, and where you're asked to rate the story when you cross off its window.
I kinda wish I a) could program and b) had a spare week!

Als, I agree with other commenters who suggest the pairing/prompt should be constants, but if it's a ratio of hit:comment that you're really wondering about then it's not strictly neccessary. If it's the relationship between comments:quality rating or quality:comment then the actual number of hits shouldn't matter.

I think about this A LOT, and it's hard, because I really want to write for myself and not care what other people think, or at least not what the masses think, per se, but I'm not always successful. I mean, on some level, we are writing for other people's enjoyment, on some level, and it kind of sucks to spend time on something and not know if people liked it. And comments, in some ways unfortunately, are the only way to TELL if people read and liked your fic.

I will admit to looking at comment counts, especially in fandoms like SGA, because there is SO much fic, and the quality, unless you know the author, can be somewhat hit or miss, so I sometimes look to see if it's been commented on a lot, if I don't have a rec, to know whether or not I should read it. I kind of hate doing that, but I don't have the time to read everything and decide for myself. I know I'm probably missing out on some good stuff as a result, but I don't know what else to do.

On the flip side, I have had a fic in SGA that has gotten LOTS of comments. My very first fic, actually, but subsequently, the comments have been sparser, for fics that *I* think are much, much better than the first one. I'm not sure what to make of that. Is there any way past that? Or around it? I try not to get too worked up about comments, but it bums me out that someone could miss something that I'm really proud of because, for whatever reason, it didn't get a lot of comments.

I think you're right - the only way around this issue is to get rid of comments entirely, and I'm not sure that's the best thing. I mean, I know that I treasure all of the feedback I get on my writing. So I wouldn't want to stop that. Maybe the solution is for people who DO read a lot to rec fics that are outside of the mainstream. I mean, there are certain authors in every fandom who, when they post, will get LOADS of comments and praise and recs. Without fail. I generally don't comment to those people, but try to comment to those who I know aren't going to get a lot. And I try to rec those people too. Maybe that's the answer. I don't know.

*ponders*

I admit to being intrigued by something where the pairing was constant, maybe, but everything else was up for interpretation, so it couldn't be written off as a pairing preference.

And then anonymous and, yes, screen comments, and see what happens.

I think it might fall out like it does now, but there would be some surprises.

Tangentially, I was talking with someone about the problem of concrit: no way to separate those who sincerely want it from those who want to hear, "no, don't change a thing!" and consequently, concrit is basically only given by those with some sort of social disability or people with an axe to grind. I was wondering if some sort of double blind set-up might be the solution there, too.

Now *that* would be interesting.

I am just here to laugh myself silly at your subject line.

We can all find it in wikipedia, anyway. *g*

They'd have to all be approximately the same length, and short enough to be read immediately.

Same fandom, same primary character, approximately the same rating.

It would probably help if there was a small team of betas (who are not allowed to comment) who beta'd all the stories. This can be in addition to the author's preferred beta(s). The official team would be there to make sure all the authors have a minimum standard of spelling, grammar, and tense/pov consistency. They can also help the authors tone down any serious 'tells' in their stories.

The story would have to have several basic things the same: mckay/sheppard, first time, 7K-10K words, non-earthside, non-AU. That way there would be room for some plot (and all stories would have to have some plot), world building would be limited to alien cultures and would not be domestic culture. I think that you might get something like what you are interested looking at last year's Back-to-Basics challenge:

http://community.livejournal.com/atlantisbasics/3812.html


Mmm. See, I like that. That would actually be perfect.

There's also the consideration of who's posting it and where. You'll get a different group of readers/commenters/lurkers depending on your source.

True. So it'd have to be defintiely posted somewhere easily accessible for all of lj part of sga fandom.

*thoughtful* Hmm.