Content Manager Guidelines for WOK question Creation Handbook

This is an extended guideline for the art of creating great questions in WOK (see a shorter version: Guidelines for question creation). It´s a general guideline to be used for teachers, administrators, and wokers who voluntarily like to contribute to creating the Worlds largest database of multiple choice questions on top of Wikipedia articles.

2.1 THE INPUT PROCESS
Questions are entered into the database using 'Drama', a browser-based program that allows questions to be entered into specific categories and subjects, then verified.
The team is assigned a subject, which the Team Leader will then break down into smaller parts for each Questioner to research.  For a film subject, it may be broken down alphabetically, with one questioner researching A to E, the next F to J, and so on.  Alternatively, a history subject may be broken down into decades or centuries.

The team leader will suggest useful websites to start with, but each team member's research will produce more.  Once the Questioner has found a good-quality website that can be used to generate entertaining and accurate questions, they are entered into the database as follows:
The question is entered into the Question field.

• The correct answer is copied into the Correct Answer field.
• The URL, or web address, of the source website, is copied into the first URL field.

• The Questioneer uses Google (or other sources) to find a second website to confirm that both the question and correct answer are accurate.

• The URL of the second source website is copied into the second URL field.
• Using the research, the SIX incorrect answers are selected and entered.  They do not need to be sourced but should be copied and pasted where possible.

• The difficulty setting of the question is set.

• The Market(s) of the question is set.

• The whole question, including sources and answers, is checked for spelling, grammatical or factual errors.
• Save the question.

The question is sent to validation (see The Validation Process)

2.2 THE VALIDATION PROCESS
The validator will check the following:

·       the accuracy of the question by reviewing the source
·       the grammar
·       the spelling
·       the difficulty level
·       the general quality of the question
·       the Market
 
If there are any mistakes, the validator will correct, if possible. If it is not possible to correct a question, for any reason, the question will be discarded. The validator will record the creator of any discarded question.
The validator will provide the Team Leader with a complete list of all discarded questions with the names of the questioner responsible for creating the question.
All questions that pass the validation process will be upgraded to classes. 

Official Sites other than Wikipedia for input and fact checking
Usually, the input process requires a second source to confirm the accuracy of a website.  However, if the first source is 'official', then only one is required.
In most cases, it is obvious that a website is 'official'-

e.g. www.thesimpsons.com, www.arsenal.com
Regarding TV or Movies, often there are several 'official' sites, including internal TV network sites-

e.g. www.scifi.com/battlestar, abc.go.com/prmetime/desperate/
Though these sites do not follow the www.'title'.com style, they may still be official.

Although the above examples may not follow the www.'title'.com

If there is ANY doubt regarding the 'official' status of a website, you MUST use a second source.  In many cases a website may be professionally designed, but cannot be considered an official source-
e.g. www.tv.com

Regarding Wikipedia, Answers.com etc, the information on these websites is almost certainly accurate, but they are NOT 'official'.  They can still be used as a source, often it is very useful to do so, but due to the third-party nature of such sites, they cannot be official.  Also, because both Wikipedia and Answers.com often retrieve their information from the same sources, they cannot BOTH be used as the two required sources.

There are many sites that may contain useful information or confirm a source but are either too unreliable to use as a source-
e.g. www.amazon.com

Or are simply too amateurish, and though this is a matter of discretion, it is also fairly obvious whether a website is the demented scrawl of a single madman or not-
e.g. www.iloverichiecunningham.co.uk
 
2.4 QUESTION DIFFICULTY

We require questions of mixed levels of difficulty. Assessing and adjusting the difficulty of a question is mostly a matter of intuition.  There is no easy method to determine the inherent difficulty of any given question; it is a matter of phrasing, of how much information is given in the question and the variety of the answers.

The following factors are some of the methods by which difficulty can be established and adjusted.

Phrasing

Questions should NEVER be written in a more complicated way in order to make them more difficult.  They should always be as SIMPLE as possible.

However, the question can be made easier by including more information in the question.  Adding a phrase like 'In 1977' to a film question can give the player a clue about the era, who were the famous film stars of the time etc, so that even if the player hasn't seen the film, they have a better chance of answering it correctly.

Similarly, use of adjectives like 'sci-fi' or 'comedy' for films, or more precise terms like 'striker' instead of 'player' for sports, can clarify the question, and therefore make it easier.  Also, more factual information about the question's context can be added, such as the author of a novel.

e.g. In which 19^th Century Russian novel does 'Konstantin Levin' appear
OR
In which novel by Leo Tolstoy does 'Konstantin Levin' appear

IS EASIER THAN

In which novel does the character 'Konstantin Levin' appear

Incorrect Answers
Good  incorrect answers are as important as any other component of question creation.  They are also often the most difficult stage.  In addition to determining the quality of a question, the answers can help determine the difficulty.

A question with numerical answers can often be made easier by spacing the answers further apart -
e.g. 1985, 1785, 1945, 1845
OR
20^th Century, 19^th Century, 18^th Century, 17^th Century

IS EASIER THAN
1985, 1986, 1987, 1988

A question with text-based answers can be made easier by either selecting correct answers from a broader pool, such as using all 19^th novel titles as a pool instead of just Dickens' titles -
e.g. Great Expectations, Middlemarch, Vanity Fair, Jude the Obscure

IS EASIER THAN
Great Expectations, Little Dorrit, The Pickwick Papers, A Christmas Carol

Or using variations on the correct answer -
e.g. Great Expectations, Great Celebrations, Great Experiences, Great Catastrophes

HOWEVER, using variations can very often make the question too easy, which can be more damaging to the quality of the play-experience than if questions were too hard.  Easy questions are often harder to write than hard questions for this reason; the difficulty must be easy, but NOT too easy.

e.g. Johnny Depp, Jamie Depp, Tommy Depp, Gary Depp
NOT
Johnny Depp, Jahnny Dopp, Jihnny Dipp, Juhnny Dupp
 
2.5 MARKET

Simply, these tags determine the suitable 'markets' or geographical/cultural areas a question is suitable for.  Most questions are suitable for all markets, BUT there are exceptions, and assigning the correct markets to these questions is as VITAL to ensuring question quality as any other stage of the process.

e.g. A Music question about the Billboard charts is ONLY suitable for USA.

Likewise, a Music question about the UK charts is ONLY suitable for UK.

The correct market(s) for any exceptions should be obvious.  If you are in ANY doubt, ask your Team Leader and only assign the specific markets you are certain are correct.

3. QUESTION STYLE
 
3.1 WHO or WHOM?
Who should always be used in preference to Whom. This is NOT because Who is always right, but  because our questions should be phrased to avoid having to use Whom.
e.g. In 1793, who invented the cotton gin?
NOT
In 1793, the cotton gin was invented by whom?

Even though the latter example is also grammatically correct, Whom can be seen as pretentious.  There is no reason to alienate players when the question can be easily rewritten.  The following is incorrect on both counts -
e.g. NEVER In 1793, the cotton gin was invented by who

3.2 WHAT or WHICH?
What should usually only be used when the pool of possible answers is either infinite -

e.g. What is the number of loaves in a baker's dozen

unknown -
e.g. What was name of President Reagan's dog

or at least very large -
e.g. What colour is the ugli fruit

Which should usually be used when the pool of possible answers is limited.  As we offer 4 choices for each question, Which is almost always the right choice in a “what or which” debate.

e.g. Which character does actor Zach Braff play in 'Scrubs'

In these cases, Which can often be followed by 'of the following'.  It can unnecessarily lengthen a question, but it may also add clarity.
e.g. Which of the following characters, played by Zach Braff, appear in 'Scrubs'

In Which
For items than 'contain' the correct answer, such as centuries, years, TV shows, In which is almost always better than What.

e.g. In which century was explorer Christopher Columbus born
NOT
What century was Christopher Columbus born

In which TV sitcom did the character 'Frasier Crane' appear
OR
Which TV sitcom featured the character 'Frasier Crane'
NOT
What TV show featured the character 'Frasier Crane'

The disadvantage of using In which is that it forces the verb associated with the preposition In to the end of the question; in the above example, 'appear'.  Using Which usually means the object of the sentence, i.e. the distinctive part of the question (in the above example 'Frasier Crane'), is at the very end of the question.  This focuses the player on the relevant part and should make the question slightly easier to understand.
 
3.3 IN or ON?
The distinction between these two prepositions is usually fairly obvious.  In indicates that one element is enclosed within another -
e.g. The city of Seattle is in which country
NOT
The city of Seattle is on which country

On, unsurprisingly enough, indicates that one element is positioned above another -
e.g. On which island did Captain James Cook die
NOT
In which island did Captain Cook die

Regarding question input, the 'in or on' debate usually depends either on the literal relations of two tangible elements -
e.g. On which street would you find 'Big Bird' and 'Elmo'
or on the divisibility of the object -
e.g. In Britain, On which day of the week does May Day lie

This is an inaccurate example, but if the object is small, or not usually broken into smaller parts (in the context of the question) then 'On' should be used -

e.g. On which island, On which day

However, if the object is large, or is considered a group name for a collection of elements usually separate, In should be used -
e.g. In which year was Christopher Columbus born 

In which continent is the city of Seattle located
 
3.4 CAPITALISATION

Capital letters should always be used for the first letter of each question.  The first letter of a quote should usually also be capitalised.

The titles of books, films, TV shows etc, should have a capital letter for each noun, as should the first article, and only if it is the first word of the title, 'the', 'a' etc -

e.g. The Dirty Dozen, Bridge Over the River Kwai

Note that in the second example, 'the' is NOT capitalized.
The pronoun 'I' should always be capitalized.
 
Proper nouns

The name(s) of a person, TV show, city etc should always be capitalized -
e.g. David, The Good Life, New York

The names of months, days are also capitalized, as are national-group words -
e.g. March, Friday, Irish, American

Deities are capitalized.  However, 'god' referring to ancient religions, such as Greek or Roman, is NOT capitalized, nor is the plural 'gods' -
e.g. God, Allah, Lord

Common nouns
The names of objects, animals etc int the question text only are not capitalized -
e.g. hammerhead shark NOT Hammerhead Shark

Adjectives
If the adjective is based on a proper noun, it may have a capital letter.  This varies from word to word -
e.g. 'Shakespearean' is capitalized, 'quixotic' is not.

Acronyms
Usually, these are all capitalized -
e.g. BBC

Occasionally an acronym may have become a bonafide word, and will only have the first letter capitalized like a proper noun.  In these cases, either style is acceptable -
e.g. Unicef or UNICEF

Titles
An individual's title, such as Mr, Sir, King, President will always be capitalized if it appears before their name, or if it used to address them instead of a name -
e.g. Mother Teresa, Sir Walter Raleigh, Who sang 'Happy Birthday Mr President'

However, if the title appears elsewhere, it is not capitalized -
e.g. Who was the 42^nd president of the United States of America

Exceptions
Brands are usually capitalized, such as Coca-Cola, but occasionally they are not -
e.g. eBay, iPod


4. PUNCTUATION
4.1 THE PERIOD
The question input process rarely, if ever, requires a period or full stop at the end of a sentence.

Where appropriate, a period should be used at the end of a quote, inside the quotation marks- e.g. Who famously said 'God does not play dice with the universe?'

Abbreviations
Periods or full stops should not be used in acronyms -
e.g. BBC not B.B.C.

In most cases, periods do not need to be used for titles -
e.g. Mr NOT Mr.

There will almost certainly be exceptions to this.

4.2 THE COMMA
The comma should be used to separate the clauses of a question -
e.g. The character 'Frasier Crane', played by Kelsey Grammer, appeared in which sitcom
 
Often the clause that the commas surround, 'played by Kelsey Grammer', can be removed and the question will still make sense.
e.g. The character 'Frasier Crane' appeared in which sitcom?

In the above example, using 'played by' improves the quality of the question, but perhaps makes it slightly easier.
The comma should also be used to create a pause, particularly in a two-part question, or to separate a statement that clarifies the question, such as defining the release year of a film -

e.g. In 1977, who directed the movie 'Star Wars?
NOT
In 1977 who directed the movie 'Star Wars?

It can be helpful to read the question slowly and sense where a pause may be.
It is unlikely that you would use more than one comma if you were not including a clarification as above, 'played by Kelsey Grammer', but the situation may arise.
e.g.  In the Summer of 1969, if you were listening to The Who, Janis Joplin and Jimi Hendrix live, where would you be

As in the above example, commas should also be used to record lists.  This, however, will rarely be relevant for question input.

4.3 THE QUOTATION MARK
Always use 'text' NOT “text”.  The single ' is less messy and obtrusive than “ and requires less effort to type.
Quotation marks should usually only be used in the question, NOT in the answers.
Use quotation marks for titles of TV shows, films, art etc, and nicknames or character names -
e.g. 'The Good Life', 'John Dorian'
Never use them for real names of people, places etc -
e.g. Ted Danson NOT 'Ted Danson'
 
Ambiguous Usage
The name of a company does not need quotation marks, such as Ford, but their products may -
e.g. either 'Mondeo' or Mondeo

Examples
Which of these characters, played by Ted Danson, appeared in 'Cheers'
NOT
Which of these characters, played by 'Ted Danson', appeared in Cheers

On the TV show 'Scrubs', who plays 'John Dorian'
NOT
On the TV show Scrubs, who plays John Dorian

4.4 QUOTES
Quotes should always be recorded as they are found in a webpage, book, etc even if the spelling, grammar etc, contradicts either English rules or these guidelines -
e.g. 'We wuz robbed' should NOT be corrected to 'We were robbed'
As discussed elsewhere, quotes should be indicated by 'text' NOT “text”.


5. WORD USAGE
5.1 Who created/Who made
When writing questions about art, TV shows, movies, 'who created' NOT 'who made' should usually be used.
e.g. Who created the artwork titled 'The Birth of Venus'
NOT
e.g. Who made the artwork titled 'The Birth of Venus'

When writing questions about manufactured items, such as cars, 'who made' can be used, but 'who manufactured' or 'who built' would usually be better.
e.g. Who manufactured the 'Mondeo' model of car
NOT
e.g. Who made the 'Mondeo' model of car

5.2 Starred in
Starred in should only be used in reference to actors and NEVER characters -
e.g. Mark Hamill starred in which classic sci-fi film
NOT
Luke Skywalker starred in which classic sci-fi film
Regarding characters, 'appeared in' or 'featured in' should be used instead.

5.3 TV
In all cases, when referring to television, use capitalized 'TV' NOT lower-case 'tv'.

5.4 Art
Questions on this subject can be difficult to phrase.  As stated elsewhere, 'who created' should be used NOT 'who made' -
e.g.
Who created the sculpture titled 'The Thinker'
Using 'the sculpture', 'the painting' etc can make the question easier and/or more interesting, but often using simply 'the artwork' or 'the work' will suffice -
e.g. Who created the artwork titled 'Starry Night'
When writing these art questions, use 'the artwork titled' rather than 'the artwork called' or simply 'the artwork'.  It seems a less awkward construction.
Use of terms like 'the portrait', 'the impressionist work' etc can make the question even more interesting, which is our goal, but they should only be used if the sources confirm that they are accurate.
e.g. Who painted the impressionist work titled 'La Femme à la Robe Verte'

5.5 Days, Months and Years
Months and days of the week should always be capitalized.
A precise date should be written in the form '25^th May 1977'.
e.g. Which famous sci-fi film premièred on 25^th May 1977?

In most cases, the year should be written in the full four-digit form. If the year follows a film, TV show etc, the year should be bracketed -
e.g. Which of these actors starred in 'The Rocketeer' (1991)
NOT
Which of these actors starred in 'The Rocketeer' (91)

If the two-digit form of a year is required, an apostrophe should precede the number -
e.g. 'Italia '90' was the title of which major sports event
Centuries should be written with 'Century' capitalized -
e.g. Which of these 20^th Century conflicts was fought between China and Japan?

6. TENSES
For the sake of argument, questions will usually only be written in either present simple/continuous or past historic/continuous tense.
The appropriate tense of a question is dependent on its subject.  In most cases, the right tense to use will be obvious.  It may even be irrelevant, as either past or present tense can be used.  One is usually better than the other -
e.g. 'The Thinker' was created by which artist
OR
'The Thinker' is by which artist
For questions regarding -

- 'permanent' elements, such as locations, names, rules
- something that, once created, has a permanent existence, such as a painting or sculpture
- something that is fictional and exists 'outside' of normal time, such as a character in a TV show
the present tense should usually be used -

Examples
Geography -  What is the capital of Belgium?
Sport - How many substitutions may an NBA basketball team make during a game?
Art  - 'Campbell's Soup Can' is a work by which artist?
TV -  Which of the following characters appears in the sitcom 'Cheers'
Movies - The character 'Ellen Ripley' appears in which sci-fi film   

For questions regarding -
- past events, such as wars, births, deaths etc
- the act of creation of something permanent, such as 'was created by', 'was founded in'
- the cast of a film or tv show, such as 'who starred in'

the past tense should usually be used -

Examples
History - When was India granted independence from the British Empire?
Sport - Who scored the first goal in the 2014 World Cup Final?
Art  'Campbell's Soup Can' was created by which artist?
TV - Which of the following actors starred in the sitcom 'Cheers'
Movies -In the sci-fi film 'Alien' (1979), which famous actress played 'Ellen Ripley'

Note the difference in tense between a present tense character question, 'who appears in', and a past tense cast question, 'who starred in'.

These are only guidelines. 
Determining tense can be simply a matter of question construction, but as stated in the examples above, some subjects have a better, or more appropriate, tense to use.  Provided the grammar and typography is correct, tense is less important.  Above all, the question must easy to understand.

Log version
Version 1.1 Updated Nov 2, 2017/ EB
Version 1.0 Updated Dec 20, 2016/ EB

Tags: guideline, Create Questions