Skip to Content View Previous Reports

Methodology

Methodology

The data for this study were collected in two parts. Much of the study is based on research conducted originally by other people or organizations. Other research, particularly the content analysis, is original work conducted specifically for this report.

For the data aggregated from other researchers, the Project took several steps. First, we tried to determine what data had been collected and by whom for the eight media sectors studied. We organized the data into the six primary areas of interest we wanted to examine: content, audience, economics, ownership, newsroom investment and public attitudes. For all data ultimately used, the Project sought and gained permission for their use.

Next, the Project studied the data closely to determine where elements reinforced each other and where there were apparent contradictions or gaps. In doing so, the Project endeavored to determine the value and validity of each data set. That in many cases involved going back to the sources who collected the research in the first place. Where data conflicted, we have included all relevant sources and tried to explain their differences, either in footnotes or in the narratives. For instance, the differences in online news use are probably explained by how survey questions were phrased — different answers emerge if the question is whether someone ever goes online for news, has done so in the last month or has done so in the last week.

In analyzing the data for each media sector, we sought insight from experts by having at least three outside readers for each sector chapter. Those readers raised questions, offered arguments and questioned data where they saw fit.

All sources are cited in footnotes or within the narrative, and listed alphabetically in a source bibliography. The data used in the report are also available in more complete tabular form online, where users can view the raw material, sort it on their own and make their own charts and graphs. Our goal was not only to organize the available material into a clear narrative, but to also collect all the public data on journalism in one usable place. In many cases, the Project paid for the use of the data.

For the original content analysis research conducted by the Project, the methodology follows.

Content Analysis Methodology

Three distinct categories of media were studied as a part of the 2006 PEJ Day in the Life of the News study — national, local, and Spanish-language media.

National media included national newspapers, national Web sites, network and cable television, and blogs. Local media included local television, local newspapers, local radio, local Web sites and blogs. Spanish-language media included newspapers and broadcast.

ADT Research conducted coding for national cable and network television. The University of Missouri School of Journalism Center for Advanced Social Research conducted coding for national newspapers and Web sites and national and local blogs. The School of Journalism at Michigan State University conducted coding of all local media except for local blogs and Web sites. Spanish-language media — first translated into English — was also coded by the team at Michigan .

Coding for all media except newspaper outlets was for May 11, 2005 . For newspapers, local and national, the May 12 editions were coded. In all, the study examined some 2,125 stories ( not including blogs): 102 in national newspapers, 100 online, 107 on network television, and 312 on cable news. The local media included  538 local TV stories,  420 on local radio, 425 in local print and 43 local online stories. Spanish-language media consisted of 78 (TV and newspaper) stories. In addition to that main analysis, 112 different postings — 104 national and 8 local — were coded under a different coding scheme.

I. National News Media

Sample, Story Procurement and Inclusion

To present a meaningful assessment of the national news content available to the public each day we studied a wide range of media outlets and programs.

Newspapers : the New York Times, the Los Angeles Times, USA Today. Stories were procured via hard copies of the publications, supplemented by the Nexis database.

All stories with distinct bylines that appeared on a particular newspaper’s front page, on the first page of the local/Metro section, and the first page of the business section were selected for analysis.

Network News :

ABC Today Show ( 7-8 a.m. )

CBS Early Show ( 7-8 a.m. )

NBC Good Morning America ( 7-8 a.m. )

ABC World News Tonight

CBS Evening News

NBC Nightly News

PBS NewsHour

The morning and evening broadcasts were procured through videotape, supplemented by transcripts. Videotaped programs were captured live in the Washington D.C. and New York markets. For evening newscasts, that meant each day’s 6:30 p.m. East Coast feed. PBS supplied the Project with the tape for the NewsHour.

In the mornings, the following content was analyzed: stories read by the newscaster in the half-hourly news blocks; feature and interview segments outside of the news blocks; banter between members of the anchor team whose import was other than to tease coming segments in that day’s program or to promote the network’s programming at some later time. Excluded from the analysis were the content of the weather blocks, local news inserts, commercials, and other content-free editorial matter such as logos, studio shots, openings and closings.

In the evenings the same rules applied, but because the content of those newscasts is less varied, concerns about news blocks, banter, weather blocks and local news inserts were not applicable.

Cable : To assess the nature of the 24-hour news on cable, CNN, Fox News and MSNBC were selected because they were the three most heavily viewed cable news channels in 2005. For each we selected a morning hour, a daytime hour, the closest thing to a traditional newscast and the highest rated prime-time talk show. The following programs were captured and analyzed:

7-8 a.m. on CNN, FOX and MSNBC

11 a.m. – Noon on CNN, FOX and MSNBC

Larry King Live (CNN)

NewsNight with Aaron Brown (CNN)

Special Report with Britt Hume (FOX)

The O’Reilly Factor (FOX)

Hardball with Chris Matthews (MSNBC)

Countdown with Keith Olbermann (MSNBC).

All cable programming was procured through videotapes and supplemented in some cases with transcripts. Taped programs were captured live in the Washington D.C. market, transcripts through the Nexis database.

Web Sites : We downloaded the top stories for the following Web sites at four different times during the day — 9 a.m., 1 p.m. , 5 p.m. and 9 p.m. The national Web sites were:

CNN.com

CBSnews.com

NYTimes.com

Dailynews.Yahoo.com (since changed to news.yahoo.com)

News.google.com

To select the Internet news sites to be coded, the Nielsen/NetRatings’ list of top 20 news sites was consulted to determine the most prominent sites. The list contained four basic types: news aggregators, newspaper sites, network news sites, and cable news sites. One site was chosen for each of those categories. For aggregators, Yahoo was selected as the highest aggregator in top 20. For newspapers, NYTimes.com was selected, again, as the highest of this kind in Nielsen’s list. For network news outlets, CBS was chosen because the study in 2004 examined the Web sites of ABC and NBC but not CBS. For cable, CNN was selected as the highest-ranked cable news site. In addition, we added Google’s news page as one of the newest forms of news content offered, with a unique method of aggregation.

Each site was visited four times on each day, at 9:00 a.m. ET, 1:00 p.m. ET, 5:00 p.m. ET, and 9:00 p.m. ET, to download stories. The order in which the sites were visited was rotated for each capture time. Each download took approximately 20 minutes.

Each time, the following method was used to determine which stories to capture:

On the news home page of each of the sites, we identified featured stories. A story at the top of a page tied in to a graphic element — commonly a picture of an event or person — was counted as a featured story and captured for study. Multiple stories on the page relating to the same graphic element were also captured as featured stories. Pages with more than one graphic element were considered to have more than one featured story, and all such stories were studied.

After the featured stories, we included the next three most prominent stories without graphics starting from the top and moving down. Those stories were recorded as non-featured.

The following rules were used in selecting stories:

Blogs : We used the following criteria to select seven blogs:

First, we looked at the most popular blogs by average daily traffic. Within that list, we started with the most popular blog and then moved down the rankings, establishing a mix of political ideology and geography. Finally, we added a seventh blog, a video blog, or vlog, to see how those differed from more traditional, text-based blogs. The blogs are as follows:

Daily Kos , Instapundit, Eschaton, Talking Points, Little Green Footballs, Power Line and Crooks and Liars as the vlog.

For each blog that day, we captured the entire sequence of postings and all embedded links on the main page. The postings were saved in Netscape and then copied and pasted into an MS Word document.

National Media Coding Procedures and Intercoder Reliability

Text-based Media

The national newspapers, Web site stories (both national and local), and blog postings (both national and local) were content-analyzed by the Center for Advanced Social Research (CASR) of Missouri ’s School of Journalism from July through September 2005. A description of the study’s objectives and data collection procedures was provided in a codebook to orient nine coders and two project managers working on the content analysis (the codebook for blog postings was modified to reflect the uniqueness of blogging in information-gathering and posting determination). The codebook contained a dictionary of coding variables, operational definitions, and measurement scales with detailed instructions and examples. Before actual coding began, extensive training was conducted to all coders, with additional assistance provided by researchers of PEJ. Weekly meetings were held between the coders and project managers to discuss problems and issues identified during the coding process. Coding sheets were designed for each of the media outlets and used consistently throughout the coding process. Because the coding variables were similar across the media outlets, reliability tests were conducted using a random selection of national newspaper stories for all the coding variables. The overall intercoder reliability was .892, calculated with Scott’s Pi (Riffe, Lacy, & Fico, 1998).

Video-based Media

The network television news and cable news programming were content-analyzed by ADT Research, publisher of the Tyndall Report. Since many of the findings are weighted by time spent on each item, it was essential to ensure accurate measurement of the duration of items. They were documented as rundowns with start times and end times, allowing for the duration to be computed as the difference between the two. All of the commercial network broadcasts were viewed twice to derive an accurate inventory and item-length statistics. To double-check the accuracy of all long-format items, coders not only designated packages, interviews (external or in-house) and stand-ups, but also documented a story summary in the form of headlines/slugs and the identity of the reporter or interview subject involved. Those headlines/slugs were also used to double-check topic and major story coding. All coding was conducted via a standardized codebook provided to ADT by PEJ. The codebook contained a dictionary of coding variables, operational definitions, and measurement scales with detailed instructions and examples. Again, before actual coding began, extensive training was conducted, with additional assistance provided by researchers of PEJ. ADT research reported strong reliability for all coding variables.

II. Local News Media

Sample, Story Procurement and Inclusion

For the local sample we selected three markets, one large, one medium-sized and one small. In selecting the three, the list of 210 markets was first divided into thirds based on population. Each group represented roughly one-third of the population, according to Nielsen Media Research. While ensuring geographical diversity, one market was then randomly selected from each group. The resulting markets were Houston (large), Milwaukee (medium) and Bend, Ore., (small).

Within each market, we captured and examined a wide spectrum of local TV, radio, print and online media.

The content selected within each medium was as follows:

The selection criteria led us to the following outlets and content within each market:

Houston

3 local TV stations (CBS, ABC, NBC affiliate)

KHOU-TV Channel 11, CBS affiliate.

6 – 7 a.m.: 11 News This Morning

6 – 6:30 p.m.: 11 News at 6

10 – 10:30 p.m.: 11 News at 10

KPRC Channel 2, NBC affiliate.

6 – 7 a.m. : Local 2 Today at 6

6 – 6:30 p.m. : Local 2 News at 6

10 – 10:35 p.m. : Local 2 News at 10

KTRK – Channel 13, ABC Affiliate

6 -7 a.m.: 13 Eyewitness News at 6

6 – 6:30 p.m.: 13 Eyewitness News at 6 PM

10 – 10:35 p.m.: 13 Eyewitness News at 10

2 local radio (2 news/talk)

News/Talk: KTRH 740 AM

ABC Affiliate and a Clear Channel Station.

1 hour morning: 6 – 7 a.m.

1 hour mid-day: 1-2 p.m.

No local evening news program

News/Talk: KSEV

Owned by Lieberman broadcasting, No affiliation.

1 hour morning: 6-7 a.m.

1 hour evening: 5-6 p.m.

No mid-day news

6 local newspapers (daily, suburban daily, alternative weekly, African-American weekly)

The Houston Chronicle: Houston ‘s main daily paper

The Baytown Sun: Houston ’s largest suburban daily; Baytown is west of Houston .

Houston Press (Alternative): Weekly publication with investigative reporting and coverage of the arts, food and clubs

Houston Forward Times ( African-American weekly): The South’s largest and only independently owned and published African-American weekly newspaper

Three Local Blogs

houston.metblogs.com: This blog is owned by Metroblogging, founded by two Californians. As of May 11, 2005 , they had blogs in 19 U.S. cities and said they would start in any city where they have at least 10 people committed to making postings. The one rule they stipulate is that all posts have to relate to that city somehow, but otherwise it’s fairly open to however individual bloggers see their own city. Subject matter varies.

BlogHouston.net: blogHOUSTON.net is a frequently updated weblog that produces regular commentary on Houston politics and current events, a steady dose of media criticism, and thoughts on Houston ‘s cultural and entertainment options: “We are about Houston .”

www.chron.com/memo (a Houston Chronicle blog): This is the Houston Chronicle blog on pop culture, which does not seem to be updated every day and is criticized by other local bloggers.

Milwaukee

4 local TV stations (CBS, ABC, NBC, Fox):

W TMJ4 – NBC Channel 4

6 -7 a.m. : Live at Daybreak

6 – 6:30 p.m. : Live at 6

10 – 10:35 p.m. : Live at 10

WDJT CBS 58

6 -7 a.m. : CBS 58 Morning News Express

5-5:30 p.m. : CBS 58 News at 5

10 – 10:30 p.m. : CBS 58 News Ten

WISN-TV Channel 12; ABC affiliate

6 – 7 a.m.: 12 News This Morning

6 – 6:30 p.m.: 12 News at 6

10 – 10:35 p.m.: 12 News at 10

WITI Channel 6 ; Fox News Affiliate

6 AM – 7 AM Fox 6 Wake Up News

6-6:30PM : Fox 6 News at 6

10:00-10:30PM : Fox 6 News at 10

3 local radio (2 News/talk, 1 NPR affiliate that does morning news):

News/Talk: 620 WTMJ Newsradio

CBS affiliated, highest rated

1 hour of local morning news: 5-6 a.m.

1 hour of local evening news: 5-6 p.m.

No mid-day local news

Local NPR Affiliate: WGTD FM News

NPR, WPR, AP

1 hour of local morning news: 8-9 a.m.

1/2 hour of local mid-day news: 12 – 12:30 p.m.

No local evening news program

News/Talk: WHBL AM News

ABC affiliate

-1 hour of local morning news: 6-7 a.m.

-1 hour of mid-day local news: 11a.m.- 12 p.m.

4 local newspapers (daily, African-American weekly, alternative weekly)

Major Daily: Milwaukee Journal Sentinel
Alternative Weekly: Daily Reporter General Interest

Alternative Weekly: Milwaukee Community Journal Alternative, “ Wisconsin ’s Largest African American Newspaper”

Suburban Daily: Waukesha Freeman, circulation 15,000

Local Blogs:

The two local blogs we found both come from the Journal Sentinel.

DayWatch: A blog hosted by JSOnline of today’s developing local news.

JSOnline weblog on state politics

Both blogs were originally selected for study. But there were no postings on either blog on May 11, 2005 so they are not a part of the final sample.

Bend

1 local TV station (NBC)

KTVZ Channel 21, NBC Affiliate

6-7 a.m. : NewsChannel 21@Sunrise (not on the air on 5/11/2005 because of a power outage)

6-6:30 p.m. : NewsChannel 21@ Six

11-11:35 p.m. : NewsChannel 21@11

1 local radio

KBND AM News, ABC affiliate

-1 hour of local morning news programming: 6-7 a.m.

-1 hour of local evening news programming: 5-6 p.m.

1 newspaper

The Bend Bulletin, http://www.bendbulletin.com

There appear to be no local blogs in Bend .

Local Media Coding Procedures and Intercoder Reliability

Stories appearing in newspapers and on television and radio for the three cities were coded for a specific day selected by the State of the Media Project. That included local newspapers, radio and local television in the communities of Bend, Ore., Houston, Tex., and Milwaukee, Wis. The tables appended to this memo report coder-reliability assessments for each medium coded. Finally, an overall summary of reliability is provided.

Personnel Involved in the Network News Coding

Faculty and graduate students in the School of Journalism at Michigan State University conducted this part of the project. The two faculty members who supervised the project have more than 40 years of combined social science experience in conducting such studies, and are two of the most published academic researchers in the field.

Two doctoral students in the Mass Media Ph.D. program at MSU, one a fourth-year student and the other a second-year student, coded most of the stories, assisted by a master’s degree student of the MSU School of Journalism.

The coding protocol was provided by personnel at the Project. The protocol called for the coding of 23 variables for all stories in each designated medium, and an additional 27 variables for selected “big stories” from each medium.

The two School of Journalism faculty members conducted extensive training sessions with the doctoral-student coders during the summer of 2005 using news stories collected for other studies. A decision was subsequently made to add the master’s degree student, and that student was trained during the fall. The reliability of this additional student was assessed against that of the doctoral students using the same sample stories used to assess the reliability achieved by the doctoral students. Comparability is therefore exact.

Procedure

The fourth-year doctoral student coordinated the coding work of all the other students. Coding was done independently, working from the protocol, without consultation among the coders.

Spanish-language media were also included in the study. To code this material, translators were employed to produce text in English. The translated copy was then coded by the regular coders in the study.

Reliability was assessed for each medium, using stories randomly sampled from the total to be coded. Reliability was first assessed on the 23 variables coded for all stories. Once acceptable reliability was achieved, all stories were coded for those variables. Reliability then was assessed for the “big story” variables, and once acceptable reliability was achieved on those variables, coding was completed for relevant stories.

In all, about 5% of all stories were used in the reliability assessments. The tables reporting reliability below are for just those variables on which substantive disagreements could occur. Percentages of agreement calculations were made to assess the coding for each of these variables. Each coder (the two doctoral students and the master’s student) were tested against each other on these variables. For news-organization stories on which all three coders participated, reliability assessments were made for all three coders and were then averaged for a total agreement.

Reliability Standards

Project procedures followed in the past relied on percentage of agreement for establishing intercoder reliability, and this study follows that precedent. The following standards, based on experience with past studies, should guide confidence in how reliably the variables were coded, and therefore the confidence researchers should have that findings represent real characteristics of the news coverage rather than coder bias or error.

In general, variables whose percentages of agreement exceeded 90% are highly reliable, and researchers can have high confidence that characteristics in the data are real rather than the result of coder bias or error. Variables whose reliabilities are below 85% but above 80 are accepted. The following chart summarizes these judgments.

Percentage of Agreement
Confidence
90% or more Very high confidence
85% to 90% Good confidence
80% to 85% Acceptable confidence
70% to 80% Very weak confidence
Below 70% No confidence

Based on those standards, 7 of 11 variables coded for all stories achieved very high confidence, with the remaining variables achieving acceptable or good confidence. Another 25 variables were coded for big stories only, and 24 of these achieved very high confidence. The remaining variable achieved good confidence.

Individual media exhibited different confidence results, however, and any analysis focusing on just one medium should examine the reliability figures provided for that medium. Even so, just one variable (Government Related stories) fell below acceptable confidence for the radio stories.

Below are the tables reporting reliability for stories examined for each medium and for all stories combined.

Coder Reliability Assessments for All Local Newspapers, Television and Radio News Stories (Figures are percent of total stories in the reliability test for each medium and for all stories combined.)

Variable
Newspaper Stories
(N=31)
Television Stories (N=33)
Radio Stories (N=33)
All Stories (N=97)
Dateline
91
85
88
97
Origin
94
88
100
94
Govt Relation
81
91
76
83
Geog. Focus
84
91
85
87
Topic
97
91
85
91
Transparency
97
100
88
95
Source Total
81
79
82
81
Female Total
97
97
97
97
Male Total
88
97
91
92
Anonymous
94
91
94
93
Opinion
94
100
97
97

Coder Reliability Assessments for Only Big Local Newspapers, Television and Radio News Stories

Variable
Newspaper Stories
(N=32)
Television Stories (N=34)
Radio Stories (N=33)
All Stories (N=99)
Dominant Source
97
100
91
96
Viewpoints
97
100
85
94
Direct Sources
85
100
94
93
Ordinary Sources
88
100
97
95
No Spl. Exp.
88
100
88
92
Spl. Exp
88
100
97
95
Local Relevance
100
100
100
100
Background
100
100
88
96
Implications
91
97
94
94
Impact
100
100
88
96
Human Face
97
97
96
96
Fact/Conjecture
100
100
97
99
Citizen Action
97
100
97
98
Consumer Action
100
100
100
100
Contact
97
100
97
98
Principles
97
100
100
99
Information
100
100
100
100
TV Newness
100
Video 1
94
Video 2
94
Video 3
94
Stills
100
Dup. Video
100
Focus Type
100
100
100
100
Focus Tone
82
97
82
87

III. Spanish-Language Media

Local Broadcast TV

KXLN-TV Channel 45 — 5 p.m. – 6 p.m. Noticias a las 5 (Univision Affiliate). Local news ran from 5 to –5:30 and Univision Network News from 5:30 to 6

KTMD TV Channel 48 5 p.m.–6 p.m. Noticias a las 5 (Telemundo Affiliate). Local News ran from 5 to 5:30 and Telemundo Network News from 5:30 to 6

Both TV stations are in the Houston metropolitan area.

Local Radio

Spanish-language radio stations broadcast in both the Houston and Milwaukee markets, but offered no scheduled news program to include in the study.

Daily Newspaper:

Rumbo de Houston: Rumbo de Houston is a daily tabloid published Monday through Friday. It is part of a group that publishes throughout Texas, including San Antonio, Houston, Austin and the Rio Grande Valley.

Story inclusion and rules were the same as the English-language process for each media type. Coding procedures differed in that the Spanish-language material was first translated into English and then coded under the same variables and rules as the English-language media.

Intercoder Reliability

Because the same coders coded the translated Spanish-language content and the English-language media, no separate intercoder reliability tests were conduced. The protocol and coders were judged reliable on the basis of the intercoder reliability discussed above.