Evaluation Report: Course Project – EdTech 505

The Evaluation Report-Course Project for EdTech 505 is what I’ve selected to exhibit mastery of substandard 4.1.  The purpose of this assignment was to evaluate a program that encouraged the promotion of Google Drive use among students so that an institution (in this case, a middle school) could decide if it would be more effective than what systems were already in place.  South Middle School (SMS) offers its students file storage on hard drives stored on the school’s network.  The program being evaluated would see SMS have their students sign up for a Google Drive account, either with a parent or on their own (depending on the age of the student).  Prior to the evaluation, I prepared program goals and objectives.  The components to the program, and procedures for each, along with evaluation methods were developed.  Costs estimates were provided to the district.

To test whether students felt Google Drive provided a better alternative to their present plan, a sample of SMS students completed an assignment with the current system, then completed an assignment using Google Drive.  After each step, students completed a survey to gauge which they felt was more effective.  Parents also did a survey to gauge how effective they felt it was for their student.  Results showed that students completed the assignments at a higher rate when they had the ability to access their work via Google Drive as opposed to the school’s hard drive.  None of the students that were randomly selected for the program had used Drive before, but upon completion 100% either agreed or strongly agreed that the school should promote Google Drive use to students.

I had never run any program like this previously.  The planning was daunting.  I had to decide what would be the goal of the evaluation program.  Some are done to evaluate effectiveness for various reasons, but mine was to find out how people felt about using Google Drive.  Sampling was also a big decision.  You have to decide if it will be random, representative of the demographics of the group as a whole, or any number of multiple variables.  I had to limit my sample size to students who had Internet access, or the program would not have worked at all.  One could argue that this possibly skewed results, but it had to be done.  Then I had to make sure the evaluation actually tested for the desired outcomes.  You don’t want your clients to have to wonder if the positive or negative results were brought on by something out of the control or scope of the program.  By keeping the focus of the program narrow, I feel I accomplished this.  This assignment gave me the tools and confidence necessary to plan, implement, and evaluate systems in the future, which is why I’ve chosen it to show my mastery of this substandard.

 

Evaluation Report

South Middle School Google Drive Program

Prepared by

Scott Hogan

December 11, 2012

Introduction

Schools in the Nampa, Idaho district are putting higher expectations on teachers, students, and parents to be able to have and use computers and the Internet to check grades, follow daily agendas, post assignments, turn in work, collaborate, etc. While this provides opportunities for higher achievement, it may also put a greater burden on families financially to have access to such tools. Schools also face infrastructure challenges as they attempt to keep their schools as up to date as possible with rapidly changing technologies. While all of this cannot be addressed in such a small evaluation, the ability of students to have access to a simple and cost effective means of creating, storing, and sharing many types of files is something that can be focused upon. With all of the available options, the promotion of a service that if free and easy to use is something that South Middle School must do to continue to best serve its students and families.

Summary

At South Middle School, and at schools all over the state of Idaho, teachers are increasingly being asked to incorporate technology into lesson plans and assignments. Students are expected to have higher levels of competence about using computers and the Internet to create and turn in assignments for their classes. South has a population of approximately 1,000 students. While at times it expects students to work with technology, it also prevents it as well. Until the 2012-13 school year students were not allowed any email access at school. Any files that students started to work on while at the school remained there. The only way students could get school files home was if they had a portable file storage (PFS) device; a disk, thumb drive, etc, with them on the day they started an assignment. Basically, students often had files stuck at school and could not work on them at home.

The program sought to see how students responded to cloud based file storage (CBFS) as a means of working on and finishing assignments from home and getting them turned in to teachers. Students, with their parents’ supervision, signed up for a Google Drive account that allowed them to open files at school and have access them anywhere else they had Internet. Students would then start an assignment in our school’s lab but finish and turn it in from an off site location.

The goals of the program are 1) to assess how convenient to use students find CBFS, 2) if students would want to use CBFS tools to create assignments, 3) whether or not students would use CBFS for turning in schoolwork, and 4) if students felt that CBFS should be promoted by South M.S. for use among students.

The purpose of this report is to show the results of the CBFS program and how students felt about when using “the cloud”. Before beginning, the evaluator met with teachers to get their input about problems they faced when giving computer based assignments in the computer lab. A survey was given to parents and students to gauge how much access to technology kids had when not at school. For the program, students started an assignment in the school’s lab and then finished and turned it in from home. A survey was then given to see how students felt about the CBFS program.

Responses to the final survey revealed that students were highly interested in using CBFS for computer based assignments. In general they stated that Google Drive was easy to use in almost every aspect and that they were likely to use it in the future. Students also felt that the school should promote the use of CBFS to students at the beginning of the school year.

Program Description

This program focused on Google Drive and its suite of productivity tools that are very similar to items that come standard on many computers with Microsoft Office installed. Students at South M.S. took part in the program that lasted for approximately two weeks. Elements of the program were completed in a computer lab at South M.S. and at the homes of the students.

The program sought to evaluate how easily assignments that were started in a school setting might be completed from the students’ homes. In the survey given before the program began, almost 60% of students expressed that they had never heard of CBFS, while 31% responded that they had but were not sure what it was. This means that the students who completed the program were not highly engaged technology users when it came to schoolwork, so results should be applicable to any student in general.

Program Objectives

Using a goal-free model, the program wanted to see how an average student would adapt to using CBFS for beginning, saving, completing, and turning in computer/internet based homework.  Because of the nature of this method there are no goals per se, the program did set out to observe the following items to see how useful Google Drive or other CBFS could be for students at South in the future.

• How easy did students report that it was for them to use the tool?

• Were they satisfied with how the tool worked for them?

• Were they likely to use this new tool to help them in the future?

• Did students think that the school should promote the use of this tool?

Program Components

• In the program students and their parents will be asked to sign up for a Google Drive account. This will be used to administer an assignment. If parents/students do not feel comfortable signing up for an account a tutorial video has been created to walk them through the process.

• Students will be asked to sign in to their accounts when at school and begin an assignment. If they don’t know how to create a Google Drive document they can access a tutorial video explaining how to do so. Once they have completed the first part of the assignment they will exit the application.

• At home (or some place off site from our school) students will again sign in to their Google Drive accounts and complete the assignment that they began at the school. Students will then use the share function of Drive to turn in the assignment.

• Students will complete a survey following the assignment that measures their attitudes toward Google Drive and how effective they feel it was for them to use.

Evaluation Method

Participants

• Before beginning the program, students in both seventh and eighth grade computer application classes at South M.S. were given a survey (by the teachers of those respective classes and outside the boundaries of this program) about their access to technology when they are not at school. The survey was taken by 152 students out of a population of 965. All of the students who eventually were chose came from this sampling group. For the purpose of the program, participants had to have a computer and internet access at their homes. Of the students surveyed, 75% had a computer and internet access at home. Students who had grades of F or lower were excluded from the program as not to create another burden on their time. Other than that no grade restrictions were placed on participants. Twelve students’ names were chose randomly by drawing names from a bucket. They then took home a leaflet outlining the expectations of the program. Two students replied that their parents had declined to have them participate. Those names were replaced by two more names chosen at random. Twelve students participated in the assignment completion portion of the program and completed the survey at the end.

• Parents of the students were also involved. Google does not allow children under the age of 13 to create accounts. This is one of the reasons that it was a requirement that parents approve of involvement in the program. Also, since it was conducted online with membership in a service, evaluators wanted to be aware of privacy concerns. Parents were asked to choose a password with the students so that parents could also have access to their minor children’s email accounts.

• Some staff at the middle school were consulted before the beginning of the program to get their opinions about the possible usefulness or need for students to have access to a CBFS system at South. In one-on-one informal interviews they were asked a few simple open-ended questions to gauge their thoughts on an evaluation of this nature.

Procedures

Before the program began the evaluator sat down with three staff members at South M.S. who use the computer lab regularly (at least 2-3 times a quarter) and asked them about the problems they experience most in the lab. Aside from students not knowing their login information, the most common response was that students could only save work to the school’s network hard drive. All three teachers mentioned that “almost never” did a student have a portable drive available on the days when the class went to the lab.

Another issue that was unanimously voiced was that many students were not able to finish assignments during the school day (slow computer speed, time to sign in, slow data connections, distracted and off task behavior, and difficulty with content comprehension were all mentioned as causes.) Teachers expressed frustration that when this happened there was no way for students without a portable drive to take the work home.

Also cited as a  problem was that there was always an amount of their students who don’t have computer/internet access (our data found the number to be 25%). It is not the intent of this program to evaluate those students, but rather to see if CBFS can be a solution for those students who have the technology available to them at home.

Due to the small size of the group involved in this evaluation, only one administrator was needed. After the selection process was executed, the evaluation program went forward as follows:

• Step 1

Students and parents sign up for a Google Drive account, with or without the help of the tutorial video. Parents also fill out the survey (Appendix A) about their students computer usage and privacy concerns they may have regarding setting up an online account for their child.

• Step 2

Those students who have been chosen for participation in the program will be taken out of their homeroom class during eighth period of a specified day and asked to sign in to their Google Drive account. They will be given a link to the assignment for completion. It will ask them to open a document in their account, name it, and answer the questions from Part One of the assignment (if they don’t know how to open a document and name it they can view a tutorial video for help.) Once they have done this they will sign out.

• Step 3

At home students will be expected to sign in to Google Drive and open the document they started at school. There they will follow the instructions for Part Two of the assignment. Once Part Two is complete they will “share” the document (instructions available through tutorial video) with the evaluator.

• Step 4

Either at home or school each student will fill out the post-program survey (Appendix B) to gauge their attitudes about the Google Drive and whether or not they think it could be useful for them.

Data Sources

For the evaluation of this program data were mainly needed to judge students’ attitudes about the use of this new (to them) technology of CBFS. However, the evaluator also used a survey that had been administered by the school’s computer teachers as a means of analyzing the levels at which students had access to computers and the internet when at home. This data was used to narrow the sample size for inclusion in the program.

Students were observed in the computer lab while they completed Part One of their work on Google Drive. Many of the students remarked that it was much faster to sign in to Google Drive than to wait for Microsoft Word to open on their computers. They explained that it often took one to two minutes for Word to open, and they felt the internet worked more quickly, which they liked.

The final survey was completed outside the observation of the program evaluator, but responses were overwhelmingly positive about the uses of Google Drive by the students involved. Students were questioned about many of the functional features about the Google Drive program. They felt that it was easy to create files (92% said it was somewhat to very simple), that the auto-save feature was convenient (75% of respondents), and that the share feature was quite simple (100% saying they were likely to use it in the future).

Results

• Technology Access Survey

This survey administered by South staff gave some insight into the needs of students:

• Results showed that roughly 70% of students have access to computers and internet when not at school. This definitely limits the ability of schools to give more technology related assignments as homework.

• While 84% of students reported having access to an email account, only 43% knew how to attach a file, which limits their ability to turn in assignments.

• 95% of those surveyed responded that the felt it would be very useful for students to have a place at school that they could save files so that they could access them when not at school.

• Post-Assignment Survey

After completing Parts One and Two of the program project students answered the following questions about their experience with Google Drive.

Table 1: Results of Post-Assignment Survey – Knowledge of CBFS

	Yes	No	Yes, but I’m not sure what it is.
Do you know what cloud based file storage is?	17%	67%	17%
	Yes	No
Have you ever used cloud based file storage?	0%	100%

Table 2: Results of Post-Assignment Survey – Ease of Use

	Auto-save much easier	Auto-save somewhat easier	Neutral	I remember somewhat easier	I remember much easier
Ease of use: auto-save	58%	17%	5%	0%	0%
	Very Simple	Somewhat Simple	Neutral	Somewhat Difficult	Very Difficult
How simple or difficult was it for you to create a document on Google Drive?	42%	50%	8%	0%	0%
	Google Drive is very convenient	Google Drive is somewhat convenient	Neutral	Google Drive is somewhat inconvenient	Google Drive is very inconvenient
How would you rate the convenience of saving files on Google Drive?	42%	50%	8%	0%	0%

Table 3: Results of Post-Assignment Survey – Usefulness in School

	Make it easier	Not change it at all	Make it more difficult
How would using Google Drive at school effect having access to files outside of school?	67%	33%	0%
	Highly Likely	Probably Will	Neutral	Probably Not	Not At All
Likeliness of use for turning in or sharing homework	42%	58%	0%	0%	0%
	Highly Likely	Somewhat Likely	Undecided	Somewhat Likely	Highly Likely
Use Google Drive to store files at school in the future?	42%	58%	0%	0%	0%
	Strongly Agree	Agree	No Opinion	Disagree	Strongly Disagree
South should promote the use of Google Drive	33%	67%	0%	0%	0%

As the above responses suggest, students find it easy to use Google Drive at home, enjoy the ease of use that it provides, and would recommend that South promotes it for use among its students.

Discussion

Program Overview

Increasingly, students at South Middle School (SMS) are being asked to create assignments using computers and the internet. They must start this work at the school and most of the time the only place they can save their work on the school’s network hard drive. This would not be a problem if students were given enough time in the labs to finish their work, but lab time is at a premium and there are rules about how often one teacher can take their class to the lab. The program sought to gauge student attitudes about the use of Cloud Based File Storage (CBFS).

Students were chosen out of a sample of students at SMS that had access to computers and the internet at home, which was critical to completing program work. Once parents had given permission for inclusion in the program, students and parents set up each student with a Google Drive account that belonged to the student. At school students were taken to a computer lab and asked to open a Google Drive document and begin Part One of an assignment. At home they accessed the same document and finished Part Two and then shared the assignment with the evaluator. After this they completed the Post-Assignment survey to measure their attitudes about how difficult or easy it was to use the CBFS. Students overwhelmingly found the program easy to use and felt that the administration of SMS should encourage its use among the student body.

Post-Assignment Survey

According to feedback received from the Post-Assignment survey, 100% of participants responded with “Agree” or “Strongly Agree” when asked if SMS should promote the use of Google Drive. Also, 100% stated that they were either “Somewhat Likely” or “Highly Likely” to use the tool for file storage and turning in homework in the future. A large majority, 67%, felt that the use of this CBFS would make file storage at school easier. In fact, there was no category of the “Ease of Use” section of the survey that garnered a single negative vote from program participants. For a group of students who had almost no knowledge of CBFS before the program, they became fans quickly.

Technology Access Survey

This was administered by staff at SMS prior to the program and was meant to determine to what levels students had use of computers, the internet, portable storage devices, and email accounts. It also sought to learn how knowledgeable parents and students were about attaching files to outgoing email and about CBFS systems. Of the 152 people who took the survey, most responded (75-85%) that they had computer and internet access when not at school. While 84% said they had email, only 43% knew how to attach a file and send it. We knew this program could be significant when responses to the survey showed that 89% did not know how to use, or had never even heard of, CBFS, and that 95% felt that it would be beneficial for students to have some file storage method from which they could access files away from school.

Parental Concern Survey

As the name suggests this was conducted not specifically to make the program work better, but mainly to engage parents and see to what level they were concerned about their students being involved in a program that required them to have an online account. The survey found that only 7% of parents were even mildly concerned about this issue, but that 68% felt that students should be monitored in such situations. In a question related to the content of the program, 42% of parents had either not heard of CBFS or had and didn’t know what it meant.

• Write an evaluative paragraph or two about the program based on each data source. Use a similar order to the one you used in the results section. That is, start with the most important source, then go to the next important one, and so forth.

• Write a paragraph or two on any components of the program that stand out for some reason. Normally, this may be because they are particularly good or bad. If the latter, make a suggestion for what to do about it if you can, based on what was found during the evaluation (not your personal thoughts, per se).

• Describe in a few paragraphs your overall evaluation of the program, as illustrated by the data. The major part of the description should address the purpose of the study.

Project Costs

The administration, monitoring, data collection, and evaluation of the program will all be completed by the lead evaluator, Scott Hogan.

Staff Daily Rate # of Days on Project Total

$500.00   4      $ 2000.00

Travel Costs # of Miles Travelled

.27/Mile  80 21.60

Daily Per Diem # of Days on Project

$35   4           140.00

Office Supplies/Printing

Report Printing & Binding # of Report Copies

$15  10           150.00

     $ 2311.60

Evaluation staff consisted of Mr. Hogan whose duties included the following:

• Meeting with staff to discuss the finding of the Technology Access Survey

• Coordinating with staff to select students who fit program criteria

• Creating program activities and surveys

• Meeting with students to explain program activities

• Creating tutorial videos to aid in successful activity completion

• Collect and evaluate post-completion surveys

• Write the evaluation report

• Printing and binding of the evaluation report

Much of the time spent by Mr. Hogan at SMS was dedicated to meeting with staff and students accomplish a needs analysis of what was missing in regards to file storage so that the program could best address those needs. Time was allocated for activity creation and administration, as well as the writing of the post-activity survey. Mr. Hogan also created tutorial videos about how to open a Google Drive account, create a Google Document, and how to share a document with others. Throughout this entire process school technology resources were used to help keep costs to a minimum. Finally, ten copies of the evaluation report were printed for distribution to all required parties.

Appendices