Written on the 473
From Taipei to Taichung
Sunday, June 21, 8:17 pm

It’s Sunday, and I’m just two short hours away from being back in Taichung and seeing Steve again! A two-week separation has been difficult, but I’ve had a lot of fun in Taipei, and Steve has learned a lot of Chinese. The long weekend for the Dragon Boat Festival (端午节) means the trains are packed with people going home and coming back to work. Since I have some time, I’ll write a bit more about my internship, especially since it’s more than half over already!

When I started talking to the Taiwan Fund for Children and Families back in January about an internship, I knew three things: I wanted to do something quantitative with either economics or statistics; I wanted to learn more professional Chinese, since all my public policy training is in English; I wanted to experience the work environment in Asia, since we’re interested in moving to this continent (and very likely, this island!). By mid-March, we had hammered out two projects for the internship.

First, I’m doing an evaluation of the youth capacity building program, which serves college students from disadvantaged families in Taiwan. These students do teambuilding activities, volunteer, and some career development activities; they also save money toward a personal or professional goal, and the organization matches their savings dollar for dollar. Secondly, I’m updating the survey that they’ve been using for ten years with new questions and new methods. Here are some of the things that I’ve been thinking about regarding this project.

Firstly, why do a program evaluation? In the US, many social programs are paid for through government tax revenue or large philanthropic foundations. Both entities are increasingly shifting toward a data-driven or evidence-driven approach. Not only do they want to know that an organization is giving job training to unemployed people, but they want to know exactly how it has decreased the unemployment rate. Evaluation can ensure accountability for the huge amount of money organizations spend on these programs, because there are certainly better ones and worse ones. However, evaluations are expensive, and people who became social workers or teachers or drug abuse counselors sure didn’t do it to become mired in paperwork. And how do you really define the effect that a policy or program has on a certain population? How can you be sure that’s what’s causing the change? That’s where policy analysts come in.

As I’ve been learning this past year of grad school, statistics addresses these questions through the same scientific method through which drugs and other medical interventions are also tested. Ideally, we randomly select half of a group of people to take part in the program (treatment group), and half of them to not (control group); you also collect data before and after the program and take into account all the factors that can possibly affect the outcomes that you’re interested in. If everything is done well, the result is a randomized controlled trial, the gold standard for scientific experiments, which is as close as we can come to making the claim that the program/treatment created an effect of X on Y population. It’s not foolproof, but statistics lets us be reasonably sure that it is not because of another factor or because of random chance. That being said, it’s nearly impossible to get this golden standard unless you’re prepared to spend years and millions of dollars!

Since early May, I’ve been learning about this program, and working with a large data set of pre- and post-program survey results from two years of students who took part in the youth capacity-building program. Statistics in the real world is rather unlike what we were doing in class or in our labs, even though the data set is rather complete, and I’m using the same software, STATA, in order to work with the data. We do have control groups, which is the first step towards claiming causality, but the recruitment of students and assignment to participate in the program was definitely not random. The program also varies hugely depending on what center, what social worker is in charge of the program, and year to year.

I think the most interesting part of the program evaluation has been figuring out exactly why we want to do the evaluation. Though it sounds incredible, we’re still figuring that out, a month and a half into my internship. It sounds rather simplistic to say, we just want to make sure the program is working. But the program has multiple aims – to help students save money, to help students develop social and economic assets, to help them increase their employability, to help them develop a greater awareness of issues in their community. And what improvement of what measures or indicators count as “good enough”? And can the questions you’re interested in today be answered by the survey that your predecessors created nearly a decade ago? There’s a lot to consider and think about before you even start working with the actual data and running specific tests.

I’m also learning a lot more about STATA, but there’s been some trial and error. Coding is in my opinion quite unforgivable. It takes you literally at your word, and if you don’t say exactly what you mean to say, you can end up with the wrong thing (and sometimes, not even know it’s wrong). I had to learn new coding commands in order to import and clean the data – make sure the variables are all labeled, change all the missing values to things that STATA recognizes. Now I’m running a bunch of t-tests, which helps us compare pre- and post-test answers to the same question and figure out if any difference between the two is a result of random chance or so statistically unlikely (5% or less) to be random that we may be able to attribute it to the program. It’s actually kind of fun, because you type in as little as three words, and watch the software pump out half a page of numbers to the fourth decimal place. I think I know how Steve feels sometimes!!

The next few weeks are going to be busy – my supervisor just got back from her training session in the US for three weeks, and starting on Tuesday, we’ll be reviewing my work so far and figuring out what further analysis is needed. Then I’ll have to write it all up, and create a short presentation, and deliver it. All by July 17! Wish me luck.

Connie