Thankful for True Partnerships

Yesterday we sent out award letters to the new 2015 Howell-CSUPERB Research Scholars, along with our annual program report to the Doris A. Howell Foundation for Women’s Health Research board.

Howell Research Scholars are undergraduates who work alongside faculty researchers in biotechnology labs across the CSU.  For this program, the research projects proposed are relevant to women’s health.  The Howell Foundation defines women’s health quite broadly. As a result Howell projects range from basic research on the effects of ethanol on neurological development in fruit flies to clinical work to explore the links between sexually transmitted infections, sexual behavior and health outcomes in a campus setting.

The impact of this program is best communicated by the Scholars themselves (see page 2 of the report, linked here).  As part of their final reports, we ask Scholars to self-report on their gains as a result of the research experience (click on the image below to get the larger, visible version!).  We use David Lopatto’s SURE format (mostly) to

Students rank their learning gains as a result of undergraduate experience using a range from "very small gain" (=1) to "very large gain" (=5).  The 2014 Howell Scholars class reported average gains in the "large gain" to "very large gain" range!

Students rank their learning gains as a result of undergraduate experience using a range from “very small gain” (=1) to “very large gain” (=5). The 2014 Howell Scholars class reported average gains in the “large gain” to “very large gain” range. Interesting, the lowest self-reported gains this year related to science writing skills.

investigate learning gains as a result of the undergraduate research experience.  Even though most of the 2014 class of Howell Scholars had worked previously in a research laboratory, all of them reported large or very large overall benefits from the Howell-sponsored experience.

Most scholars work on their funded projects part-time during the spring term, then immerse themselves in the project over the summer. This kind of opportunity – to take a deep dive into a single research project – remains open to only a minority of undergraduate CSU STEM students.  In answer CSU faculty and administrators continue their work to develop new external partnerships, internship programs, and project-based learning in courses throughout the undergraduate curriculum.

Many of the Howell Scholars mentioned the importance of their faculty mentor. We know the immersive, working partnership and team-building between student, peer and faculty researchers offers a context in which students see themselves as scientists (often for the first time).  The teamwork and camaraderie students experience in working laboratories helps students persist even in the face of technical set-backs (and long experiments!).

“…there was a week in mid July; my experiment was behind schedule, and I need to report my data to my PI the next week. I therefore decided to work full time over the weekend. I told my fellow students that they didn’t have to go. However, all of them, all 3, decided that they’ll come in anyway, to help me get my work done. It was that moment that I realized how good of a team I had with me.” – Phuc Nguyen (CSU Long Beach, 2014 Howell Scholar)

True partnerships – shared goals, camaraderie, teamwork – are special things.  Our thirteen-year-long partnership with the Doris A. Howell Foundation is something we are thankful for here at CSUPERB.  Together we’ve been a good team!  The Howell donors remained committed to undergraduate researchers during the Great Recession; not all organizations were so steady in their giving during those years.  Likewise, CSUPERB was fortunate to maintain our budget for this program even in the face of deep cuts in state support for the CSU.

Both CSUPERB and the Howell Foundation have a long-view (influenced by the remarkable Dr. Howell) on educating physicians, clinicians and researchers. We know the effect of two terms spent in a research group plays out over decades and careers. We look forward to hearing from our Howell Scholar alums in February; every three years we reach out to them to find out where they are.  They often write heartfelt notes; they are still exceedingly thankful for the opportunities the Howell Foundation, its board and its donors made possible.

A New Part of NSF’s Innovation Corps

While it’s been quiet here on the CSUPERB blog, we’ve been travelling, consulting, building relationships and developing programming for a new NSF-supported Innovation Corps (I-Corps) Biological Site for the CSU!

icorpsAll the leg-work led to today’s system-wide call for Teams and Applications for CSU I-Corps opportunities this fall.  Contact CSUPERB, find an FCG member or ask at your campus research office to get details about our first CSU I-Corps informational webinar at noon on Friday, June 20th (sent out as part of a system-wide email today).

The last time CSUPERB formulated a strategic plan,* we decided to add an emphasis on entrepreneurship education.  Our simple aim is to teach CSU researchers about “what is needed to take a life sciences idea to a commercial product.”  In 2012 we organized the CSUPERB-I2P Early-stage Biotechnology Commercialization Challenge – an immersive entrepreneurship experience for CSU science, engineering and business students. Based on our experience with that program and the CSUPERB Entrepreneurial Joint Venture grant program, we submitted a grant proposal to NSF’s I-Corps Site program last May.

Due to the government shut-down and other federal budget wrangling, NSF didn’t make I-Corps Site awards until this May – but we did win an award! As a result we now have NSF backing to expand and institutionalize our biological sciences entrepreneurship educational programming.

The CSU I-Corps will:

This I-Corps Award is significant for the CSU.  Recipients of CSU I-Corps microgrants will be eligible to apply for NSF’s I-Corp Team grants.  Until the Site awards were granted, only NSF PIs had access to this program.

CSU I-Corps programming will help CSU researchers build teams and the skills to compete for follow-on funding from NSF, but also SBIR/STTRNCIIA E-Team and on-ramping opportunities at incubators and accelerators.  I should also note that NIH is embracing the I-Corps program** so soon there may be follow-on funding opportunities from that agency as well.

Our status as an I-Corps Site also gives us access to cutting-edge curriculum and resources of the National Innovation Network (NIN) that NSF has created, in addition to the life science entrepreneurship curriculum we’re developing with San Diego State University’s Lavin Entrepreneurship Center (Alex DeNoble, co-PI), Zahn Center (Cathy Pucher) and College of Sciences (Stanley Maloy, Dean & SPC member).  I attended the NIN meeting in April and brought home to the CSU many of the ideas and approaches I heard about there. The nationwide network of PIs continues to meet by videocon monthly – we have much to learn from each other about commercializing federally funded ideas!

Recognizing the work needed to build a solid and responsive network of alumni and partners, we also wrote a proposal to the CSU’s STEM VISTA program.  We are fortunate that not one, but two, VISTA members will be joining the CSUPERB program office in July to help us with organizational capacity building, student outreach and matching mechanisms for teams and mentors!  We are really looking forward to working with the AmeriCorps VISTA organization – you can imagine the energy and can-do effectiveness “domestic” Peace Corp members will bring with them to CSU I-Corps!  We hope their enthusiasm and talents will engage students enrolled at urban and rural, biotech hub-based and far-flung campuses across California in biological sciences entrepreneurship. I learned at the NIN meeting that NCIIA has a similar cadre of NSF-funded Epicenter innovation fellows (watch the cool ~1 minute video here). In fact the Spring 2014 cohort of University Innovation Fellows includes a Cal Poly alum! We plan to share notes on effective outreach and student engagement with the Epicenter program as well going forward.

After we submitted the I-Corps proposal in May 2013, Steve Blank and UCSF offered a LeanLaunch Pad course for life sciences (lessons learned can be found on his blog!)  Blank’s team discovered what we did running the I2P Challenge: it is critical for researchers to get out of the lab and off campus to talk with potential customers and industry experts about product development concepts, customer channels and regulatory affairs.  We partnered with an amazing array of campus innovation centers and biotechnology industry associations to organize meetings and workshops for curious academic researchers statewide – we have a partnership meeting in a couple of weeks to start scheduling!

We all hope that these immersive experiences will set researchers up for future success – whether it’s licensing out a promising idea, finding additional financing, taking a job at a start-up company or deciding more research and development is needed to commercialize a biotechnology idea.  At minimum – students say the team-based entrepreneurship experiences are eye-opening and lead to valued, real-world skills.   I found out this week that Warren Smith and Manmeet Singh (Sac State’s 2014 I2P first place finishers) won an NSF I-Corps Team grant, suggesting CSUPERB’s biological sciences entrepreneurship pipeline is primed!

CSUPERB gets glee in breaking down barriers between scientists, engineers and business folks.  We are grateful that NSF and AmeriCorps have provided fuel to continue our work for the next three years!


*During the Fall 2014 CSUPERB will embark on new strategic planning discussions for 2015-2018.  If you have ideas, suggestions or quibbles – contact us or your FCG and SPC representatives!

**I linked to the Science article about I-Corps above, but I do recommend reading it for background strategies and outcomes expected for this type of an entrepreneurship education program.  For more scholarly background, I also suggest Roman Lubynsky’s Kauffman Foundation article. From it you’ll get a very good sense of how long it really takes to commercialize research-based ideas as compared to technology-based ideas (and why many of us think researchers have perverse incentives – SBIR grants – to form bioscience companies too early)!  As I crafted the I-Corps proposal last spring, I collected these and other biotechnology entrepreneurship education resources at our site.

Spring Grant Announcements

We’re almost done with this year’s grant-making here at CSUPERB.  We just announced recipients of the Presidents’ Commission Scholars awards, as well as our “Major” Grant programs.  The Travel Grants committee is still working on their reviews – but we’ll announce those grants well before the end of the fiscal year.

These announcements usually mark our transition from grant-making to report-writing here in the program office; our annual report should come out in late July. It’s my favorite part of the year because we get to read Final Reports from the students and faculty members we’ve funded in previous years.  Even in these tough funding times, CSU faculty continue to win follow-on funding, broker partnerships and design curriculum to inspire and support teams of student researchers. Each year I pledge to myself to tell more of their stories and so the font size in the annual report gets smaller and smaller! Now that we’re down to 9-point font, I’m hoping public affairs staffers across the system might help out and will choose to feature some of the new awardees’ stories in campus publications, websites and blogs. The range and scope of science and engineering projects CSUPERB funds makes for fascinating reading!

I also want to take the opportunity to give a shout-out to some of our Genomic Analysis and Technologies Committee (the GATC task-force, of course!) members.  Anya Goodman (Cal Poly San Luis Obispo), Aparna Sreenivasan (CSU Monterey Bay) and Jim Youngblom (CSU Stanislaus) have worked with the Genomics Education Partnership (GEP) for the last handful of years. The national network of genomics educators, led by Sarah Elgin at Washington University, brings research experiences into the classroom and involves a virtual “army” of undergraduates in an ongoing fly genome annotation research program. In fact the GATC organized a workshop in July 2011 to develop curriculum modules for genomics education and invited Dr. Elgin to participate. The resulting modules were presented to CSU faculty at the 2012 CSU Biotech Symposium.  Last month the GEP network published a paper assessing the effectiveness of these course-based research experiences.  I won’t spoil the take-home messages because I think it’s good reading for all student-centered educators and administrators.*  It’s very good to see CSUPERB’s faculty researchers taking part in the ongoing national conversation about effective STEM education!


*I’ll add that the March 2014 issue of CBE Life Sciences Education is chock-full of interesting articles for mentors, educators and researchers interested in assessing how effective they are using “high-impact” practices.

Program Update: New CSYou Submission System

As Howell and Travel Grant applicants are finding out, CSUPERB is shifting our proposal submission, application and review systems over to a whole new system.

Our Online Application and Review System (OARS) system has been robust; in fact it’s accepting hundreds of symposium poster abstract submissions today!  But for a variety of reasons including the fact that none of us in the program office are really card-carrying software engineers, web designers or security experts, we plan to shift operations over to the CSYou system.

The CSYou system is open to all CSU faculty and staff.*  It serves as a system-wide intranet; its content ranges from collaborative project portals and Cal State news items to the handy-dandy “Find People” button that we use all the time here at CSUPERB.  To gain access to CSYou, point your browser to Things seem to work best when using Microsoft Explorer or Mozilla Firefox.  First thing you’ll see is a drop down menu; select your campus home.  You’ll be sent to your campus email portal, where you’ll enter your campus email username or ID number and password to gain entry to CSYou (today’s screen shot below). Thanks to some middleware magic, your campus credentials gain you access to the shared intranet resources.


CSYou is built on top of a Microsoft SharePoint database; that’s what got us interested here in the program office.  Relational databases make all of us really happy here in the program office (it’s true). But more importantly, even us non-software-engineers can create familiar-looking, web-based forms to capture applications and reviews.  Since Tyson Gadd joined the CSUPERB office, he’s been working with Ronnie Phipps at the Chancellor’s Office building forms and web pages for our grants programs.  We’re piloting the new CSYou system with the Howell-CSUPERB Student Research Scholars program.  During the next year we expect to migrate all our programs to CSYou, culminating with the complex 2015 symposium poster abstract submissions systems next fall.

When you use the link listed at the Howell Scholars application submission website, you’ll be taken directly to the submission webform (screen shot below). If you haven’t logged into CSYou yet that day (or have an open browser session already), you’ll have to log in using your campus user name and password before landing on the webform. Again, things seem to work best when using Microsoft Explorer or Mozilla Firefox.  Be warned: auto-fill entries aren’t captured properly by the system (if you see a yellow box, you’ll need to type your content to enter data).  You can upload proposals and supporting documents in Adobe PDF or Microsoft Word formats.


We expect some technical glitches and bumps as we bring these new systems online. As always, if you have difficulties, call the program office (619-594-2822) before the deadline and we’ll help you out.  Hopefully the majority of applicants will find the CSYou system intuitive and easy-to-use; if not, please let us know how we can improve it!


* For a variety of reasons – including mentoring, financial aid and privacy reasons – CSU students cannot submit applications and proposals directly to CSUPERB.  All our grant, award and symposium programs require the support of a CSU faculty or staff mentor. As a result we accept applications, nominations and proposals from CSU administrators, faculty or staff only.

Howell Foundation Board Meeting

Each summer I meet with the board at the Doris A. Howell Foundation for Women’s Health (DAHF,  Since 2001 CSUPERB has partnered with the DAHF to jointly fund undergraduate researchers.  In total we’ve supported 129 undergraduates at 20 different CSU campuses! During the summer we meet to plan the next Request for Applications (RFA) and smooth out the administrative wrinkles that come with partnerships.

For the last three years we’ve tried to assess the impact of the Howell-CSUPERB Awards on our scholars and track their career paths.   We now have OK, but spotty, data on the scholars funded between 2006-2012.  I’ve reported on student outcomes here before; the Howell Scholars data is a subset of the data presented in our annual reports. We extracted some of that data to report to the Howell Board on the program overall; here’s our 2013 Howell Board report (with photos, of course!).

Annual reporting is always tough on us data geeks – there is so much good stuff that never makes it into a report.  Like – did you know that 90% of the 2012 Howell Scholars had previous research experience? Most worked ~10 hours/week in an academic lab before becoming Howell-CSUPERB Scholars. Interestingly 90% of them are first-in-their-family to prepare for a biomedical research career. 38% of the 2012 Scholars report they “thought the research experience might help me get a job or into graduate school.”*  54% report they selected their faculty mentors based on faculty research pages at campus websites (31% did it the old-fashioned way – they took a class their mentor taught).  Bottom line, 100% of the 2012 Scholars thought the experience was (1) overall a good one, that the experience helped them (2) decide on what career path to follow, and that they (2) plan to pursue research opportunities in women’s health in the future.

It’s hard to decide how to tweak a program when the student-reported outcomes are that good!  As a result the Howell Board agreed to keep the RFA pretty much the same as last year.  That is – we plan to make up to 12 awards ($3500 each) to CSU undergraduate researchers interested in studying problems related to women’s health. We hope to see applications involving student-faculty teams “from life, physical, computer and clinical sciences, engineering, agriculture, math and entrepreneurship or business departments” (and, yes, that includes public health studies and medical device development!).  Look for the 2014 RFA to issue later in August after the SPC signs off August 6th (applications usually due the first week in October)!

We can not thank the Howell Foundation donors enough – they’ve chosen to support undergraduates and as a result have had incredible impact on students’ trajectories and career paths.  What a great way to give back!



*31% said the reason they became Howell Scholars was that “I thought having time with my mentor in lab would help me with my academic and career goals.”  The third most popular response (16%) was “I wanted to find out if a career in biotechnology might be something I’d like.”


Summer Break

It’s been so long since my last blog post that I had to install two new versions of WordPress software and renew an expired spam filter before getting started here!

I’ve written a lot lately – just not here on the CSUPERB blog.  I was working with state-wide teams on two NSF grant proposals (WIDER & I-Corps Site) that were due the first of July.  I like the work needed to pull high-performing teams together; “alliance management” is one of the skills I’ve picked up over the years.  I’ve found that pulling together and aligning teams around a grant proposal is often easier than getting the work done, if funded.  So recruiting motivated, can-do, and enthusiastic proposal-writing partners is critical to success on the back-end!  (To protect the innocent I won’t name names unless we’re successful!)

One of the things that motivated the team writing the WIDER proposal was Freeman Hrabowski’s 2012 TEDx talk. Spend 15 minutes once a term or so listening to this talk to regain your sense of purpose as a public higher education professional! Coincidentally Stanley Maloy (SDSU, SPC member) forwarded the link to me this morning – it’s good to see that video going “viral” in CSUPERB circles – so I’ll give it a nudge here!

I’ve also been reading a collection of hopeful NSF CAREER award proposals written by CSUPERB PIs.  That grant program is “dedicated to stimulating the discovery process in which the excitement of research is enhanced by inspired teaching and enthusiastic learning.”  We hope that CSUPERB PI’s absorb enough of our emphasis on the integration of education (teaching) and research to compete well for CAREER awards. So I’m finding myself pointing PIs to the “effective STEM education” literature (here) and community networks (here and here and here) we referenced writing our own WIDER proposal.  Most remain unaware of ways they can improve undergraduate education and instead propose activities to transmit their research excitement and discoveries “off campus” to museums and K-12 programs. Admittedly that’s good too, but selfishly, I’d like to see the excitement of research translated to the CSU’s undergraduates and potential transfer students!  Most PIs I’ve been emailing are not aware we’ve tried to collate materials for proposal writers at our website; but worse I think – when they get there – they’re overwhelmed with the need to open and read articles based on titles only.

So – I’m experimenting with a new way to curate materials of interest to the CSUPERB community – – so I don’t have to keep cutting and pasting web addresses (& I can share directly to LinkedIn or Facebook)! Here’s my first “” collection of networks, policy and reports on “Effective STEM Education.”  I’m hoping the graphics, tags and commentary help PIs decide if the link is worth following. Some PDFs can be read on the platform (or mobile app). You can filter or search on “network,” “NSF,” “biology,” “pre-med” tags, as well as many others.  I’ll keep adding links and tags to STEM Education materials I think are notable or influential (email me or comment here with suggestions!).  I’ll probably pull together an Entrepreneurial Education collection next, so stay tuned.

Good luck, grant proposal writers!

Spring Grants and Peer Review

It’s been a quiet month on the blog, but April is an important month at CSUPERB so we need to celebrate!

We announced the CSUPERB “major grant” awards and the Presidents’ Commission Scholars this week.  The Faculty-Student Collaborative Research Grants and the Presidents’ Commission Scholars are two of the most popular CSUPERB programs, as gauged by campus participation. So our normally quiet office enjoyed the email buzz from students, PIs, chairs and deans this week!

Campus participation defined by applications received from each campus to CSUPERB grant program, award program or as symposium registration. Data shown for AY06/07 – AY12/13.

Campus participation defined by applications received from each campus to CSUPERB grant program, award program or as symposium registration. Data shown for AY06/07 – AY12/13.

CSUPERB made 36 grant awards totaling $574,685 to CSU faculty at 17 CSU universities. Awards were made as part of four competitive CSUPERB grant programs: New Investigator, Research Development, Entrepreneurial Joint Venture and Programmatic Development. Faculty review panels evaluated 95 proposals from principal investigators (PIs) at 19 different CSU campuses. Averaged across the four programs, awards were made to 38% of the proposals received.

I use the scare quotes around “major grants” because these are the largest awards CSUPERB makes, but they are all seed grants that pay out $15,000 – 25,000 spent over 18 months.  The aim of these programs is to support preliminary work that can lead to follow-on funding from external agencies and organizations.  These follow-on grants support collaborative faculty-student research, innovative educational programs, and knowledge and technology transfer.  The reality of biotechnology-related scholarship is that significant funds (>$15k/year) are needed to support research programs.  Students gain deep learning opportunities working with PIs or participating in courses that are built on faculty scholarship.  As a consequence grant-getting is fundamental to biotechnology education and research.  We wish all our new PIs the best of luck in the lab, field and clinic!

Sixteen undergraduate researchers, the 2013 Presidents’ Commission Scholars, will be carrying out faculty‐mentored biotechnology research projects on 12 different CSU campuses this summer.  CSUPERB provides $8000 to support these summer research projects. This year’s request for proposals invited applications from CSU students early in their academic career.  The majority of applications were still from students in or starting their junior (3rd) year, but the selection committee funded freshman and sophomores as well.  Jaimey Homen, a chemistry student finishing her first year at Sonoma State University, will be working with Dr. Carmen Works to characterize photochemically activated molecules.  The group’s long-term goal is to engineer molecules that deliver carbon monoxide (CO) to specifically protect certain biological tissues. For context, CO has been shown previously to improve organ transplant survival rates.  Ms. Homen became interested in undergraduate research opportunities and met Dr. Works by participating in SSU’s Freshman Learning Community.  We hope Ms. Homen and the other 2013 Scholars have a wonderful summer!

CSUPERB’s peer review process starts in February when proposals are received.  This spring 57 faculty from 20 CSU campuses worked on six different proposal review panels.  The major grants were reviewed at meetings April 13-14 in San Jose; four different panels discussed and evaluated proposals that weekend.  The travel grants and Presidents’ Commission Scholar applications are reviewed by panels working on the internet and by teleconference.  Overall our faculty reviewers do a great job selecting promising research projects to fund.  For every major grant dollar awarded by CSUPERB between 2004 and 2010, PIs went on to win $14 (a 1400% fiscal “return on investment”) in grants from external organizations.  This, of course, is a direct credit to the excellent and competitive faculty scholars at work in the CSU.

We celebrate and justify our grant programs by pointing to the fiscal return-on-investment, but we also monitor student impact and knowledge transfer (publications, collaborations).  But any measure of peer review “success” must come with an acceptance of failure as well.  Not all the engineered strains survive, not all the experiments work, not all the hypotheses pan out.  Not all the PIs write well-crafted follow-on grant proposals, not all the research collaborations hold together, not all the innovative ideas find a good fit at a funding agency or an angel investing group.  Some ideas are ahead of their time, some skate too close to the bleeding edge, some are out of step with prevailing opinions. We teach our students and assistant professors that their success will depend on their ability to shake off failure and move on to write the next draft, design the next experiment, or repeat the test until it’s significant.  Some of those successes will come within the year, but scientific triumphs often take longer than we expect or come later in a career than hoped.

Expert scientists, engineers and clinicians are familiar and comfortable with these truths. None of us can predict the research projects that will work or have the greatest impact on society. But if we don’t talk about the failures inherent in scientific research and development, unintended and “disastrous”* consequences result.

Scientific peer review came under increased congressional scrutiny this week.**  Rep. Lamar Smith challenged the National Science Foundation (NSF) peer review processes and proposed new review criteria.   Rep. Smith went on to request access to the “scientific/technical reviews and Program Officers Review Analysis” for five specific NSF grants.  Yesterday President Obama defended scientific peer review during a talk at the National Academy of Science, stating, “I will keep working to make sure that our scientific research does not fall victim to political maneuvers or agendas that in some ways would impact on the integrity of the scientific process.”

Faculty reviewers and PIs probably don’t think often enough on the integrity underlying our peer review systems.  More often we grumble about nit-picking reviewers, the lack of high-risk, high-impact ideas, program officers’ insistence on well-written, on-time reviews, and the dearth of funds needed to support biotechnology innovation.  But if we sit back and ponder the implications of Rep. Smith’s requests to NSF, we suddenly see the wonder and power of our grass-roots, peer-driven national science agenda.  This is a process that serves to select the best science as-we-see-it, to plant the seeds of new technologies and therapies, and to train generations of the nation’s best-and-brightest scientists, engineers and clinicians.  The U.S. peer review systems underlying our research and development enterprise aren’t always pretty or perfect or innovative, but like our democracy, they’re highly regarded worldwide despite inherent incrementalism and consensus-building.  The corollary is that the aggregate outcome of peer review is the aggregate outcome*** of our nation’s research enterprise that remains envied worldwide.

Can we improve the system? Sure.  Even at CSUPERB we evaluate our programs, iterate our processes, and tune the strategic intent of our grant programs.  We do that with significant input from the expert science and engineering faculty involved with the program. We adjust to the budgets supplied by the taxpayers via the California legislature and the governor.  We keep our eyes on how biotechnology is defined by the external life science community. But – as of yet – we have not had to change how and what biotechnology research we fund in response to political pressure of any kind.

I understand the politicians in Washington, D.C. hold the purse-strings, but I sincerely hope political committees will not dictate how and what American science is done going forward.  To go that unscientific and undemocratic route would, indeed, be disastrous to our research and development enterprise.



* Characterization attributed to Bruce Alberts at Nature Blogs.

**The blogosphere is just getting heated up about this political power-grab of peer review, but some good context is provided by Derek Lowe and The AmericanScience bloggers. 

***U.S. research outcomes can be reported many different ways, for example, see NSF’s measures and outcomes and Ben Bernanke’s take.