Kenya Threshold Program Monitoring & Evaluation
How MCC Tracks Progress
MCC emphasizes both achieving and measuring results in its threshold programs. During program development and implementation, MCC worked closely with Kenya’s Ministry of Finance, which coordinated program design on behalf of the government, and with USAID, MCC’s on-the-ground representative. Both USAID and the Government contributed to establishing program milestones and refining program indicators for the Kenya Threshold Program.
Kenya’s Threshold Program indicators were tracked quarterly by MCC in a Results Reporting Table—provided below—that reported progress over each quarter compared to the baseline and program targets.
Final Implementation Reports
Three months after the threshold program’s conclusion, USAID provided MCC with a final report based on information provided by implementers and USAID program managers.
Results Report Table
Kenya’s Threshold Program indicators were tracked quarterly by MCC in a Results Reporting Table that reported progress over each quarter compared to the baseline and program targets.
| Performance Indicator | Target | Baseline | 2007 Q3 |
2007 Q4 |
2008 Q1 |
2008 Q2 |
2008 Q3 |
2008 Q4 |
2009 Q1 |
2009 Q2 |
2009 Q3 |
2009 Q4 |
Notes | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall Indicators | ||||||||||||||
| 1. Improve transparency, accountability and corruption in the public sector. | 3.5 | 3.0 (March 2007) | Target | N/A | N/A | N/A | N/A | N/A | 3.2 | N/A | N/A | N/A | 3.5 | WBI indicator; annual. |
| Actual | 3.0 | – | – | – | – | 3.0 | – | – | – | 3.0 | ||||
| Component One – Procurement Reform | ||||||||||||||
| 2. Comprehensive procurement reviews on high value procuring entities (PE) and published on PPOA website | 6 | 0 (October 2007) | Target | N/A | N/A | N/A | 1 | 2 | 3 | 4 | 5 | 6 | 6 | |
| Actual | 0 | – | – | – | 2 | 3 | 3 | 4 | 8 | 10 | ||||
| 3. Record Keeping: Key PEs implement proper procurement record keeping procedures | Six PEs achieve a score of 65 points by the end of the program | 6 baselines TBD by survey Jan 08-Jun 08. MOH – 19 pts. MOE – 21 pts. KEMSA – 34 pts. MOEd – 35 pts. MORPW – 24 pts. PPOA – 28 pts. |
Target | N/A | N/A | N/A | N/A | N/A | Manual finalized; 15 ministries/depts. and 3 local govt. PEs trained | 30 ministries/depts. and 6 local govt. PEs trained | N/A | Score of 65 for 6 PEs | 50% | New scores: MOE – 25 pts. KEMSA – 43 pts. MOEd – 45 pts. MOH was split into two ministries: MOMS – 35 pts. & MOPH – 24 pts. MORPW was split into two ministries: MOR – 24 pts. & MOPW – 35 pts. OP – 27 pts. (See footnote.) |
| Actual | N/A | N/A | N/A | N/A | N/A | Manual finalized. Record keeping was added as a module within the general sensitization course (for 36 PEs during the quarter). Targeted record keeping workshops were conducted in early Oct 2008 and will be reported next quarter. | 19 ministries, 16 paratstals, 5 local government and 25 departments trained. | – | As detailed in the Notes column, no PE has achieved an acceptable score. | – | ||||
| 4. Use of framework contracts by GOK (3 Milestones) | Achievement of 3 milestones | 0% | Target | N/A | N/A | N/A | N/A | 1. GOK circular released | 2. Key PEs trained | N/A | 3. 10 framework contracts in place and in use | 50% | ||
| Actual | N/A | N/A | N/A | N/A | Circular has not been released. | The circular was released to 8 pilot PEs. The final version of the circular will be released January 2009. | 39 PEs trained | Framework contracts are being piloted by the GOK in 5 PEs. | The GOK continues piloting of framework contracts in 5 PEs. Guidelines have not been broadly released. | Workshop held during the quarter for pilot PEs. Guidelines still have not been broadly released. | ||||
| 5. Number of PEs reporting large procurements to PPOA | 48 PEs report to the PPOA | 16 PEs reported for the quarter ending June 2007 | Target | N/A | N/A | N/A | 28 | 32 | 36 | 40 | 44 | 48 | 48 | There is little reporting for the current quarter as reporting is generally lagged one quarter. Previous quarters have been adjusted per updated information. |
| Actual | N/A | N/A | N/A | 13 | 24 | 64 | 27 | 97 | 128 | 3 | ||||
| Component Two – Improving Healthcare Delivery | ||||||||||||||
| 6. Average percentage of stock records that correspond with physical counts for a set of indicator drugs in the central KEMSA warehouse | 100% | 0% (April 2008) | Target | N/A | N/A | N/A | N/A | 100% | 100% | 100% | 100% | 100% | 100% | Collected quarterly. Following training of warehouse staff, the stock management system was overhauled to ensure timely update of records and improved internal tracking procedures. |
| Actual | N/A | N/A | N/A | N/A | 0% | – | 40% | 70% | 70% | 90% | ||||
| 7. Average of time out of stock for a set of indicator drugs in the central KEMSA warehouse | 0% | 22% (April 2008) | Target | N/A | N/A | N/A | N/A | 0% | 0% | 0% | 0% | 0% | 0% | Collected quarterly; see footnotes. This indicator continues to deteriorate arising from lack of stocks due to delayed procurement and depletion of previously available stock. (See QNR Issues.) |
| Actual | N/A | N/A | N/A | N/A | 22% | 0% | 2.5% | 16.7% | 22.14% | 30.8% | ||||
| 8. Percentage by value of KEMSA medicines purchased through competitive tender | 100% | 100% (July 2007) | Target | N/A | N/A | N/A | N/A | 100% | N/A | N/A | 100% | N/A | N/A | Collected annually. |
| Actual | N/A | N/A | N/A | N/A | 100% | N/A | N/A | 92.8% | N/A | N/A | ||||
| 9. Percentage of average international price paid for last regular procurement of a set of indicator drugs | < 100% | 60.3% (July 2007) | Target | N/A | N/A | N/A | N/A | < 100% | N/A | N/A | < 100% | N/A | N/A | Collected annually; see footnotes |
| Actual | N/A | N/A | N/A | N/A | 60.3% | – | – | 43.1% | N/A | N/A | ||||
| 10. Turn-around time for hospital orders (provincial and district) | 7 days | 24 days (Sept. 2008) | Target | N/A | N/A | N/A | N/A | N/A | N/A | 7 | 7 | 7 | 7 | Collected quarterly. The deterioration in this indicator is attributable to the delay in KEMSA’s completion of their annual stocktaking exercise, which resulted in new orders not being processed. |
| Actual | N/A | N/A | N/A | N/A | N/A | N/A | 14.3 | 9.5 | 14.1 | 20 | ||||
Footnotes:
1. Source: World Bank Country Policy and Institutional Assessment (CPIA). Scores range from 1 to 6, where “6” is the best and “1” is the worst. Baseline Year: 2005. Data for 2007 was released June 2008; there was no change in the value from 2006 to 2007. CPIA results for 2008 were released during the quarter; there was no change in the value.
2. Under the Threshold Program (TP) we will, in concert with the Public Procurement Oversight Authority (PPOA), perform in-depth reviews of six key procuring entities. The results of these reviews will be publicly available.
3. Under the TP, we have provided training to selected central government and local authority procuring entities (PE) to improve their procurement record management systems. We have assessed and scored six key PEs on a scale of 0-65 points to establish a baseline at the beginning of the Program, and again in early 2009. The six procuring entities are: Ministry of Health (MOH), Ministry of Energy (MOE), Ministry of Education (MOEd), Kenya Medical Supplies Agency (KEMSA), PPOA, Ministry of Roads (MOR). In addition, a baseline review of the Office of the President (OP) was conducted during FY 2009 2nd Quarter. The assessment rates PEs on such factors as: existence of a records management system and policy; assignment of responsibility to one or more individuals; completeness of key files such as tender committee files, accounting files, contracts, etc., and; organization.
4. Increased use of framework contracts is a key indicator of procurement efficiency and a GOK priority. With the PPOA, we have developed milestone indicators to measure progress towards implementation of framework contracts.
5. In order to successfully carry out its mandate, as well as to support transparency, PEs must report their procurement activities (exceeding Ksh 5 million or approximately $63,000) to PPOA on a quarterly basis. The figures reported herein reflect submissions received during the quarter that were accepted by PPOA. Submissions by PEs that were rejected due to inaccuracies can be expected to be re-submitted at a later date; thus, the figures here can be expected to increase retrospectively. While PPOA identifies over 31,000 GOK PEs in total, the vast majority engage in only low value procurement, if at all. Statistics on procurement volume are weak, however one can consider the following to be “key” groups of procuring entities: 30 Ministries or Departments; 90 Local Authorities; 180 State Corporations and Universities. However, not all PEs in these key groups have substantial procurement activity.
6. – 10. These five indicators were chosen in consultation with the GOK and selected for relevance and feasibility. The comprehensive assessment of KEMSA, which was completed in April 2008, established baselines and targets.
8. – 9. The baseline information provided herein was based on the outturn of the annual KEMSA procurement, conducted each year in June and July. The baseline information reflects the June-July 2007 procurement.
9. Indicator drugs are a selected list of drugs that serve as a representative sample when studying stock-outs. There are 10 indicator drugs. They were selected on the basis of being commonly used commodities in virtually all levels of health facilities. A sample is necessary because of the impracticability of analyzing the stock status of all commodities which are of course very numerous.
10. This indicator monitors the “ground level” impact of reforms in KEMSA warehouse operations (specifically in warehousing, logistics/distribution and customer service) by tracking the number of days for an order for medical commodities from a provincial or district hospital to be fulfilled. It is a key indicator of efficiency in operations. Data is no longer reported with a one-quarter lag.