Executive Summary

Continuing to release public datasets on data.sandiego.gov according to the Open Data Policy is a major focus of the Performance & Analytics Department. But fulfilling the Open Data Policy is just part of a larger mission of helping people inside and outside of the City get and use data.

Toward that larger mission, P&A’s Data & Analytics team has, in the last fiscal year, initiated a data science practice, hired the City’s first Data Science Program Coordinator, built a custom Open Data Portal, built technology that will eventually allow management to receive updates on key metrics, and continued community outreach. Below are summaries of our major areas of work during the past year.

Data releases

The Open Data Policy mandates that all high value public datasets in the City’s inventory be released by 2020. 44 datasets were released with the launch of the Open Data Portal in Fiscal Year 2016. 30 more datasets were added during Fiscal Year 2017. Through FY 2017 the City has released 46 percent of the datasets in the inventory, which exceeds a goal of 27 percent set as a P&A Key Performance Indicator in Volume II of the City’s FY 2018 Budget.

The inventory is not static; Performance & Analytics staff work with colleagues in departments across the City on projects to improve processes, gather survey data, increase efficiency, track performance, and improve data collection and management. As new high-value public datasets are discovered during this work, they are evaluated and added to the list. The inventory originally included 136 datasets and has grown to 155.

Each dataset on the inventory has a target fiscal year for release. This schedule is part of the inventory, which exists as a downloadable dataset on the Open Data Portal.

Data Science program

Performance & Analytics hired a Data Science Program Coordinator and has started a Data Science Program. The Data & Analytics team has begun to engage with City departments to create data-driven solutions to challenges they face. Arnaud Vedy, P&A’s data scientist, has already helped Development Services Department, Environmental Services Department, City Treasurer, Risk Management, and the Get it Done program.

Custom Open Data Portal

When the Open Data Portal first launched, the City purchased a portal product from a vendor. That portal product, called DKAN, had many advantages, including that it was easy to get up-and-running so we could launch the Open Data Portal as quickly as possible.

The desire to have more control over the portal as well as save the City money led the Data & Analytics team to build a custom Open Data Portal in Fiscal Year 2017.

Completing our automation pipeline

Automation is a crucial component of the City’s Data & Analytics program. This is because data should be timely and reliable, and no City staff should be responsible for remembering to run a report on a daily, weekly, or monthly schedule and manually upload it to a data portal. Much of the automation pipeline for publishing City data on the Open Data Portal was in place when the portal launched last Fiscal Year, including our connections to City data sources and a tool that transformed the data for publication.

In Fiscal Year 2017, the Data & Analytics team focused on automation, specifically the scheduling component that allows data updates to run on a dedicated server on a specific schedule without any human involvement. When the first portal was launched, the City relied on a proprietary software to transform data for publication, but this year has transitioned to a scheduling component written in the Python programming language that works well with the new custom data portal.

Next steps

Besides releasing additional data, the Data & Analytics team has the following objectives for Fiscal Year 2018:

Continue to improve the custom Open Data Portal

The Data & Analytics team tracks how people use the portal and collects feedback on the design and functionality. Because the portal was built in-house, the team can maintain a list of improvements and bugs in a Github repository and make fixes without the additional costs that would come with asking a vendor to make changes.

Some of the improvements on the list for FY18 include:

• Expanding data preview options beyond the simple rows-and-columns view that is currently available on each dataset page to include maps and charts where applicable
• Giving users the ability to create a chart on a dataset page without having to download the data, and offering summary metrics about the data at the top of the dataset page
• Reorganizing geospatial data from a generic category of “GIS” (Geographic Information Systems) to other topical categories that will be more relevant
• Adding a page with portal and dataset usage numbers

Increase data portal users

The data portal has three target user groups: technologists, City employees, and residents. The City knows from usage numbers that people are visiting the portal and downloading data, but the Data & Analytics team is always looking for ways to increase usage. Strategies for getting more users include making the portal easier to navigate, adding features that increase relevance for each user group, and continuing to promote the portal and all the great data it contains in as many different forums as possible.

Since the portal launched in June 2016, datasets have been downloaded 140,000 times for a total of 1.2 terabytes of data downloaded.

Develop data microservices and APIs

Offering portal users granular data (i.e. all requests for service made through the pilot Get it Done program) helps residents and City staff take a deep dive into the data to find solutions and improve life in San Diego.

Now that the data portal features many granular, automatically updated datasets, Data & Analytics can add value by delivering data in custom formats to different users. For example, staff in a City Council office might like to know how many potholes were reported to Get it Done in the district and get those numbers delivered via email.

Continue developing the data science program

Data & Analytics will work with City departments to uncover and solve challenges. The team is also committed to strengthening connections with the higher education community and create a partnership that puts our data in front of data science students.