Developing a web scraping tool for the Australian Adaptation Database

Australia faces emerging climate risks that require coordinated adaptation efforts across government, the private sector and civil society. The Australian Adaptation Database is a world-first national stocktake of climate change adaptation activities, providing crucial insight into where and how adaptation is occurring across different scales, sectors and geographic locations.

Each example is categorised against key criteria such as location, type of organisation implementing them and climate hazard they are responding to. This data is visualised online on a publicly available dashboard.

As a repository of examples of adaptation work, it helps practitioners share new approaches and learn from one another in planning adaptation. The database has already proven valuable, informing the National Climate Risk Assessment, National Adaptation Plan, and policy work of organisations including the Australian Local Government Association and Victorian Council for Social Services. However, to expand, improve and scale up the project, new methodologies and increased efficiencies are required.

To date much of the project’s data collection, entry and management has been done manually by the research team. Current gaps in data collection mean that local-scale adaptation, private sector adaptation and adaptation led by Traditional Owners are underrepresented. To expand, improve and scale up the project new methodologies and increased efficiencies are required.

The aims of the collaboration with MDAP are to:

• Develop a web scraping method to expand data collection
• Automate data updates
• Automate data coding and cleaning.

MDAP's expertise is essential for implementing these improvements. They will develop web scraping techniques to review websites of local government areas. MDAP will also automate data updates by flagging possible changes to existing entries and automate data coding and cleaning processes.

Their involvement will incorporate large language models to search the web with greater nuance, capturing adaptation activities that avoid conventional terminology . MDAP will also support more complex data visualisation and improvements to the AAD website.

These new capabilities will affect a step change in the data coverage and quality of the Australian Adaptation Database, making it an invaluable resource to the Australian climate policy community and researchers. It will also provide a template for policy makers and researchers in other countries seeking to replicate this work.