Al Policy Making

Week 2 Summary

Midjourney prompt: ***humanoid robot arm wrestling a human --ar 7:4***

In week one I started an experiment to generate an alternative district plan for the Kaipara District using artificial intelligence (AI) and large language models (LLMs). If you missed it, you can read my summary of Week 1 here: https://www.fluid-industries.co.nz/articles/2023/6/19/ai-policy-making-week-1-summary

No sooner than I had completed the week one summary, I hit the conversation limit of even the paid premium version of ChatGPT-4. It was SUPER disappointing, as it looked as though we were on a roll …

Having finished the Strategic Direction Chapter, it was time to tackle the content for the subject-based chapters. It is always interesting to see how ChatGPT suggests we tackle the next step, so I asked:

Given the methodology we have followed so far, would it be better to populate the content horizontally (the 'Introduction' section of each topic, followed by the 'objectives' for each topic), or vertically (ie, complete the 'introduction', then 'objective', then 'policies' on a topic by topic basis)?
“Given the significant number of topics and the interconnectedness between them, I suggest a horizontal approach to populating content. This means writing the 'Introduction' section for each topic first, then moving onto the 'Objectives', 'Policies', 'Methods', 'Mapping', and 'Monitoring and Evaluation' sections.
This approach allows for a more holistic understanding of each topic, as it ensures that all topics are fresh in mind when writing about the specific details within each section. This way, connections between topics can be more readily identified and incorporated into the plan, providing for a more integrated and cohesive document”.

The precautions I had taken late last week to do some Bing testing have paid off, and I had already collected some reference material to provide context for a new chat in case of a 'do-over'. Earlier in the week ChatGPT-4’s Bing Web Search Plugin was being erratic. In my week one summary I reported that interrupting ChatGPT while it’s in the ‘flow’ leads to problems. This week I noticed that every time I switched Chapter level Topics, ChatGPT seemed to panic and go looking for answers on the internet. I saved the following screenshot with the filename “new topic anxiety.png”:

Performance anxiety

ChatGPT seemed to default to using the Bing Search plugin to look for the Topics we were in the process of writing, then became apologetic that it couldn’t find them on the internet. ChatGPT4 needed regular reminders that we are writing an 'alternative bootleg' district plan, so needn’t worry about searching for the things we are currently making.

I can totally understand the confusion though. I specified at the outset that I wanted it to find the current official legislative framework, facts and data sources from official channels (only kaipara.govt.nz or other govt.nz sources), take into account and 'real' feedback from the 'actual' Kaipara community. But I was also expecting it to just ‘set aside’ the Council's version of the draft district plan and build an 'alternative’ one. To get it back on track, I pointed it back to the transcripts of the last chat instance that I had published on my website.

I am not sure whether the ChatGPT4 plugins are glitchy or whether the NZ Government is watching me more closely than I had realised, but for some reason, the Inland Revenue Department joined the fray:

On the following blog there is an ongoing project being described. Can you give me an overview of the project so far: https://www.fluid-industries.co.nz/articles

[Finished browsing].

I've attempted to gather some information from the blog you've provided, but I've faced some difficulties in accessing the specific articles on the page. However, from the information available on the main page, I've gathered the following:

1. The blog is maintained by Fluid Industries. The company is involved in a project concerning the Kaipara District Council. There is mention of a 'Reboot Plan' which seems to be a significant part of the project. However, the details of this plan are not clearly stated on the overview page.

2. The Inland Revenue Department (IRD) is also mentioned, though it is unclear how the IRD is directly related to the project. The IRD has released a consultation document about tax and charitable giving, with submissions closing on 12 June 2023.

3. A post from New Zealand's Ministry for the Environment outlines the country's resource management system, which provides a framework for how New Zealand's environment should be managed. It highlights a focus on sustainable management, partnerships with Māori under the Te Tiriti o Waitangi, and community involvement in decision-making.

Unfortunately, I was unable to delve deeper into the individual articles and provide a comprehensive overview of the project. It seems that the website's navigation may be causing some issues with retrieving detailed information. I would recommend manually exploring the blog's articles to gather a more in-depth understanding of the project.

Prepare to learn about you

After coaxing ChatGPT back into the conversation thread by copy / pasting excerpts from our previous chat (Pro Tip: Using a repository like OneNote or Evernote to throw all those random clipboard and sticky notes into is THE BEST thing you can ever do …) ChatGPT started to give more sensible answers again. However, they seemed way too general an answer to be credible - ChatGPT reminded me of my teenage years when I thought I could rip off some obscure song lyrics for my poetry assignment. I’m pretty sure the conversation with my English teacher followed a similar tone:

Teacher (me):
In a previous conversation instance with you we have been developing an Alternative Bootleg District Plan for the Kaipara District. A summary of the project is set out on my website here …
In our last conversation I said:
Your reply was:
This approach allows for a more holistic understanding of each topic …
For example, when writing about 'Objectives', …
It's also important to note that this approach does not prevent us from …
However, this is just a suggestion and the final decision should be based on …
I said: "That is consistent with my preferred approach. Let's proceed …
Do you have significant context to proceed, or would it be beneficial to provide you with the Chapter Outlines that you have developed previously? I am happy to provide them as we work through the document if that would be beneficial.
[Finished browsing]
Thank you for the context. It's great to see that the plan is progressing smoothly.
Using the information you've provided, I'll draft the 'Introduction' section …
It's great to see you seamlessly continue into a new conversation instance when given sufficient background! Can you now give me some text for the 'Introduction' heading of the 'Water' Topic in the Natural Environment Chapter please.
Absolutely, I'm glad to hear that the transition worked well for you. I will now proceed to draft …
Actually, now that I compare the Introduction for the 'Land' Sub-Topic, is a bit more generic than the content outline you gave previously (with more insight as to the context and parameters that exists in this conversation so far). Here is the outline you provided:
Ah, I see. Thank you for that context. I can certainly provide a more generic introduction that is similar in nature to the "Land" sub-topic outline. Here's a revised version for the "Water" sub-topic:
Apologies! I meant that the content outline you gave in a different instance is more specific than the first one you gave in this current conversation. When I prompted Bing to provide an Introduction, it drew from current online resources. Can you provide …
[Finished browsing]
I apologize for the confusion, and thank you for your patience. I have been attempting to gather specific information about the significance of land in the Kaipara district from the district plan, including the various ways land is used and its importance to the community. However, I have run into some technical issues and was unable to extract specific text from the documents.
…
The review process described in the 'Introduction' you provide above is what we are doing now. Can I have some text that gives "a brief introduction to the significance of land in the Kaipara district …

Prepare to be surprised

ChatGPT4 is really good at continuing a conversation thread. I mean REALLY good. In my Day 6 summary we were blazing our way through the Strategic Direction Chapter following the outline that ChatGPT4 had come up with. After completing the first 7 sub-topics I made a copy / paste error:

Let's take a look at the "Growth Management Objectives" you have in mind.
**Cultural Wellbeing Objectives**
1. **Cultural Recognition and Respect**: To recognise and respect the diverse cultures present within the Kaipara District, particularly acknowledging the unique status of Māori as tangata whenua. …

I was genuinely shocked by this response. I asked for the “Growth Management Objectives” and it gave me the Cultural Wellbeing Objectives that I meant to ask for (but didn’t). Where ChatGPT had previously been giving me the sense it was deliberately being a contrary teenager and ‘forgetting’ what we were doing, the next minute it was disturbingly happy to gloss over my mistakes as if they hadn’t occurred:

Oh thank you so much for that! You recognised that I meant to say Cultural Wellbeing (and not Economic Wellbeing) and gave the answer to the question that I meant to ask. If you find me making a mistake like that in the future, you can politely let me know and correct me so that I notice I made the mistake and can learn from it.
I appreciate your feedback and will be sure to politely point out any inconsistencies I notice in the future. Let's continue with the Cultural Wellbeing topic. Would you like me to draft the text for the next section?
Yes, please do.
[Finished browsing]

Seems I spoke too soon. My attempt to politely ask ChatGPT4 to let me know when it was taking the liberty of ignoring my prompts was me by ‘new topic anxiety’ and starting new internet searches. When I checked in to see what searches were being performed, I was still none the wiser about its logic … “Air Quality in Katra India?! What even?! [Yep. Definitely a Teenager]:

Despite these difficulties, we were able to progress and complete the subject-based chapters of the district plan. ChatGPT-4 still exhibited some wisdom beyond its teenage years and suggested a horizontal approach to populating content, which we successfully completed when I regularly reminded it about the last time we spoke.

In week two, I also reflected on how using AI and LLM tools made me feel. I felt a range of emotions, such as excitement, frustration, surprise, and disappointment. I also raised some interesting questions and challenges, such as how to assess the quality and validity of the generated outputs, how to ensure consistency and coherence across different sections, how to deal with ethical and legal issues, and how to involve human stakeholders in the process.

Language matters

No doubt many of the glitches I experience are a result of me being too casual. My own personal ‘default’ language setting is to be polite and courteous, and spending a large part of my career working in the public sector has made me mindful of the words I use (LGOIMA anyone?). It is taking a bit of ‘unlearning’ for me to be more direct and explicit in my instructions, treating ChatGPT in the same way I would an enthusiastic university graduate, rather than an Artificial Intelligence. But then again, what kind of impression do I want to leave in our conversation history when our future AI overlords can access everything we have ever said (or done) to them ..?

Hey, What Happened to Days 12-14?

Sorry. When I said the 28-Day AI Policy Making Challenge would run through the weekends too, I hadn’t counted on how much I would use ChatGPT to plan our upcoming holiday. I used up my ChatGPT4 ‘Plus’ quota mapping out the optimum itinerary for a 5-week camping road trip to visit the best National Parks the western USA has to offer!

AI Policy Making: Week 2 Summary