.webp)
In the past, when using OpenAI’s JSON mode there was no guarantee that the model output will match the specified and predefined JSON schema. In my view this really made this feature unreliable in a production environment, where consistency is important…
However, this has changed with something OpenAI refers to as Structured Outputs, which they describe as the evolution of JSON Mode.
As seen in the image below, the old JSON Mode is still available, parallel to the more recent Structured Outputs. However, as I have already noted back in Nov 2023, without the possibility to enforce / ensure model adherence to the structure, alternative avenues would work better.
Hence for this reason OpenAI recommend users to always use Structrued Outputs instead of JSON mode.
JSON is one of the most widely used formats in the world for applications to exchange data.
Structured Outputs offer several key benefits. They enhance type-safety by eliminating the need to validate or retry improperly formatted responses.
Additionally, safety-based model refusals can now be detected programmatically, making it easier to handle these situations.
Furthermore, they simplify prompting, as consistent formatting is achieved without the need for strong or specific prompts.
Function calling
Function calling is also an avenue to create structured output…here is some background…
Levels of Autonomy
In function calling with language models, the model gains a level of autonomy by determining whether to invoke a specific function or to rely on its default processing approach.
When the model identifies that a function is required, it autonomously switches to a more structured mode, preparing the necessary data and parameters for the function call.
This ability allows the language model to function as a mediator, efficiently handling the function while maintaining flexibility in processing requests.
The autonomy of AI can be viewed on a spectrum, where the degree of independence depends on how the system is designed.
By incorporating function calls into generative AI applications, we introduce not only structure but also an initial layer of autonomy, enabling the model to assess requests and decide whether it should use a function or provide an answer based on its default capabilities.
As AI technology progresses, this autonomy is expected to increase, with models becoming more independent and capable of handling increasingly complex tasks on their own.
This evolution enhances the ability of AI systems to take on more sophisticated responsibilities with minimal human intervention.
JSON mode evolved into structured outputs
As I mentioned earlier, Structured Outputs is the next step in the evolution of JSON mode.
While both guarantee the production of valid JSON, only Structured Outputsensure strict adherence to the defined schema.
Both Structured Outputs and JSON mode are supported across the Chat Completions API, Assistants API, Fine-tuning API, and Batch API.
Chain of thought
Below is a complete working example of Python which you can copy and paste into a Notebook. The code will prompt you for your OpenAI API Key…
This is an example where thegpt-4o-2024–08–06 model is made use of…the model is instructed to output the answer in a structured fashion, and also step-by-step to guide the user through the process.
You can ask the model to output an answer in a structured, step-by-step way, to guide the user through the solution.
Notice that this code only shows the Chain of Thought implementation, and how the sequence of reasoning is portrayed. In this example no JSON schema is given to the model to adhere to.
# Install the necessary packages!pip install openai pydantic# Import the modulesfrom pydantic import BaseModelfrom openai import OpenAIimport getpassimport json# Prompt the user for their OpenAI API keyapi_key = getpass.getpass("Enter your OpenAI API key: ")# Initialize the OpenAI client with the provided API keyclient = OpenAI(api_key=api_key)# Define the Step and MathReasoning classes using Pydanticclass Step(BaseModel): explanation: str output: strclass MathReasoning(BaseModel): steps: list[Step] final_answer: str# Use the OpenAI client to parse a chat completion for a math problemcompletion = client.beta.chat.completions.parse( model="gpt-4o-2024-08-06", messages=[ {"role": "system", "content": "You are a helpful math tutor. Guide the user through the solution step by step."}, {"role": "user", "content": "how can I solve 8x + 7 = -23"} ], response_format=MathReasoning,)# Extract the parsed math reasoning from the completionmath_reasoning = completion.choices[0].message.parsed# Convert the math reasoning to a JSON string and print itprint(json.dumps(math_reasoning.dict(), indent=4))
And the response…
{ "steps": [ { "explanation": "The equation given is 8x + 7 = -23. To solve for x, we first need to isolate the term with x on one side. We should move the constant on the left side, which is 7, to the right side by subtracting 7 from both sides.", "output": "8x + 7 - 7 = -23 - 7" }, { "explanation": "Subtracting 7 from both sides simplifies the equation. The left side becomes 8x because 7 - 7 is 0, and the right side becomes -30.", "output": "8x = -30" }, { "explanation": "Now that we have isolated the term 8x, we need to solve for x. Since 8x means 8 times x, we divide both sides of the equation by 8 to find the value of x.", "output": "x = -30 / 8" }, { "explanation": "Dividing -30 by 8 gives -3.75. This is the value of x that satisfies the equation.", "output": "x = -3.75" } ], "final_answer": "x = -3.75"}
Using structured outputs with response_format
Below is example code which is fully working and which can be used as-is within a Notebook.
Notice in this example, how the JSON schema is defined to which the model output must adhere.
!pip install openaiimport openaiimport jsonclass OpenAIClient: def __init__(self, api_key: str): """ Initialize the OpenAI client with the provided API key. """ openai.api_key = api_key def get_structured_output_with_schema(self, prompt: str, schema: dict): """ Call the OpenAI API and request a structured output based on the given prompt and schema. """ try: # Call the OpenAI API with response_format set to 'structured' and provided schema # Include the schema in the prompt prompt_with_schema = f"{prompt} \n\n Response should be in JSON format according to this schema: {json.dumps(schema)}" response = openai.chat.completions.create( model="gpt-4", # Use a model that supports structured output messages=[ {"role": "user", "content": prompt_with_schema} ], max_tokens=150, ) return response.choices[0].message.content.strip() # Extract and return the response except Exception as e: return f"Error occurred: {e}" @staticmethod def prompt_for_api_key(): """ Prompt the user to input their OpenAI API key. """ api_key = input("Please enter your OpenAI API key: ") return api_key# Example Usage:# Define the JSON schema for the structured outputschema = { "type": "object", "properties": { "benefits": { "type": "array", "items": { "type": "string" }, "minItems": 1, "maxItems": 5 } }, "required": ["benefits"]}# Prompt user for API keyapi_key = OpenAIClient.prompt_for_api_key()# Initialize the OpenAI clientclient = OpenAIClient(api_key)# Test the function with a sample promptprompt = "Provide a list of 3-5 benefits of using structured outputs in AI applications."result = client.get_structured_output_with_schema(prompt, schema)# Print the resultprint("Structured Output Response:")if result.startswith("Error occurred"): print(result) # Print the error message if there was an errorelse: print(json.dumps(json.loads(result), indent=2)) # Pretty print the JSON response
And the result from the query…
{ "type": "object", "properties": { "benefits": { "type": "array", "items": { "type": "string" }, "minItems": 1, "maxItems": 5, "default": [ "Structured outputs can handle complex relationships and dependencies within and between output elements.", "Structured output prediction can be more finely controlled, providing more accurate and specific results.", "Structured outputs help in better representation and abstraction of complex problems and tasks in AI.", "Utilizing structured outputs can reduce misunderstandings and errors in communications between different components of an AI system.", "Structured outputs can lead to better interpretation and understanding of the AI processes and their conclusions." ] } }, "required": [ "benefits" ]}
Finally
Language models now offer advanced functionalities such as function calling, structured output, and reasoning, enabling developers to offload complex tasks directly to the model.
This shift allows for streamlined workflows but also requires makers to carefully decide how much functionality they want to delegate to the model versus implementing within their application logic.
Offloading too much to the model can result in business applications becoming tightly coupled to specific models and their unique capabilities, making future updates or changes more challenging.
As models evolve, this coupling can hinder the flexibility of the application and limit its adaptability. On the other hand, by keeping the application logic more modular, developers can design model-agnostic systems capable of orchestrating multiple models for diverse tasks.
This approach enables greater flexibility, allowing businesses to integrate a variety of models as they emerge and adjust to future needs without being locked into a single solution.
