Implementing AI Design with Graph Engineering

Code Factory plans to focus on designing AI agent architectures to perfectly match objectives through graph engineering. To this end, specific learning topics are presented using the process of researching Airbnb accommodations for travel as an example.

The main goal is to understand the concepts and practical implementation methods of graph engineering, thereby helping AI agents perform complex tasks efficiently. This goes beyond simple theoretical learning, relying on real data and scenarios.

Code Factory emphasized building and validating a single pipeline before creating the entire graph. This process involves using real web data to investigate a single region and confirm that the pipeline functions correctly.

This approach enables an efficient strategy mixing parallel and serial tasks, and is designed to incorporate 'human-in-the-loop' for final verification through human intervention at the final approval stage. This ensures the stability and reliability of the graph generation process.

Graph engineering is not a technique suitable for environments that directly execute AI, such as Codex. Instead, it operates with agents functioning independently and transferring states to each other. Each node is considered an independent agent, designing data and transferring states through API calls.

This method ensures smooth data processing between independent environments and contributes to AI agents performing their roles within complex workflows and effectively integrating their results. This increases the flexibility and scalability of the overall system.

A unique run ID is generated to execute Airbnb research, setting up the project environment. This is done by structuring folders into three types: markdown folder, raw folder, and log folder.

Dividing folders this way enables systematic project management, improves work efficiency by storing and managing files according to each data type, and facilitates identifying the cause of issues later. This helps establish a stable development environment.

In processes that produce deterministic outcomes, such as environment setup, it is more efficient to run a general script instead of using a Large Language Model (LLM). This helps prevent unnecessary token consumption and reduces costs.

Therefore, by operating nodes as deterministic and non-deterministic, a strategy is employed to maximize token efficiency and improve overall system efficiency. This allows for the application of the optimal execution method tailored to the characteristics of each node.

Before expanding the graph and introducing parallel tasks, it is crucial to first execute and validate the pipeline with only one region. Running everything in parallel from the start can lead to unnecessary token consumption and long processing times.

Therefore, a strategy is recommended to gradually expand after confirming the stability of the pipeline through single-region testing. This saves development time and costs, and enables efficient graph construction.

Code Factory explained how to use Oxylabs' Web Scraper API to efficiently extract complex web data. This API can access public web pages, render JavaScript-driven pages, and handle proxy operation and result return in a single API call.

This allows users to extract accurate data in real-time without worrying about the complexities of web scraping. This is especially useful when collecting data from dynamic web pages and simplifies the overall data collection process.

The first step in data collection is to set a prompt that clearly specifies the format. A request is made for an Oxylabs API search for a single area in Gangnam-gu, explicitly stating not to use dummy data or arbitrarily generated accommodations to prevent hallucination.

This contributes to increasing the accuracy and reliability of data collection by precisely setting input files and travel conditions, and explicitly specifying the result storage path and file format. These pre-settings are an important step in ensuring data quality.

To execute web scraping tasks using Oxylabs CLI, you need to create and set up a username and password in the Oxylabs users tab. This process secures API call permissions and proceeds with actual scraping.

After receiving data, an HTTP 200 status code is checked to confirm that the data was transferred correctly, ensuring stable browser accessibility in the graph agent environment. This is a critical step in increasing the reliability of the data collection process.

When an error occurs in a data pipeline, it's important to design for fault tolerance so that you can recover from the point of interruption instead of restarting from scratch. By saving and referencing the original responses received from Oxylabs, if a problem occurs in the next graph node, the task can be restarted by referencing the raw file of the previous node.

This node-specific state design increases execution efficiency and reduces wasted time. Additionally, log files allow tracking execution processes and diagnosing problems, enhancing overall system stability.

A strategy to reduce dependency on Large Language Models (LLMs) by writing code to programmatically validate the next file was presented. This includes deterministic verification techniques such as checking for the existence of markdown files, successful JSON parsing, and confirmation that the searched region is 'Gangnam-gu'.

Using LLMs can lead to unnecessary token consumption and time, so direct program code is utilized in these validation steps to improve speed and accuracy. For example, code to automatically check data conditions (region name, rating, etc.) is written to automate and streamline the validation process.

This method contributes to securing both the stability and economic efficiency of the entire workflow by processing parts that can be reliably verified by a program without relying on AI model judgment.

A multi-agent parallel processing architecture is constructed by having the main agent create and instruct three independent sub-agents for regions like Gangnam, Mapo, and Jongno to run in parallel. This conducts data research for each region simultaneously, shortening overall processing time and maximizing efficiency.

Each agent independently collects and processes data, and finally, the main agent integrates the results to generate a context report. This method is particularly useful for large-scale data processing or complex research tasks.

Using the Oxylabs Scraper API allows for independent research without needing to worry about browser control. This is because the API automatically handles complex processes like Chrome control to retrieve the necessary results.

This independent execution environment prevents fault propagation by ensuring that a failure in a specific task does not affect other tasks. Furthermore, retries for individual tasks are easier, improving the overall system's stability.

While scraping tasks are in progress, real-time statistics can be checked via the dashboard. This allows for a clear understanding of task status by identifying the number of requests and data usage.

Once the parallel regional research is complete, the graph proceeds to independently validate each set of results. In this step, the code previously used for single pipeline validation is reused to confirm the validity of the results for each region.

Each region's results are generated as independent reports and are marked as 'pass' if validation is successful. This is an important process for ensuring the reliability of multiple parallel processed data sets and for determining whether to proceed to the next stage.

After data validation is complete, a Large Language Model (LLM) performs the task of recommending Airbnb accommodation candidates based on the results. This process utilizes the LLM's judgment to provide the user with optimal choices.

The recommendation results include detailed information such as rankings and list IDs, and these results are saved as files to transfer the state to the next node. This is a crucial step in ensuring that information flows systematically within the AI agent workflow.

The 'Human-in-the-loop' stage is applied to generate a final approval request only when the AI agent's recommendation verification result is 'passing'. This process involves a person directly reviewing the artifact (result) generated by the AI and deciding whether to approve it.

At this stage, users can reject or request modifications if necessary, and the workflow can be re-executed accordingly to enhance data accuracy and user satisfaction. The goal is to ensure the reliability of the final outcome through human intervention.

The agent graph workflow generally begins by configuring the graph you want to work with. Afterward, a single pipeline is built and tested directly to verify its operation.

Based on the validated pipeline, agent nodes are created and arranged for parallel tasks, validation, and Human-in-the-loop nodes, thereby creating an efficient AI agent environment. This process allows for the construction of a robust system capable of systematically handling complex tasks.