Integrating OpenAI o3-mini or Grok into a Java Spring Boot application is straightforward because both APIs are OpenAI-schema compatible and the Spring AI project provides a mature abstraction layer. The harder problem is enterprise-grade tool orchestration: routing between models, handling retries, managing secrets, and wiring LLM tool calls into existing Spring service beans without turning your codebase into a prototype.
Analysis Briefing
- Topic: LLM Tool Orchestration in Java Spring Boot
- Analyst: Mike D (@MrComputerScience)
- Context: Originated from a live session with Grok 4.20
- Source: Pithy Cyborg | Pithy Security
- Key Question: How do you wire o3-mini or Grok into Spring Boot without it becoming a mess?
Configuring Spring AI for OpenAI o3-mini and Grok in One Project
Spring AI’s OpenAiChatModel accepts a configurable base URL and API key, which means switching between OpenAI and Grok requires only environment-level configuration changes, not code changes. Both providers speak the same schema. Your tool definitions, message structures, and response handling are identical across both.
Add the Spring AI dependency to your pom.xml:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
<version>1.0.0</version>
</dependency>
Configure both providers in application.yml using Spring profiles:
spring:
profiles:
active: openai
---
spring:
config:
activate:
on-profile: openai
ai:
openai:
api-key: ${OPENAI_API_KEY}
chat:
options:
model: o3-mini
---
spring:
config:
activate:
on-profile: grok
ai:
openai:
api-key: ${XAI_API_KEY}
base-url: https://api.x.ai/v1
chat:
options:
model: grok-4
Switching from o3-mini to Grok in CI or production is now a profile flag, not a code change. This matters for enterprise deployments where model swaps happen during incident response or cost optimization cycles and you cannot afford a redeploy to test an alternative provider.
Never hardcode API keys in application configuration files. Both ${OPENAI_API_KEY} and ${XAI_API_KEY} should resolve from environment variables injected by your secrets manager, whether that is AWS Secrets Manager, HashiCorp Vault, or Kubernetes secrets. Claude Code security risks examines exactly the class of credential exposure that happens when API keys drift from secrets management into config files during rapid prototyping.
Wiring LLM Tool Calls Into Spring Service Beans
Spring AI’s function calling support lets you register any Spring bean method as a callable tool using the @Bean and FunctionCallback patterns. The model receives a tool schema, decides when to call it, and Spring AI handles the dispatch to your actual service method.
Define your tools as Spring beans:
@Configuration
public class LlmToolConfig {
@Bean
@Description("Retrieve the current account balance for a given customer ID")
public Function<AccountBalanceRequest, AccountBalanceResponse> accountBalanceTool(
AccountService accountService) {
return request -> accountService.getBalance(request.customerId());
}
@Bean
@Description("Search the product catalog by keyword and return matching items")
public Function<ProductSearchRequest, ProductSearchResponse> productSearchTool(
ProductCatalogService catalogService) {
return request -> catalogService.search(request.keyword(), request.maxResults());
}
}
The request and response records must be serializable to JSON. Spring AI uses Jackson to convert between the model’s JSON tool arguments and your Java types automatically:
public record AccountBalanceRequest(String customerId) {}
public record AccountBalanceResponse(String customerId, BigDecimal balance, String currency) {}
public record ProductSearchRequest(String keyword, int maxResults) {}
public record ProductSearchResponse(List<ProductItem> items) {}
Wire the tools into your chat service:
@Service
public class LlmOrchestrationService {
private final ChatClient chatClient;
public LlmOrchestrationService(ChatClient.Builder builder) {
this.chatClient = builder.build();
}
public String processEnterpriseQuery(String userQuery) {
return chatClient.prompt()
.user(userQuery)
.functions("accountBalanceTool", "productSearchTool")
.call()
.content();
}
}
Spring AI handles the full tool call loop automatically: it sends the tool schema to the model, receives tool call requests, dispatches to your bean methods, injects results back into the conversation, and continues until the model returns a final response with no pending tool calls. You never write the loop manually.
Production Concerns: Retries, Timeouts, and Observability
A working prototype becomes an enterprise liability without proper resilience configuration. LLM API calls are slow, occasionally fail, and can run long enough to exhaust default Spring MVC timeouts during tool-heavy orchestration chains.
Configure Resilience4j for retry and circuit breaking on your LLM calls:
@Configuration
public class LlmResilienceConfig {
@Bean
public RetryConfig llmRetryConfig() {
return RetryConfig.custom()
.maxAttempts(3)
.waitDuration(Duration.ofSeconds(2))
.retryOnException(e -> e instanceof ChatClientException)
.build();
}
@Bean
public CircuitBreakerConfig llmCircuitBreakerConfig() {
return CircuitBreakerConfig.custom()
.failureRateThreshold(50)
.waitDurationInOpenState(Duration.ofSeconds(30))
.slidingWindowSize(10)
.build();
}
}
Wrap your orchestration service with the retry and circuit breaker annotations:
@Retry(name = "llmRetry")
@CircuitBreaker(name = "llmCircuitBreaker", fallbackMethod = "fallbackResponse")
public String processEnterpriseQuery(String userQuery) {
return chatClient.prompt()
.user(userQuery)
.functions("accountBalanceTool", "productSearchTool")
.call()
.content();
}
public String fallbackResponse(String userQuery, Exception ex) {
log.error("LLM orchestration failed for query: {}", userQuery, ex);
return "Service temporarily unavailable. Please try again shortly.";
}
For observability, Spring AI integrates with Micrometer out of the box. Token usage, latency, and tool call counts are automatically emitted as metrics when you include the Micrometer dependency. Wire these into your existing Prometheus or Datadog stack without additional instrumentation code.
Set explicit timeouts on your RestClient to prevent LLM calls from blocking Spring MVC threads indefinitely during tool orchestration chains that involve multiple sequential model calls:
@Bean
public RestClient.Builder restClientBuilder() {
return RestClient.builder()
.requestFactory(new HttpComponentsClientHttpRequestFactory(
HttpClientBuilder.create()
.setDefaultRequestConfig(RequestConfig.custom()
.setConnectionRequestTimeout(Timeout.ofSeconds(5))
.setResponseTimeout(Timeout.ofSeconds(120))
.build())
.build()
));
}
The 120-second response timeout accounts for deep tool orchestration chains where the model makes three to five sequential tool calls before returning a final answer.
What This Means For You
- Use Spring profiles to switch between o3-mini and Grok, not code branches. Profile-based model swapping lets you A/B test providers in staging without touching application logic.
- Register your existing Spring service beans as LLM tools directly. Spring AI’s
Functionbean pattern means you wire AI capabilities into real business logic rather than building parallel tool implementations that drift from production code. - Always configure Resilience4j retry and circuit breaking on LLM calls. LLM APIs return transient 429s and 503s in production at a rate that makes unprotected calls unreliable for enterprise SLAs.
- Set explicit HTTP response timeouts of at least 90 to 120 seconds for tool orchestration endpoints. Default Spring MVC request timeouts will kill legitimate multi-tool chains that happen to run long.
- Emit token usage as a Micrometer metric from day one. Cost tracking on LLM API calls is a first-class operational concern in enterprise deployments, and retrofitting it after launch is harder than instrumenting it at build time.
Enjoyed this deep dive? Join my inner circle:
- Pithy Cyborg → AI news made simple without hype.
- Pithy Security → Stay ahead of cybersecurity threats.