Agentic platform probe — Jacob Urmanec

Summary

Portfolio demonstration: a short, freezeable probe of agentic platform patterns. Domain vocabulary is MarTech-shaped and illustrative. The demo walks through agent → grounded Q&A → idempotent API.

Problem

Marketing operations teams need to turn high-level campaign intents into concrete, constraint-checked next steps (channel, timing, theme) for specific segments, while also answering questions about past guidance with evidence. Doing this reliably at even a small scale requires an agent that can call tools, a retrieval path that fails closed when evidence is insufficient, and an API that does not re-run expensive generation on retries.

Approach

Three experiments (E1-E3) — Python, LangChain, and a thin AWS floor.

E1 — Tool-using agent

A fake marketing-ops workflow: brief + segment stub, four stub tools, one journey step. The model proposes; constraint validation runs in code before any side effect. Happy path (allowed channel) and fail path (disallowed proposal rejected) are frozen and CLI-demoable.

Example CLI (happy path)

python -m e1.run \
  --brief "Promote a new fiber upgrade to existing residential customers in the Northeast." \
  --segment-id seg_residential_northeast_active \
  --path happy

Example result (JSON)

{
  "segment_id": "seg_residential_northeast_active",
  "proposal": {
    "channel": "email",
    "message_theme": "fiber_upgrade_promo",
    "timing": "immediate",
    "rationale": "Email is allowed by policy and suitable for promoting upgrades to active residential customers in the Northeast."
  },
  "validation": {
    "approved": true,
    "violations": []
  },
  "status": "approved",
  "tool_calls": [
    {
      "name": "get_segment",
      "args": {
        "segment_id": "seg_residential_northeast_active"
      },
      "ok": true
    },
    {
      "name": "get_audience_rules",
      "args": {
        "segment_id": "seg_residential_northeast_active"
      },
      "ok": true
    },
    {
      "name": "propose_journey_step",
      "args": {
        "brief": "Promote a new fiber upgrade to existing residential customers in the Northeast.",
        "segment_id": "seg_residential_northeast_active",
        "channel": "email",
        "message_theme": "fiber_upgrade_promo",
        "timing": "immediate",
        "rationale": "Email is allowed by policy and suitable for promoting upgrades to active residential customers in the Northeast."
      },
      "ok": true
    },
    {
      "name": "validate_constraints",
      "args": {
        "segment_id": "seg_residential_northeast_active",
        "channel": "email",
        "message_theme": "fiber_upgrade_promo",
        "timing": "immediate"
      },
      "ok": true
    }
  ]
}

E2 — Thin grounded Q&A

Lexical retrieve over a fixed markdown corpus. Score below a minimum →status: unknown with no model call. Otherwise grounded generation with citations, or refuse. In-corpus and out-of-corpus paths are frozen. Embeddings and a vector store stay deferred; the cite / fail-closed contract is written so a later retrieval swap need not change the JSON shape.

Example CLI (happy path)

python -m e2.run \
  --query "Is SMS allowed for residential Northeast active segments?" \
  --path happy

Example result (JSON)

{
  "query": "Is SMS allowed for residential Northeast active segments?",
  "answer": "No, SMS is not allowed for residential Northeast active segments.",
  "citations": [
    {
      "doc_id": "channel-policy.md#1",
      "path": "channel-policy.md",
      "score": 5.0,
      "excerpt": "## Allowed channels\n\n- **Email** — allowed for residential active segments when the customer has email consent on file.\n- **In-app** — allowed for residential active segments with an active app session or push token.\n- **SMS** — **not allow…"
    },
    {
      "doc_id": "channel-policy.md#2",
      "path": "channel-policy.md",
      "score": 4.0,
      "excerpt": "## Segment note\n\nFor the illustrative segment `seg_residential_northeast_active` (residential, Northeast region, active accounts):\n\n- SMS is disallowed (`rule_no_sms_residential`).\n- Prefer email for upgrade and billing notices.\n- Prefer in…"
    },
    {
      "doc_id": "channel-policy.md#0",
      "path": "channel-policy.md",
      "score": 3.0,
      "excerpt": "# Channel policy — residential segments\n\nIllustrative marketing-ops policy for audience segments that are **residential** and **active**."
    }
  ],
  "status": "answered",
  "retrieval": {
    "method": "lexical",
    "hit_count": 3,
    "min_score": 2.0
  }
}

E3 — Idempotent RAG API

HTTP API → Lambda → DynamoDB keyed by idempotency_key, wrapping E2 only (not the E1 agent loop). Hit returns the stored answer; miss runs retrieval/generation once and persists. Happy, fail (refuse), and idempotent retry paths are verified and shown below.

Example request (happy)

JSONcurl

{
  "query": "Is SMS allowed for residential Northeast active segments?",
  "idempotency_key": "demo-happy-001"
}

curl -s -X POST "$API/ask" \
  -H "Content-Type: application/json" \
  -d '{"query":"Is SMS allowed for residential Northeast active segments?","idempotency_key":"demo-happy-001"}'

Example response (happy)

{
  "query": "Is SMS allowed for residential Northeast active segments?",
  "answer": "No, SMS is not allowed for residential Northeast active segments (disallowed per `rule_no_sms_residential`).",
  "citations": [
    {
      "doc_id": "channel-policy.md#1",
      "path": "channel-policy.md",
      "score": 5.0,
      "excerpt": "## Allowed channels\n\n- **Email** — allowed for residential active segments when the customer has email consent on file.\n- **In-app** — allowed for residential active segments with an active app session or push token.\n- **SMS** — **not allow…"
    },
    {
      "doc_id": "channel-policy.md#2",
      "path": "channel-policy.md",
      "score": 4.0,
      "excerpt": "## Segment note\n\nFor the illustrative segment `seg_residential_northeast_active` (residential, Northeast region, active accounts):\n\n- SMS is disallowed (`rule_no_sms_residential`).\n- Prefer email for upgrade and billing notices.\n- Prefer in…"
    },
    {
      "doc_id": "channel-policy.md#0",
      "path": "channel-policy.md",
      "score": 3.0,
      "excerpt": "# Channel policy — residential segments\n\nIllustrative marketing-ops policy for audience segments that are **residential** and **active**."
    }
  ],
  "status": "answered",
  "retrieval": {
    "method": "lexical",
    "hit_count": 3,
    "min_score": 2.0
  },
  "cache": {
    "hit": false
  }
}

Example request (fail)

JSONcurl

{
  "query": "What is the 2099 lunar billboard advertising budget for AcmeCorp?",
  "idempotency_key": "demo-fail-001"
}

curl -s -X POST "$API/ask" \
  -H "Content-Type: application/json" \
  -d '{"query":"What is the 2099 lunar billboard advertising budget for AcmeCorp?","idempotency_key":"demo-fail-001"}'

Example response (fail)

{
  "query": "What is the 2099 lunar billboard advertising budget for AcmeCorp?",
  "answer": "I do not have enough information in the marketing-ops corpus to answer that question.",
  "citations": [],
  "status": "unknown",
  "retrieval": {
    "method": "lexical",
    "hit_count": 0,
    "min_score": 2.0
  },
  "cache": {
    "hit": false
  }
}

Example request (idempotent retry)

JSONcurl

{
  "query": "Is SMS allowed for residential Northeast active segments?",
  "idempotency_key": "demo-happy-001"
}

curl -s -X POST "$API/ask" \
  -H "Content-Type: application/json" \
  -d '{"query":"Is SMS allowed for residential Northeast active segments?","idempotency_key":"demo-happy-001"}'

Example response (idempotent retry)

{
  "query": "Is SMS allowed for residential Northeast active segments?",
  "answer": "No, SMS is not allowed for residential Northeast active segments (disallowed per `rule_no_sms_residential`).",
  "citations": [
    {
      "doc_id": "channel-policy.md#1",
      "path": "channel-policy.md",
      "score": 5.0,
      "excerpt": "## Allowed channels\n\n- **Email** — allowed for residential active segments when the customer has email consent on file.\n- **In-app** — allowed for residential active segments with an active app session or push token.\n- **SMS** — **not allow…"
    },
    {
      "doc_id": "channel-policy.md#2",
      "path": "channel-policy.md",
      "score": 4.0,
      "excerpt": "## Segment note\n\nFor the illustrative segment `seg_residential_northeast_active` (residential, Northeast region, active accounts):\n\n- SMS is disallowed (`rule_no_sms_residential`).\n- Prefer email for upgrade and billing notices.\n- Prefer in…"
    },
    {
      "doc_id": "channel-policy.md#0",
      "path": "channel-policy.md",
      "score": 3.0,
      "excerpt": "# Channel policy — residential segments\n\nIllustrative marketing-ops policy for audience segments that are **residential** and **active**."
    }
  ],
  "status": "answered",
  "retrieval": {
    "method": "lexical",
    "hit_count": 3,
    "min_score": 2.0
  },
  "cache": {
    "hit": true
  }
}

Python 3.11+; LangChain + xAI for the frozen CLIs
Explicit bind_tools loop (E1); lexical retrieve + cite/refuse (E2)
AWS SAM: API Gateway HTTP API, Lambda, DynamoDB idempotency (E3)
Slice specs, freeze notes, and deferred proposals kept out of frozen paths

Key judgment: put the LLM where drafting helps; keep validation, retrieval gates, and retry semantics in deterministic code and infrastructure.

Results & limits

E1–E3 are frozen and demoable: local CLIs for the agent loop and grounded Q&A, plus an idempotent HTTP API over the RAG path. Together they show tool orchestration with constraint checks, cite-or-refuse answers, and happy / fail / cache-hit retry semantics on the cloud floor.

Links

GitHub repository

Reach out via Contact to discuss this project or schedule a walkthrough.