Apa saéné risiko sing duwé AI agent wewenan kanggo tumindak, lan kepiye carane nglimitasi kanthi aman?

Question

Accepted Answer

Yèn agent bisa **tumindak** — mbusak file, njalanake perintah shell, nelepon external API, ngetokake dhuwit — kesalahane (utawa prompt sing maling) dadi akibat ing donya nyata. Pertahanan kuwi **least privilege plus approval gates plus isolation**: mènéhi mung apa sing dibutuhake, merlokake konfirmasi kanggo apa-apa sing ora bisa dibali, lan jalanake ing panggonan sing ora bisa ngakibatake cilaka permanen.

## Risiko

```text
- DESTRUCTIVE actions  → rm -rf, DROP TABLE, force-push → irreversible data loss
- ARBITRARY shell      → run anything → privilege escalation, exfiltration
- EXTERNAL APIs/$$     → send emails, place orders, spend on cloud/LLM tokens
- SECRET exposure      → leak API keys / .env / tokens into logs or outbound calls
- PROMPT INJECTION     → untrusted input (a web page, an issue) hijacks the agent
```

Prompt injection kuwi sisih sing paling tajam: konten sing agent *maca* bisa ngandhut instruksi, dadi sembarang tool sing agent bisa nelepon bisa dicapai déning penyerang sing kontrol konten kasebut.

## Guardrails

```text
- LEAST PRIVILEGE / ALLOWLIST → only pre-approved commands & paths; deny by default
- APPROVAL FOR DESTRUCTIVE    → human confirms deletes, payments, prod writes
- SANDBOX / DRY-RUN           → preview the action; isolated container, no prod creds
- ISOLATED ENVIRONMENT        → ephemeral VM/branch; blast radius = throwaway box
- AUDIT LOGS                  → record every tool call + args for review
- NO SECRETS IN CONTEXT       → inject via env at runtime; never paste keys into prompts
- NETWORK LIMITS              → egress allowlist; block arbitrary outbound requests
```

```yaml
# Conceptual permission policy
tools:
  read_file:   { allow: true }                 # safe, reversible
  run_shell:   { allow: ["npm test", "git status"] }  # allowlist only
  delete_file: { allow: "ask" }                # human approval required
  send_email:  { allow: "ask", sandbox: true } # dry-run first, then confirm
network:
  egress: ["api.github.com"]                    # everything else blocked
```

Polane kuwi kepercayaan bertahap: **maca** gratis, **reversible writes** murah, **aksi sing ora bisa dibali utawa eksternal** merlokake konfirmasi, lan **dhuwit utawa produksi** merlokake isolation plus manungsa ing loop.

## Kenapa iki penting

Nilai agent asale saka ngbisakke Wong tumindak, nanging tumindak kuwi persis ngendi kesalahan sing percaya diri utawa prompt sing dibajak dadi data sing mbusak, kunci sing bocor, utawa beban nyata. Dadi rancang wewenan minangka least-privilege allowlists, gerbang aksi sing ora bisa dibali ngarep persetujuan manungsa, njalanake ing sandbox kanthi audit logs, lan ngluwari rahasia lan network egress kanthi sigrak ngidimu bisa nduwe leverage otonomi tanpa pertaruhan produksi bakal model iku bener saben wektu.