Recent rogue AI agent incidents have raised concerns about letting agents manage your finances.