After using idasql for a while I have noticed that, depending on the model selected and the binary at hand, inaccurate conclusions are still an issue, inevitably. But the actual problem is that after such a "turn" where the model hallucinates and populates the database with names, annotations, structures etc that are not accurate, then it's quite difficult to Undo the damage in the IDB.
In the harness chat itself, on the other hand, an obvious option is to do "/rewind" or "/undo" and go back one (or more) turns, this is implemented in Claude Code and Github Copilot Chat. But the database will still remain polluted and manual intervention is needed.
To solve this issue I suggest using the built-in Snapshot management mechanism in IDA (ida_kernwin.take_database_snapshot and ida_kernwin.restore_database_snapshot):
- Being able to enable/disable a mechanism that triggers a snapshot before each turn would make the most sense
- The restore itself might need some level of human intervention as the "/rewind" command doesn't seem to allow integration with an external checkpointing mechanism AFAICT. But at least there would be a way to effortlessly undo issues.
Here's an example. Assume the chat had 20 turns so far:
- you issue a command to the harness (annotate, re-source, etc), it finishes and this counts as turn 21
- idasql takes a database snapshot "Snapshot before turn 21" and then commits all the changes to the database
- you review it, it's inadequate in some way and it's difficult to undo the changes manually
- at this point the options could be:
- do "/rewind" one turn: in the harness you go to the prompt for turn 21 and in the connected IDA instance you can manually restore to the previous snapshot titled "Snapshot before turn 21"
- at turn 22 you tell the harness to revert the changes and it restores "Snapshot before turn 21" and then you rewind two turns (the revert and the faulty inferrence)
- next you (re)start at turn 21 and change the prompt or the model or whatever is needed
What do you think?
After using idasql for a while I have noticed that, depending on the model selected and the binary at hand, inaccurate conclusions are still an issue, inevitably. But the actual problem is that after such a "turn" where the model hallucinates and populates the database with names, annotations, structures etc that are not accurate, then it's quite difficult to Undo the damage in the IDB.
In the harness chat itself, on the other hand, an obvious option is to do "/rewind" or "/undo" and go back one (or more) turns, this is implemented in Claude Code and Github Copilot Chat. But the database will still remain polluted and manual intervention is needed.
To solve this issue I suggest using the built-in Snapshot management mechanism in IDA (ida_kernwin.take_database_snapshot and ida_kernwin.restore_database_snapshot):
Here's an example. Assume the chat had 20 turns so far:
What do you think?