Littlefinger tells Catelyn that the assassin´s dagger belongs to Tyrion now (including an easily disprovable lie about how Tyrion won it).
As a move to pit Stark against Lannister it makes very little sense.
The most likey outcome involves Littlefinger dying painfully:
Catelyn ("family, duty, honor") tells Ned.
Ned (more honor than brain) tells Robert.
And Robert´s reaction would have been
"Littlefinger said WHAT? That´s not his dagger, that´s one of mine. Ned and Tyrion, shake hands. Sir Payn, attend to Lord Baelish."
Robert will always trust Ned more than LF.
There are many other scenarios in which Littlefinger loses his head.
What if someone actually did recognise the dagger as one Robert got as a present some years ago?
Most of these factors are well out of LF´s control, and he has little chance to save himself.
Even after the events of the highly unlikely chance meeting between Catelyn and Tyrion in King´s Landing in an inn full of Stark bannermen,
there are many other possible outcomes:
- Tyrion dies, trial in King´s Landing
- Tyrion escapes, trial in King´s Landing
- Lannister men or Robert´s men intercept them, trial in King´s Landing
- Catelyn takes Tyrion north, letters fly
and Littlefinger will die.
(Littlefinger didn´t even make up some story why that information should be kept secret.)
Less dangerous moves to pit wolf against lion would have been:
- Claim one of his spies in the Lannister camp claimed to recognised the dagger. Spies sometimes tell tall tales for gold. If necessary LF could sacrifice someone who actually did spy for him at the lannister camp.
- Just tell Cersei that Catelyn is in town, plotting against Joffrey.
- Just let Eddard find out about the twincest, then tell Cersei that he found out
There could be another reason for LF making such a dangerous and nonsensical move (like the whole thing being Varys´ idea, and Varys promising LF protection), but I think it´s probably just a mistake by the author.
Maybe he originally envisioned the story differently, and later changed his mind? Or he originally didn´t envision LF as way less intelligent?